Abstract

”Monitoring and evaluation [M&E] has been designated as an essential component in national AIDS response by international funders and in the context of large scale funding programmes based on results-based disbursement, much pressure is being brought to bear on programme developers to incorporate M&E (monitoring and evaluation) in their planning and operations. However, the situation on the ground at national, regional and local level is that there is lack of understanding of the challenges involved, lack of understanding of the basic principles and processes of M&E and lack of capacity to develop and manage M&E systems.

The paper presents the results of a country-level assessment of M&E practices which was developed as part of a capacity building process for M&E to be incorporated into multisectoral AIDS responses in Swaziland. The article develops a critical perspective on the emphasis of international programmes for M&E capacity development...

The objectives of the evaluation were to:

• examine national values, policies, needs, priorities, preparedness and practices relating to M&E;
• identify existing HIV/AIDS M&E programmes, resources, best practices and gaps at the national, district, sectoral and community levels;
• assess M&E training needs;
• assess the use of data to improve programs and to make recommendations for improved data utilisation;
• generate a detailed report with clear recommendations for the development of M&E capacity and practices within the HIV/AIDS response system.

Methodology

There were two stages to the assessment. Firstly, a scoping exercise was conducted to provide in-depth understanding of the context, and to identify key issues relating to M&E capacity in HIV/AIDS organisations across Swaziland. Secondly, an organisation survey was conducted to identify prevailing M&E practices, to take stock of existing skills and gaps, and to understand M&E capacity development needs....A total of 103 organisations responded to the questionnaire, representing an 82% return rate. These included government departments, nongovernmental organizations [NGOs], community based organisations [CBOs], faith based organisations [FBOs], private sector institutions, academic institutions and donor agencies.

Findings

There is a high level of computer ownership within service organisations in Swaziland (86% of NGO/FBOs; 85% of government agencies). This means that most organisations have the basic technology for implementing electronic data capture and information management. However, one in four organisations do not have internet connections and this would need to be remedied to facilitate relaying of programme data in an electronic M&E system.

Very few organisations (12% of NGOs/FBOs; 8% of Government/local government agencies) have an M&E unit or staff whose primary designation is M&E. More than half the NGOs/FBOs and almost two-thirds of government service provision agencies have not assigned M&E responsibilities to a member of their organisation. Interestingly, 38% of all organisations plan to hire designated M&E staff in the future, and of these many cite financial constraints as a reason they have not done so to date.

Although a high number of organisations indicated that they have annual work plans for the organisation, less than one third have an M&E strategy. There is a marked decrease in the number of organisations in the sequence from having a documented set of key indicators (69% NGOs/FBOs), to use of monitoring tools (48%), to capturing monitoring data on a computer (29%), to charting monitoring information (19%). It is suggested that this means that many organisations have made some investment in M&E development, but that this has not been supported and carried through as a programme activity.

• Less than half of NGOs/CBOs produce reports which contain monitoring data
• Only 12% produce monthly reports (36% quarterly reports) which contain monitoring data.
• Almost one third of NGOs/CBOs never produce a report on their HIV/AIDS activities. An even greater proportion of government agencies never report on their HIV/AIDS activities.
• It is likely that if there is no report there is no review, and no real attempt to take stock of programme achievements. It also suggests a strong need to build an appreciation of the value of reviewing programme activities and outputs, beyond the requirement of reporting to funders.

...The following are some of the issues that need to be addressed if M&E systems are to be adequate to the task of ensuring M&E systems that are integrated from local to central levels.

• There is a need to adopt a more systematic approach to development of M&E systems within projects and to track implementation of the same. Each funded project should have an M&E system, including indicators and measures and tools, that is appropriate to its operations and is consistent with national reporting categories....
• There would also be strong value in creating an M&E manual...
• There is a strong need for all funded projects to undergo a basic training in M&E principles and practices. The value of M&E as both a reporting and programme management practice needs to be endorsed...
• There is a need to develop much stronger field support and technical assistance in M&E...

The following needs for M&E capacity development for HIV/AIDS service providers emerged from the assessment.

• There is need for development of a M&E capacity building programme which will need to include the following components: basic principles and practices of programme evaluation; development of M&E strategies and frameworks at organisaitonal level; development of indicators and measures; basic data capture and analysis; report writing; and alignment with national M&E systems.
• Needs for training in more specialised areas of M&E are described in the report...
• There would be long-term value gained in supporting the development of M&E modules and courses which might be incorporated within curricula at the University of Swaziland...
• There is need for a register of consulting organisations with details of the specific areas in which they are equipped to provide technical assistance and training. There would be value in orienting consulting and training organisations to approaches and standards for M&E in the HIV/AIDS field...."