Global Public Policy Institute (GPPi)
"Monitoring and evaluation (M&E) is critical for understanding the performance of aid, ensuring accountability to affected populations and donors, and allowing effective continuation of programmes amid insecurity. Nonetheless, insecurity can hamper every aspect of M&E, from the collection of evidence and its interpretation, to the sharing and dissemination of M&E information."
This final report summarises findings from the 3-year Secure Access in Volatile Environments (SAVE) research programme, which a United Kingdom Department for International Development (DFID)-funded effort to contribute solutions for providing effective and accountable humanitarian action amid high levels of insecurity. Humanitarian agencies have been working with different approaches to mitigate gaps in current practice and to address the challenges posed by insecurity. SAVE research focused on 3 of those: responding to community feedback in insecure settings; utilising third parties to monitor where access is constrained; and using technologies for monitoring. The research team investigated different strategies for monitoring aid in 4 contexts: Afghanistan, South Central Somalia, South Sudan, and Syria. The study assesses available options for aid agencies to monitor assistance, given the constraints that insecurity entails, and identifies principles of good practice and particular aspects where monitoring approaches could be improved. This paper describes its main findings, based on over 250 interviews, 65 focus group discussions, a review of over 300 M&E documents, and an online survey that garnered 3,313 responses across the 4 countries. Taking the sum of all findings into account, the report also offers higher-level lessons on monitoring in insecure environments and suggests broader policy implications.
In all 4 countries, there is a widespread perception that M&E systems need to be improved. A SAVE survey revealed that 37% of aid agencies reported to be "not so satisfied" or "not satisfied at all" with their own M&E system. The further an organisation is from the field, the greater is the demand for monitoring systems and the higher its concerns about it: The majority of the United Nations (UN) agencies from the 4 settings were "not so satisfied" with their own M&E systems. International non-governmental organisations (INGOs) are slightly more positive in their assessment, as around two-thirds find their own M&E satisfactory. National NGOs are the most positive.
Practitioners see a lack of capacities as the biggest constraint to better M&E. This relates in particular to a lack of skilled staff and technical knowledge on M&E within aid agencies and their partners. Furthermore, a lack of willingness to share data and increase the cooperation between agencies in these settings constrains the effectiveness of M&E. Implementing agencies and local organisations close to the ground need to lead efforts to communicate with affected populations. In addition, SAVE found that monitoring systems are comparatively weak at showing aggregated and country-wide effects, impact on conflict drivers, and demonstrating longer-term impact.
The study found that current monitoring systems are perceived as largely inadequate for achieving accountability to affected populations. Over 85% of the aid recipients surveyed indicated they had never been consulted about the aid efforts. Communities criticised aid agencies for relying too much on community leaders or "gatekeepers" and for not involving them in the planning and implementation of projects, and complained about not hearing back after providing feedback. Communities appreciated phone and internet-based feedback systems where they work, but noted that these cannot substitute for face-to-face conversation.
The research identified the following lessons and principles of good practice:
- Invest more in communicating with and involving communities: "...Where access is constrained and opportunities for informal and spontaneous personal interactions with communities are rare, it is critical to offer alternative communication channels to communities....Rather than reinventing the wheel, aid agencies should adhere to standard good practice for feedback mechanisms. This involves asking communities about their communication preferences at the onset of a project and combining technology with more traditional approaches....But the demand for more-direct communication and inclusive programming processes cannot be met with better feedback mechanisms alone. A broader set of approaches is needed in insecure settings, such as providing communities with timely and reliable information on the crisis situation and on available humanitarian services. Inter-agency or collaborative feedback systems can be helpful to collect, aggregate and analyse feedback, and to collect overall data on community perceptions....Multiple stakeholders need to work together to enhance communication with affected populations....Implementing agencies and local organisations close to the respective populations need to lead communication efforts."
- Continue to invest in monitoring by agency staff and contract third parties to collect and verify monitoring data as a last resort. Good practices for third-party monitoring (TPM) include:
- Ensure that adequate capacity is in place for selecting and training monitors.
- Invest in internal systems for using data from TPM and getting it to decision-makers.
- Further develop the use of technological devices to increase control over field monitoring.
- Jointly assess security risks for monitors and existing security systems of third-party monitors.
- Coordinate use of TPM among peers and exchange on emerging lessons.
- Limit primary reliance on TPM to exceptional areas with constrained access.
- Regularly reassess TPM and invest in access strategies to return to (more) internal monitoring.
"To facilitate as much [as one's] own monitoring as possible, TPM should always be complemented with acceptance-building measures and community feedback systems, and overall transparent communication with communities." Editor's note #1: SAVE has produced a report dedicated to the use of TPM in Afghanistan, Somalia, and Syria. The full report can be found here [PDF], along with a short summary [PDF] of the main findings and recommendations.
- Explore which technologies can support monitoring in each setting: "...To provide a more comprehensive and independent overview and help practitioners make more informed decisions about the use of technologies for monitoring, this research assessed major technology types and reviewed available documentation on individual applications. The results of this exercise are presented in a detailed 'Menu of Options'. Overall, the following benefits stand out:
- Mobile phones can broaden the reach of feedback systems, but provide little information on sensitive issues such as aid diversion.
- Digital data-entry applications save time, enhance data quality and speed up transmission and analysis.
- Satellite imagery, while still rarely used, can provide independent data and support situation and impact monitoring.
- Movement tracking devices can help identify and prevent [aid] diversion.
- Radio can inform communities with interactive programme formats.
- Online communication platforms offer an alternative where phone networks do not work, but internet is available.
Using innovative technologies in highly insecure settings inevitably involves risks. Aid organisations should therefore consider the following risk mitigation strategies before and when using technological applications for monitoring:
- Understand who influences and spreads information in your context before choosing tools.
- Work with users when inventing, designing and testing tools. Use trainings and meetings with local staff, authorities and community members to test and explain technological applications.
- Develop standards for handling data, ideally before a crisis hits.
- Put analogue alternatives for data collection and management into place.
- Use security-conscious, free and open-source software.
- Only collect data that you know you will use.
- Collaborate with others to share costs and risks."
- Create monitoring approaches that include applied learning and capacity development. Good practice combines verification and capacity development. For example, monitoring and reporting specialists working with the Common Humanitarian Fund (CHF) in South Sudan help partners design their M&E systems, provide mentoring and hands-on advice, and verify partner reports. "The contributions of the M&R specialists were highly valued....Overall, they were seen to contribute to the transparency and accountability of CHF-funded projects. The case study offers lessons for similar approaches in other countries."
- Increase alignment of data needs "up the chain" and encourage greater transparency of results: There is "potential for making monitoring more efficient higher up the monitoring chain, at the country office, cluster, consortia and headquarters levels. Actors at these levels should align their monitoring indicators as much as possible and agree on the exact variables to be used as the core minimum indicators. Any changes to existing monitoring arrangements should be carefully considered, taking into account effects on field teams. Moreover, monitoring requirements need to be more flexible so that they can be adapted to the type and scale of goods and services delivered in an emergency....Finally, teams involved in monitoring at all levels should attach greater priority to sharing to their data sources, results and how they are used."
The findings suggest that investments in monitoring need to be better-targeted and more-strategic. Rather than creating additional layers of monitoring at partner, agency, cluster, consortium, donor and country levels, more reflection is necessary on which monitoring functions are needed, at what level, and the appropriate overall level of investment." Other policy implications include: More-inclusive programming can be encouraged by donors and supported by third parties collecting data on community perceptions. Communication efforts, however, need to be led by implementing agencies and local organisations close to the respective populations. Furthermore, strengthening capacity should be the first priority for further investment. Finally, "[t]he decision to provide assistance in highly insecure environments is a risk shared by donors and their partners. Donors should reward partners for being transparent about good and bad results, rather than for keeping an appearance of total control and accountability."
SAVE website, January 3 2017; and email from Lotte Ruppert to The Communication Initiative on January 4 2017. Image credit: © Nichole Sobecki/AFP/Getty Images