University of North Carolina Chapel Hill, Carolina Population Center (Calhoun); Johns Hopkins Center for Communication Programs (Shaw); University of North Carolina Chapel Hill, Carolina Population Center (McGuire); Johns Hopkins Center for Communication Programs (Shore); University of North Carolina Chapel Hill, Carolina Population Center (Speizer)
“Measuring a program’s success should begin when the program is being designed.”
This end-of-project report shares the experiences and lessons learned from the Measurement, Learning & Evaluation (MLE) Project, which was the evaluation component of the Urban Reproductive Health Initiative (URHI), a multi-country programme in India, Kenya, Nigeria, and Senegal that aimed to improve the health of the urban economically poor. The MLE project’s goal was to promote evidence-based decision-making in the design of integrated family planning and reproductive health (FP/RH) interventions for the URHI. The URHI was implemented by country consortia (CCs) in each of the four countries, and MLE collaborated with the four CCs to measure the impact of their programmes, to use monitoring and evaluation results to adjust programme activities, and to identify promising best practices (see Related Summaries below for more information about the project).
The MLE project was tasked with three objectives that guided programme activities:
- Monitor and evaluate the impact of the URHI within and across target countries using a rigorous study design and multiple data collection approaches. This involved assessing the change in individuals, families, or communities that can be attributed to a project, programme, or policy.
- Build country and regional capacity to undertake rigorous measurement and evaluation of population, FP, and integrated RH activities with a focus on urban and peri-urban economically poor and vulnerable populations.
- Facilitate knowledge sharing by documenting and disseminating best practices across CCs, in the region and within the global community of practice.
To ensure successful local, national, regional, and global dissemination and use of programme results, the MLE project and the CCs fostered collaborative relationships with key stakeholders to encourage evidence-based policymaking and allocation of resources that improve urban FP programmes.
The report explains in detail the project's evaluation design and elements. In order to achieve the first objective of monitoring and evaluating the impact of the URHI, MLE deployed a hybrid study design, which integrated cross-sectional and longitudinal data to assess the direct impact of innovative FP programmes and identify any change in the contraceptive prevalence rate (CPR) and behavioural and attitudinal norms in intervention cities over the course of the programmes. These design elements allowed the MLE project to measure programmatic impact across cities, over time, and among the urban economically poor. Where possible, the team used programme cost analysis information to contribute to an impact analysis that captured the cost of individual interventions. To explain the scope of the evaluation, the report provides information in the form of a table that gives the numbers of women, facilities, providers, and clients surveyed at each phase of the survey, as well as the response rates in the longitudinal sample.
The results of the impact evaluation are explained in the report as follows (complemented by graphs and tables): "In target cities in all four countries, mCPR [modern contraceptive prevalence rates] increased significantly between baseline and endline. These mCPR increases were generally observed for all women, including those in the highest and two lowest wealth quintiles. Greater increases were often observed among women in the lowest wealth quintiles. The figures do not reflect large increases in use of long-acting and permanent methods including implants, IUDs [intrauterine devices], and mostly in India, female sterilization. These increases were observed among all women and across the wealth groups. Using statistical methods that take advantage of the data on the same women and facilities over time, the project demonstrated that numerous factors were associated with increased modern contraceptive use in each of the study contexts. In India, women exposed to brochures/billboards/posters, pamphlets, community health workers, and living close to project facilities were more likely to be users. In Nigeria, the key factors driving use were exposure to local radio programs, television programming, outreach activities where family planning was discussed, and living close to a program facility. Likewise, in Kenya, local radio programming, as well as exposure to community health workers and exposure to program facilities, led to greater modern method use. Finally, in Senegal, among women, those exposed to ISSU [Senegal Urban Health Initiative] radio programming on FP and those exposed to community-based activities were significantly more likely to be using modern family planning at endline. MLE also found that for Senegalese men, exposure to religious leaders discussing family planning and a variety of other URHI program activities were associated with increased reported use.”
To achieve the second objective - to build country and regional capacity to undertake rigorous monitoring and evaluation efforts - MLE hosted trainings to continue to build capacity among staff and regional partners. To achieve objective three, which involved the sharing of knowledge, team members published numerous papers, presented at conferences, and maintained communities of practice on regional and global levels. These efforts served to strengthen FP interventions and understanding of monitoring and evaluation methods. The report lists some of the project’s outputs in the form of publications and trainings together with hyperlinks to download the documents.
Finally, the report shares some of the lessons learned related to the evaluation design and process. It shares some of the lessons on the collection of longitudinal data in dynamic, chaotic, heterogeneous environments where there is high population mobility and ever-evolving FP supply environments. It looks at the importance of open and collaborate engagement with all stakeholders in order to coordinate a study design that was comparable across countries. It highlights the importance of using more than one evaluation tool, as well as the importance of preparation, having the right team, and face-to-face interactions between MLE staff members and country partners working in the field.
Measurement, Learning & Evaluation Project website on July 17 2017.