Center for Global Development (CGD)

This 29-page document is a transcription of a May 31 2006 event - "When Will We Ever Learn: Improving Lives through Impact Evaluation" - hosted by Center for Global Development (CGD), an independent think tank based in the United States that works to reduce global poverty and inequality through research and active engagement with the policy community. As an initiative of the CGD, the Global Health Policy Research Network convened an Evaluation Gap Working Group to address the lack of knowledge about the effectiveness of social programmes in low- and middle-income countries. The Working Group sought to understand the reasons for the lack of good impact evaluation, with a focus on health and education sectors, and to explore possible ways to make significant progress toward solving the problem. The meeting transcript summarised below emerges from this effort. Click here to learn more about the event itself, to download the report that emerged from the event, to stream/download remarks and a PowerPoint presentation from the event, and/or to sign a Call for an International Initiative to Foster Independent Impact Evaluation of Social Sector Programs and Policies.

As indicated within this document, the first substantive address at the Washington, DC, USA event was made by the Working Group co-chair Ruth Levine, Director of Programs and Senior Fellow, CGD. Levine here speaks about the motivation and purpose of the Working Group initiative, which was created through a web-based survey of, as well as in-person consultations with, development personnel around the world. The problem, as she puts it, is that those in the development business and decision makers in developing countries who care about improving health, education and other social outcomes do not have the information they require to really make good decisions about how to spend money. She stresses that there is a lack of incentive at the institutional level to carry out systematic, independent, rigourous evaluation - such as evaluation that is part and parcel of the design of a programme, with design and data collection for the evaluation initiated at square one. This is due perhaps to the following 3 observations/challenges: the cost-benefit calculation at the institutional level does not include the public good that comes from an evaluation; the rewards for institutions and professionals within them come from doing, not from building evidence; and there is concern over the consequences of unfavourable results.

Levine's address is followed by words from the second keynote speaker, Working Group co-chair William D. Savedoff, Senior Partner, Social Insight. He begins by identifying several components that the Working Group found were central to a good evaluation. These evaluations:

1. tend to start in the design phase of the program itself
2. involve the policy makers and the managers from the start
3. include external actors and external actors to help with expertise with quality and with integrity
4. focus on impact - attribute the outcomes observed in the population to the specific programme
5. document the context and the process in the operations
6. address enduring questions
7. are selected and strategic

One explanation for the dearth of evaluations that follow such principles, according to Savedoff, is what he sees as the separation of researchers, managers, and policy makers. For that reason, collaboration is the key strategy that he and the Working Group have devised to address the incentive issues highlighted in Levine's remarks, particularly among developing country governments themselves as well as aid agencies. Savedoff observes that there are many good initiatives being carried out - "There's a lot of data collection going on. There's capacity building....There are also a variety of initiatives specifically aimed at improving impact evaluation." These efforts are "complementary and necessary and need to be strengthened to continue if impact evaluations, the information from impact evaluations is going to be useful in context." In addition to keeping focused on the work to be done, developing country governments can, collectively, raise standards for evaluation. The third thing that a collective initiative can do, Savedoff claims, is to bring independence to the review process. Finally, he notes "there have to be funds, substantial funds dedicated to good impact evaluation in some form or other." In closing, Savedoff proposes what the Working Group is calling the International Council on Impact Evaluation, a body that would be linked with experts who would "provide the panel reviews and the expertise for designing impact evaluations and keep the standards and rigor up to snuff."

The keynote speakers were followed by a panel discussion that featured: Jon Baron, Executive Director, Coalition for Evidence-based Policy; David Gootnick, Director of International Affairs and Trade, US Government Accountability Office; Kenneth Peel, Deputy Assistant Secretary for Development Finance and Debt, US Treasury; Nilmini Rubin, Professional Staff Member for International Economics, US Senate Foreign Relations Committee and; Franck Wiebe, Managing Director for Economic Analysis, Millennium Challenge Corporation. The remainder of the report shares individual reactions to, and dialogue about, the proposals made by the keynote speakers and the working paper more broadly. Levine and Savedoff engage in debate with those present. For example, Levine responds to a comment by sketching the shape of how a collective evaluation initiative could be linked with existing activities: that, is, by being designed as a sort of clearinghouse which has a networking function. Such an exchange would pick up on and share information such as that coming from the research departments in multilateral institutions where most of the impact evaluations are actually being done, as well as the evaluation work that’s being done in developing countries and within non-governmental organisations (NGOs).

e-CIVICUS Issue No. 294, June 2 2006 (click here for the archives); and CGD website.