This report examines the methodology for measuring the effectiveness of media development programmes among media development organisations, tracing the increasing use of monitoring and evaluation (M&E) among practitioners in the media assistance community.

The author of this piece, journalist and editor Andy Mosher, has learned through interviews with more than a dozen donors, media assistance implementers, and professional evaluators that M&E has become more rigorous since the 1990s as its importance has increasingly become appreciated. While terminology and methods vary widely, all organisations in this sector seem to agree that the M&E process must begin long before the project itself - with the formulation of a proper plan. M&E practitioners also share a number of tools, techniques, and approaches, such as marshalling of baseline data, use of content analysis, balancing quantitative and qualitative data, and employing outside evaluators. The heart of many M&E plans, Mosher observes, is the logical framework, or "logframe". This is a tool that charts the path of a project, from gathering resources and identifying a problem (inputs), to developing activities to address the problem (process), to documenting the results of those activities (output), to establishing what effect the activities have on participants (outcome), and to determining the longer-term effects on the society in which the participants live (impact).

Mosher discusses the different shapes that M&E for media assistance can take, and the challenges associated with - for instance - figuring out how to establish cause-and-effect relationships between programmes and societal change (the so-called "attribution issue"). In one text box within the report, he notes that along with the evaluations generated by individual projects, a handful of broader studies look at the "media landscape". Examples he discusses include Freedom House's Freedom of the Press Index, International Research and Exchanges Board (IREX)'s Media Sustainability Index, and Reporters Without Borders' Press Freedom Index. In Mosher's estimation, these big-picture analyses can complement ground-level evaluations of individual projects.

One element of Mosher's discussion focuses on the strategy of establishing a hierarchy of impacts. One International Center for the Journalists (ICFJ) representative quoted here explains that his organisation looks first at whether a programme resulted in the establishment of something concrete and measurable, such as a journalism centre, a journalists' organisation, or a self-sustaining digital platform for media. The second level includes increases in skills among individual journalists, evidence of structural change at media outlets, or changes that benefit journalists as a whole. Last come changes to society; this is the most difficult to measure. However, "there was a clear consensus that if they can show that they are conducting their projects flawlessly but cannot show that they are making a difference in people's lives, they are failing their funders and the people they are trying to serve."

Section III of the report focuses on "what works and what doesn't". Here are some key points, which are illustrated throughout the section with quotations from M&E practitioners:

• Baseline data are crucial; it is impossible to show that a programme has fostered change if no information has been gathered to establish what went before it.
• While acknowledging they are inherently difficult to use, many evaluators find logframes invaluable as a means to ensure that a project's various steps follow a logical path from start to finish.
• Documenting the number of participants in a training programme serves only a limited purpose, as it doesn't tell you whether they learned something or made a change in their reporting as a result.
• The focus group methodology can be useful in showing clear attitudinal shifts in the audience of a media intervention.
• Tracking the effect that a media assistance project is having on what media outlets publish or broadcast is a technique employed throughout the sector. Content analysis typically involves monitoring a newspaper's articles, a broadcaster's programmes, or a digital platform's content and assessing its accuracy, variety, readability, or other attributes.
• The fruits of M&E are useful not only to donors and implementers but also to the people and organisations their projects are designed to serve.
• All of those interviewed for this report saw value in including both statistical data and narrative prose in evaluations.
• There has been a trend toward strengthened in-house expertise, as implementers feel they can handle the relatively simple tasks of reporting, compiling, and analysing data while a project is being conducted - that is, monitoring. But, whenever possible, they rely on outside evaluators to take on the more complex and demanding job of assessing the overall impact of a project - that is, evaluation.

According to Mosher, "while all parties say they regard credibility as extremely important, the question of money appears to weigh most heavily in determining whether outside evaluators enter the picture at all." Section IV of the resource focuses on the issue of cost, noting that media assistance practitioners agree that M&E costs a lot, but there seems to be no agreement as to what percentage of total project cost M&E should be, or about whether it is practical to think in such terms.

"The growing emphasis on more and better M&E for media assistance projects shows every sign of intensifying in the coming years. The push will be both driven and enhanced by advances in technology..." To illustrate this trend, Mosher offers several examples of new M&E undertakings drawing on information and communication technology (ICT); to cite only one, the Catholic Media Council, or Cameco, is developing a wiki (an online, open-platform repository of information) to which media assistance practitioners from around the world can contribute M&E plans, evaluations, best practices, and related material. The extent to which the M&E field will evolve over time, however, "depends in large measure on the willingness of media assistance organizations to share information about M&E...that is not competitive or proprietary..." There may be hope for progress in this sense: the Global Forum for Media Development (GFMD) is bringing together media development organisations, key media researchers, and existing initiatives to prepare a handbook or toolkit on M&E of media development programmes. This endeavour will involve amassing data about the media landscapes in individual countries as well as compiling indicators, data sources, and tools for measuring impact. GFMD is described here as an open forum with broad participation level globally, which might make it particularly suited as a platform for M&E cooperation.

Concluding recommendations, discussed in more detail in the resource, include:

• Fund M&E more aggressively, and establish equally aggressive requirements.
• Develop a shared - but adaptable - approach to M&E methods.
• Sharing country- and region-specific baseline data could save organisations a considerable amount of money.
• "[S]teps should be taken to at least estimate the heretofore uncalculated costs of in-house M&E. Only when that is done can organizations determine what proportion of a project's budget is being spent on M&E, and they should seriously consider setting a target proportion and sticking to it."

CIMA website, July 23 2009.