There is growing recognition of the need to take a multi-stakeholder approach to evaluation, which promotes local ownership and builds capacity for reflection, learning, improved performance and ultimately self determination. As a story based technique, the [most significant change, or] MSC approach helps to identify and lends value to changes that were unintended or unexpected but were nevertheless significant impacts for those involved."

This 37-page report explores the use of the "most significant change" (MSC) technique as a participatory monitoring tool to evaluate a specific communication-based programme. It was presented at the conclusion of a 13-month Department for International Development (DFID)-supported programme on community-driven approaches for addressing the feminisation of HIV/AIDS in India.

Launched in February 2006 by the non-governmental organisation (NGO) India HIV/AIDS Alliance (Alliance India), the campaign in question involved fostering community-driven efforts to address the feminisation of HIV/AIDS in India. In keeping with the mandate of the Alliance, this project was based on an understanding that community sensitisation, mobilisation, and empowerment efforts can contribute to ensuring that women's needs and rights are accorded greater prominence in designing and implementing HIV/AIDS and sexual and reproductive health (SRH) activities. In brief, theatre, film, and posters were used in an effort to develop and strengthen community-centred approaches to meet the HIV/AIDS-related needs of women in low-income settings. Alliance India further sought to focus on creating increased informed demand and enhancing awareness and knowledge of HIV/AIDS and SRH, with special emphasis on increasing access to health, social, and legal support services for women affected by HIV/AIDS or vulnerable to HIV. Alliance India worked to accomplish these goals through partnership with 19 NGOs in 17 districts across 6 Indian states of Tamil Nadu, Andhra Pradesh, Punjab, Manipur, Orissa, and Delhi.

This project has also sought to create a body of knowledge that can be readily used by a range of stakeholders, including policymakers, donors, NGOs, the mass media, and academia. To that end, this report focuses on how the MSC technique can be used to gather and analyse stories of important or significant changes from a cross-section of the group(s) a project is designed to reach. While Alliance India stresses that this MSC should not be used as the sole monitoring and evaluation (M&E) technique for gauging the measurable difference a project has made, the organisation has found that this evaluation methodology can provide a richer picture of the impact of programme interventions.

Despite its potential, the MSC approach was found during this evaluation process (January-February 2007) to raise some challenging questions, such as: What do we mean by impact in a project with a life cycle of one year, and how does one assess the impact of such a project? How can you train teams - especially, those with little or no prior experience in data analysis - in less than one day on effective facilitation and documentation so that they are able to collect significant change stories with requisite and relevant details?

Here is an excerpt from the document which details the character of this methodology, and the findings that emerged from its use in analysing the success of Alliance India's initiative [Footnote numbers have been omitted from the selection]:

"MSC can be better understood through a metaphor - of newspaper, which picks out the most interesting or significant story from the wide range of events and background details that it could draw on. Dynamic values inquiry is a central and critical part of MSC. When key stakeholders select stories of significant change, they participate in an ongoing process of deliberation about the value of individual outcomes. Designated groups of stakeholders continuously search for significant programme outcomes and then reflect on the value of these outcomes....Rick Davies, the creator of MSC, reiterates how the emphasis on storytelling makes MSC different to formal monitoring techniques. Instead of introducing new professional skills, MSC takes advantage of everyday communication practices. "Every language has an expression for 'What’s new?' says Rick."

...Conventional quantitative monitoring of predetermined indicators only tells us about what we want to know and think we need to know. It does not drive us to explore beyond what is obvious - the indefinable, subtle and indirect consequences of our work. By getting this information on a regular basis, and taking time to reflect on what this means, organisations can adjust and modify their direction of effort so that they achieve more of the outcomes they value.

...[MSC] emphasises story-telling as a form of participatory monitoring, unique in its ability to capture direct views of beneficiaries, including unexpected and evolving outcomes. It is also unique in presenting a methodical, systematic way of collecting and analysing individual stories and selecting the most compelling...

In many organisations, there are designated personnel involved in information collection, analysis and dissemination. Indicators are often identified by senior staff or the M&E specialist. In contrast, MSC gives those closest to the events being monitored (e.g. the field staff and participant communities) the right to identify a variety of stories that they think are relevant and important. These are then summarised by selection when other participants choose the most significant of all the stories reported. By valuing diversity of views, it gives an opportunity for organisations to decide what direction it wants to go and how. This enables having a more shared vision....With predefined indicators, the nature of information and its meaning is largely defined from the beginning. With MSC, participants are encouraged to use their own best judgement in identifying stories and selecting stories collected by others. This involves the use of open-ended questions such as: "From your point of view, what was the most significant change that took place concerning the quality of people's lives?" This freedom empowers participants involved in the process to think and analyse deeper the nature and impact of their actions - field staff and target population do not only collect information (or give information), they also evaluate that information according to their own perspective, understanding and experience.

...MSC makes use of what has been called 'thick description': detailed accounts of events placed in their local context. Each MSC story has the storyteller's interpretations of what is significant. This also enables a changing focus on what is relevant and important as the MSC process is dynamic and is responsive to the context in which it is used. Participants choose what to report within specified domains.

In addition to its monitoring and evaluation functions, MSC can also help in

• Fostering a more shared vision;
• Building staff capacity in monitoring and evaluation;
• Providing material for publicity and communications;
• Providing material for training staff; and
• Celebrating success...

...The key findings of the evaluation are:

• The project has achieved significant impacts on people's lives where it was implemented in conjunction with other ongoing programmes of the implementing organisation. For instance, Child Survival India (CSI) integrated the project with its longer term programmes on women empowerment and adolescent girls' vocational skills training. Similarly, Social Awareness Service Organisation placed this project in those areas where their Care and Support programme with Injecting Drug Users (IDUs) were ongoing. Most of their 'target population', therefore, were wives of current IDUs or ex-IDUs. As a result, not only were the existing groups further strengthened by awareness on sexual and reproductive health issues, legal rights, and linkages with health care providers, but there was wider community acceptance and influence, as was seen in Delhi. In Manipur, this led to the affected women getting
  the much needed psychological support, awareness and opportunities for earning an income.
• Four out of the six Implementing NGOs (iNGOs) chose a domain on 'changes in quality of people's lives'. The changes reported in this domain included, increased levels of confidence among target population to challenge stereotypes and stigma; enhanced ability to overcome fear of being deserted by family; and a more positive outlook towards life.
• Through stories it came out that total involvement of men in the projects is very much important to mitigate tensions arising when power structures and institutional norms are challenged. Adolescent girls and women...have reported facing ridicule and shame in their communities when they attempted to challenge stereotypes or patriarchal norms.
• In terms of building the capacity of iNGOs on different issues, all stories in the domain 'changes in staff capacity' reveal an increased level of awareness and confidence of field staff and volunteers. All staff revealed that they have undergone a distinct change in their perspectives and mindsets.
• Almost all stories from the different domains reveal the critical and significant role played by peer educators in increasing the project's sphere of influence and acceptance within communities.
• One year is too less to gauge qualitative impact of a pilot project; however, it is evident that organisations who have placed this project with their earlier ongoing, longer term interventions, have shown significant changes in communities challenging social and cultural norms.
• Many stories in the document clearly reiterate the need for a longer term intervention plan especially on issues related to sexuality, gender and altering existing power equations. Moreover the use of this technique has clearly
  highlighted that enough efforts need to be invested in mobilising the whole community and not just the key population while pursuing such programmes. This will result in gaining the confidence of the entire community and provide enabling environment to women including adolescent girls."

Click here to access the full publication online. For information about how to request a printed copy of this report, contact info@allianceindia.org.

Email from Priya Mohanty to The Communication Initiative on May 7 2007; and India HIV/AIDS Alliance website.