Currently evaluation is an ad hoc process conforming to the needs of the moment and limited by lack of skills, understanding and resources. As a result, INCORE has undertaken a pilot project seeking to examine the current state and utilisation of evaluation in conflict resolution interventions. The first part of the project consisted of a literature review, calls for information and a series of field interviews. That research resulted in the publication of this document. The second phase involved convening an international working group to begin addressing the questions and challenges of conflict resolution evaluation.
A conflict-specific framework has been developed to integrate the different aspects of an intervention that can be evaluated. It serves both to provide a platform for the field's thinking about current evaluations and to highlight gaps in the evaluation process requiring further research and refinement. The framework is structured around three themes, each of which has an essential question to guide thinking.
- Goals and Assumptions explores the use of conflict analysis when planning an intervention and the theoretical and ideological basis of an organisation's strategy. This theme asks: Why and how is the agency conducting this particular intervention?
- Process Accountability assesses the implementation of an intervention from the perspectives of management, cost-accountability and process. This theme asks: How was the intervention operationalised?
- Range of Results considers the results achieved through the intervention, both inthe immediate term and with respect to the longer and broader impact on society. This theme asks: What were the short and long-term results of the intervention? Beyond these themes, two additional concepts have been expanded to provide typologies by which one can discuss ideas essential to the evaluation of conflict intervention. The first, Focus of Change (p.34), addresses what an interaction is seeking to influence, while the second, Tiers of Influence (p.38), considers who an intervention is targeting.
Realities in the Field
In order to ground discussions about the evaluation process in current field experience, research was conducted with practitioners, evaluators and funders to get their views on conflict resolution evaluation. Some of the findings of this study were:
- Evaluations are conducted for a variety of reasons. Practitioners tend to engage in evaluation for the purposes of learning and in order to fulfil funding requirements. Conversely, funders are motivated by a need to review funding allocations and to ensure that agency goals are being met. These different motivations can cause frustrations when the end results of the evaluation only serve the requirements of one of the parties.
- Practitioners tend to conduct evaluations at the level of individual interventions whereas funders usually perform evaluations of a cluster of projects based on a particular theme or geographic commonality. Because of the use of different levels of analysis for evaluations, there are frequently misunderstandings and differing expectations in terms of what the final evaluation results will offer.
- All stakeholders considered the measurement of the intangible changes in attitude and perception that are an important part of peace work a particular challenge for which they had limited tools and resources. Without being able to explain or measure this type of change, it is very difficult to prove the impact of conflict resolution work.
- There is a preference for external evaluators as they are seen to be unbiased, legitimate and transparent. However, a number of practitioners expressed the desire to learn about alternative methods, such as self and participatory evaluation.
- Unlike for other fields, there is no training available in conflict-resolution specific evaluation methods. Many evaluators have some knowledge of social science methodology but they rely on their own experience to learn about the challenges of working in conflict. Practitioners found their lack of training put them at a disadvantage for planning and contributing to their own evaluation processes.
Innovative Approaches to Conflict Resolution Evaluation
There are several pioneers within the field who are developing innovative ways to address some of the shortfalls encountered when applying current evaluation models and frameworks to the conflict resolution context. Some new ideas involve the adaptation of existing models to meet the specific need of conflict resolution interventions, others focus on the establishment of field-wide criteria, while others are moving away from pre-established models altogether and using context-adaptable guidelines.
Questions and Challenges
If the field seeks to improve its ability to deliver meaningful evaluation results, a number of questions and challenges need to be addressed. Some of these include:
- The Challenge of Conflict Context: How can the ongoing changes in the context in which an intervention is operating be reflected in the evaluation process?
- Freedom to Acknowledge Failure: Is there an affirmative culture in conflict resolution whereby it is unacceptable to admit that interventions did not go as well as intended? How can the field address this challenge, particular in light of concerns over continued funding and legitimacy?
- Positive and Negative Unintended Effects: If unintended effects of an intervention are found during the evaluation process, how should they be balanced or weighed against the intended effects?
- Macroevaluation: Do individual interventions on the ground synergise to contribute to the development of a peaceful society? If so, how does this process occur?
- Concept of Success: There is no clear definition of what constitutes ‘success' inconflict resolution, so how do we know when an intervention has been successful?
As the discipline of conflict resolution matures, the need for the field to be able to understand, articulate, measure and compare will become increasingly important. New tools need to be designed, disseminated within the field, tested and refined. If those directly engaged in the work do not take up the challenge of finding methods and approaches that are suited to the unique challenges of conflict resolution, other, less useful methods will be imposed by those requiring evidence of the effectiveness of thiswork. Whether evaluation is tailored to meet the needs of conflict resolution or conflict resolution is tailored to meet the needs of evaluation remains to be seen - and is ultimately the choice of those engaged in the field.
Click here for the evaluation in PDF format.