Valuing Cultural Scorecards: What Counts? For Whom? Cultural Scorecards as Communication Measures

This article, presented at the National Communication Association (2011), New Orleans, United States, and published in Spanish (in an earlier highly-edited version) in Folios, a journal of the Universidad de Antiouquia in Medellin, Colombia, draws upon research conducted by the authors in Uganda, India, Peru, and Senegal. The article discusses the concept of cultural scorecards for research and evaluation projects. These are described as "culturally-embedded, user-defined, and non-textocentric measures for understanding communicative meaning(s), components, and sites of change."

From the abstract: "The first part of this article makes explicit the connections between participatory, non-textocentric methods and the emergence of cultural scorecards as communication measures. It then describes certain attributes of cultural scorecards and illustrates their value as user-defined metrics of assessment in four projects: one in Northern Uganda during an assessment of a child protection project; another in rural India during an assessment of the impact of an entertainment-education radio soap opera; another in the Peruvian Amazon during an assessment of an NGO’s on-air and on-the-ground capacity-building initiative; and another in Dakar, Senegal, during implementation of a youth development program. Based on the analysis and epistemological conclusions of these examples, the authors encourage participant-investigator collaboration to attend to culturally-embedded assessment metrics, test their reliability and validity, and expand traditional notions of what constitutes data."

The dilemma of externally imposed measurement tools is posed as: "Skeptics might argue that participants’ voices are not wholly absent from contemporary research; after all, practitioners commonly administer written surveys, structured interviews, and guided focus groups. However, these instruments require participants to express in words their ideas and feelings (and often, these ideas and feelings pertain to researchers’, as opposed to participants’, areas of interest). This embattled translation process limits full participation and circumscribes participants’ range and depth of self-expression...." For various reasons, potentially valuable participants may be excluded, whether by taboos of discussion of sensitive topics, by a small sample size, by alienation of participants, or for other reasons.

The article lists some techniques that are not from the cadre of Westernised instruments (surveys, focus groups, etc.) used in research. They include: performances (artistic, musical, oral, and visual); participatory visualisation techniques (e.g., participatory photography and sketching) accompanied by oral narratives and storytelling; and the Most Significant Change (MSC) technique of soliciting participants’ change-narratives.

It then describes the emergence of the cultural scorecard and its use: "Cultural scorecards emerge from a wide variety of data gathering techniques, encompassing but not limited to those described above. Operationalizing and validating cultural scorecards is achieved through triangulation and theoretical sampling; respondent groups are chosen for theoretical rather than statistical reasons and respondents are added until theoretical saturation (incremental learning is minimal) occurs.....

Our analysis suggests that a scorecard exhibits one or more of the following attributes: (1) it is culturally-embedded - that is, it involves data that may be “invisible” to outsiders such that one must be a member of the culture in order to appreciate its significance; (2) it is user-defined - that is, the assessment metric is created by participants themselves; and (3) it is (usually) non-textocentric - that is, the indicator of change is not captured by (and perhaps cannot be wholly captured by) textual methods."

Following examples from fieldwork that includes drawings, photo images, and stories, the authors conclude:
"...Our analysis of cultural scorecards points to two main conclusions: (1) In carving a space for locally-relevant, culturally-embedded, user-defined metrics, cultural scorecards legitimize grassroots epistemologies, and (2) the potential for the discovery and understanding of cultural scorecards exists beyond the scope of the mentioned research and beyond structured qualitative assessments.... [We] encourage communication scholars and researchers of other social science disciplines to: (1) engage research partners and participants in monitoring and assessment activities from the beginning of a project; (2) co-construct multiple means of both gathering participants’ insights and translating this rich data; (3) listen appreciatively to locally-defined ways of knowing; and (4) multiply document these metrics and findings, in print and non-textocentrically."

The full article is available in PDF format in English through Google documents or by contacting: Karen Greiner: kgreiner - at - gmail.com