Overview, Case Studies, and Lessons Learned

Dennis Reineck
Laura Schneider
Christoph Spurk
Esther Dorn-Fellermann
Charles Nyambuga
Roland Schuerhoff
Publication Date
October 1, 2017

"Though audience research has also been part of the toolbox of media development organizations for some time, it was only when focus in international development shifted from producing outputs to effecting impact that research of media viewing, listening and reading gained the importance for media development it has today."

This DW Akademie study defines audience research in media development, gives an overview of recent developments in the field, and presents three case studies conducted on behalf of DW Akademie in various fields of audience research (measurement of media and information literacy, quasi-experimental design, triangulation of social science, and digital analytics methodologies). The case studies, which include discussion of results and lessons learned, are meant as an orientation and inspirational source for future projects in this domain. Based on the information assembled in this study, the authors make the case that media development needs focused audience research in order to gain specific and relevant knowledge directly related to the interventions. That is, media development actors need to know more about the impact on their final beneficiaries if their work is to be truly successful.

Chapter 1 defines audience research in media development, outlining its functions and presenting criteria for good practice. (Audience research in media development is defined here as "any rigorous empirical enquiry into the behavior, knowledge, and attitudes of persons in the developing world receiving, engaging with, and/or non-professionally contributing to media content, on the basis of social science methodology and/or technical measurement. Audience research in media development can have functions for exploring potential target groups, monitoring the progress of media projects, measuring the outcome of media projects, evaluating the impact of media projects, or contributing to the sustainability of media outlets.") It is argued that audience research today has to take both active and passive roles of media audiences into account, relying on traditional social science data collection methods and digital metrics to obtain a valid description of the behaviour, wants, and needs of viewers, listeners, and readers. The overview chapter also looks at the current field of audience research in media development. A content analysis of 163 studies in the field identified 5 sources of import to people involved in media development:

  • Research conducted by members of the Conference of International Broadcasters' Audience Research Services (CIBAR) provides valid data available on media access, use, and attitudes in the developing world, though the detailed research results are only available at a financial cost.
  • Studies by state bodies and non-governmental organisations (NGOs), especially prevalent in Latin America, can also deliver valuable information for the media development practitioner.
  • Media and market research can be a cost-effective way of obtaining audience research results, though publicly funded research is not readily available.
  • Academic studies are most useful when they test research methods that might be used in a media development context.
  • Media development organisations themselves deliver research of varying quality and have very different approaches in terms of the transparency of their research.

The aim of this project was to gather experience in the field of audience research. Three case studies were conducted for this purpose and are presented in Chapters 2-4. In brief.

  1. The first case study addresses the measurement of Media and Information Literacy (MIL) of young media users aged 12 to 16 years in the Palestinian Territories (West Bank). From the media development perspective, the following insights were gained:
    • The schools that had taken part in MIL courses ("project schools") fared slightly better than those that did not, though the differences were smaller than expected.
    • Girls fared much better than boys. Project school girls also received significantly better test results than their non-project compatriots.
    • The project school participants fared worse than their non-project counterparts at only 3 skills (source transparency, source diversity, and respect of privacy).

    Lessons learned from a methodological audience research perspective were:

    • Measuring MIL in this age group requires hands-on testing. Abstract questionnaires do not seem to have the same validity as practical tests.
    • Media examples and item language have to be adjusted to the levels of juvenile age groups.
    • Absolute comparisons of test items are not possible because it is almost impossible to construct MIL questions that are exactly of the same difficulty. Thus, relative comparisons should be drawn between different groups of pupils.

  2. The second focuses on the impact of radio broadcasts on women and land rights (station: Nam Lolwe) in Kenya. It was designed as a quasi-experiment, comparing listeners of the radio stations to potential listeners and non-listeners. Though the researchers could hardly find significant differences between the two groups, the research is valuable, they say, because it shows that "media effects research" is only possible under certain preconditions. The findings in terms of media development practice included:
    • Media development organisations must monitor the content and its quality; this is the pre-condition to achieve impact.
    • Personal communication and radio were found to be the main sources of information for all participants.
    • Age has a significant influence on people's attitudes towards land rights and women, elderly people tending to be more conservative and favouring paternalistic traditions.
    • No significant differences in levels of knowledge, opinions, and activation levels were found between listeners and non-listeners of the radio station. An exception was getting involved in a social group that addresses land rights questions, which was significantly more often the case for listeners of the programmes.

    For audience research purposes, the following lessons were gained, among others:

    • In quasi-experimental designs, special attention has to be given to the selection of the treatment and the control group, making sure that the distribution of socio-demographic and other relevant variables is similar.
    • The stimulus should be clear and focused and of considerable duration in order to have an effect.
    • When measuring the impact of media, it is imperative to document the contents themselves, to match the questionnaire items with the content, and to make sure that the content has been distributed at all.

  3. The third study examines the social media strategies of two community radio stations (Granada Stéreo and Contacto 10) in Colombia. It uses a mixed methods design: digital analytics, a survey and focus groups. (A note on mixed methods approaches: "Each method should be designed to shed light on the data provided by the other methods, to allow for insights that go beyond what any single method can produce.") In short, the researchers found that community radio stations are successful in community building if they interact with their audiences rather than using social media as a channel to distribute press releases. Key findings related to media development practice were:
    • While Granada Stéreo generated a great deal of interaction with listeners on social media, Contacto 10 used social media primarily to distribute information.
    • Readiness by (potential) users to engage on social media was found to be highest for topics with a local reference, but there was also considerable interest in the peace process and the associated recent history.
    • Social media content was found by the participants to be most engaging if it focused on a wide variety of topics, both informative and entertaining. Photos and videos were especially popular.

    Lessons learned in terms of audience research practice included:

    • When drawing up a mixed methods design, the contribution of every method should be reflected to ensure that the data complement each other to deliver in-depth insights not possible with standalone methods.
    • Provisions should be made to ensure that samples across focus groups are comparable and all segments of the group intended to be reached are represented in the sample, e.g., quota plans or random sampling.
    • If remote management is necessary, when the principal investigators cannot be on location all the time, monitoring at key intervals in the research process can ensure that agreed-upon research designs are implemented in accordance with agreements laid down at the outset.

Chapter 5 summarises the findings and lessons learned, giving recommendations for future audience research in media development and identifying aspects that deserve further attention. For instance, in all cases, close cooperation between project partners and media development organisations was found to be imperative, because the partners knew the beneficiaries best and could tailor methods to suit their interests and behaviours. When it comes to developing instruments such as questionnaires or focus group guides, partners can provide important knowledge of the local media landscape, as well as impressions of usage patterns and information needs. Furthermore, it is important that common standards be defined together from the outset in order for audience research to live up to the necessary requirements in terms of validity and reliability. Among the other lessons discussed is the issue of intentional and unintentional effects of audience research. "Research has the potential to change people participating in it and it has the potential to change project partners and media associated with the research."

In exploring next steps on the audience research agenda, the lead author suggests that, depending on available resources, media development organisations should be encouraged to test and develop a standard repertoire of methods for audience research. Yet, "[o]nce an organization has a standard repertoire, it should not take this as the end point of its endeavors. Rather, the dynamic development of existing media and information ecologies will always challenge organizations to optimize existing tools and develop new ones more suited to what it is they want to find out. The methods tested in this volume should be pursued and adapted to other contexts themselves to find out whether they indeed are replicable and generalizable. Digital analytics presents a host of methods and metrics that can be harnessed for the good of media development....Quasi-experimental or experimental designs should increasingly become standard practice in audience research aimed at measuring impact....Tests using media examples seem to be a good way of testing [MIL] of children and teens, as the Palestinian study has shown."

A final reminder is that the methods are "only a means to an end, that end being measurement of the outcome and impact of media development activities for its beneficiaries, documenting medium- and long-term effects of activities, collecting proof that positive changes have happened, and identifying weakpoints of the activities." Media viability depends on giving audiences what they want and need - to ensuring that media outlets connect with their audiences in a meaningful way - after all.


DW Akademie website and "Connecting with audiences" (interview with Dennis Reineck), January 5 2018. Image credit: © DW / D. Reineck