Search

Evaluating Mass Media Health Communication: The Use of Evaluation Data to Improve Theory

How valuable is this shared knowledge to your work?
4
Average: 4 (2 votes)
Your rating: None
Author: 
Joël Lehmann

Publication Date

June 1, 2007

This 137-page thesis, submitted (and accepted, with honours) to the Master's programme at the University of Fribourg, is an exploration of ways to create a closer symbiosis between public health communication campaign theory and practice, for the mutual benefit of both. The author, Joël Lehmann, asserts that the evaluations could be a valuable contribution to improve campaign theory itself. In essence, Lehmann explores an "engineering" approach to campaign design, where theory and scientific findings are systematically used and adopted for practical problems - with evaluation research serving as an interface between science and practice. The core research question is: What is measured in the evaluation of health communication campaigns - and how?

In the first part of the paper, Lehmann outlines four conditions that need to be met to create coherent, comprehensive, relevant, and applicable communication campaign theory:

  1. Theory integration: Concepts and theoretical approaches from different relevant disciplines have to be integrated in a coherent, non-contradictory framework, where each concept is defined and operationalised consistently.
  2. Practice integration: Practitioners need to have a common point of reference, with goals and objectives operationalised consistently.
  3. Scientifically sound research: The methodology used for basic or applied research needs to meet scientific standards, and research must refer to the theory.
  4. Interaction of science and practice: Practice needs to operate according to theory-based strategic models, and both theorists and practitioners should design applied research in a way that it serves their respective purposes.


Lehmann then analytically breaks down public communication campaigns into three dimensions, in an attempt to synthesise various ways of analysis presented in campaign literature. The first dimension is the campaign infrastructure, which is composed of three variable groups: type and means of communication, audience variables, and institutional context. The second dimension is the intention of the campaigners, which captures (at least) the following five variables: proximity of campaign goals, type of desired outcome effects (cognitive, behavioural, biological, etc.), outcome levels addressed (individual, micro-, meso-, macro-social), number and explicitness of objectives, and clarity of campaign goals/objectives. The third dimension, theory, comprises the conceptual framework of campaigns.

Lehmann is particularly interested in this third dimension, which he details in an extended theoretical section that explores the difficulties and potentials of the science-practitioner interaction and offers a relatively detailed overview of some of the theories, models, strategies, and concepts used – or potentially useful - for mass media health communication.

  • Amongst the theories and hypotheses he outlines there are: the diffusion of innovations theory, the theory of reasoned action, the social learning theory, the theory of the spiral of silence, theories of emotional activation, consistency and reactance theories, the theory of inoculation, and the knowledge-gap hypothesis.
  • Next, he explores several models, including: stages of change models (includes the transtheoretical model, the precaution adaption process model, and the steps to behaviour change model), models that view action as the outcome of a cognitive process in which expected benefits are weighted against expected costs (e.g., the protection motivation theory, the subjective expected utility theory, and the theory of reasoned action/theory of planned behaviour), the hierarchy of effects model, the health belief model, the information motivation behavioural skills model, the behaviour ecological model, and the elaboration likelihood model.
  • Lehmann then describes several strategies, such as social marketing, the use of tailored messages, prevention marketing, and the entertainment-education strategy.
  • Perspectives detailed here are: the ecological perspective, the social network perspective, the supply and demand perspective, the agenda-setting perspective, the system-theory perspective, a focus on interpersonal communication, the dynamic-transactional approach, the framing perspective, the uses and gratifications perspective, social influence research, and persuasion research.
  • Lehmann concludes this section by identifying a number of concepts from various fields of social and behavioural sciences, including: priming, learning, and involvement.


In order to glean these various offerings into a framework that might be useful for the evaluation of mass media health campaigns, Lehmann identifies patterns and similarities across theories in their respective conceptual logic and causal or procedural pathways. As he details in the next portion of the thesis, he believes that all variables of each theory can be placed into one of the following 5 categories: (1) Facts that are not perceived by the individual, (2) objects of individual perception, (3) perception itself, (4) short-term neurocognitive structures or processes, and (5) long-term cognitive structures or processes. Most approaches do not use variables from all these categories. Lehmann presents a graphical model which is assumed to facilitate the understanding of similarities or differences in the logic of the various approaches.

Next, Lehmann explores the meaning and role of evaluation, and defines many of the terms used in evaluation research. He details various evaluation methodologies, exploring their application to the evaluation of mass media health campaigns - all the while highlighting why this type of research has to be different from "regular" evaluations. Special challenges associated with evaluating mass media health campaigns are examined – for example, the difficulty of realising experimental designs, the two steps of exposure and effect, and the challenges related to having narrowly defined "target audiences".

The second (shorter) part of the thesis begins with an analysis of methodologies and potential shortcomings of 32 recent unpublished evaluations of mass media health communication campaigns in over 16 countries. This endeavour is based on Lehmann's conviction that "meta-analysis", which involves studying prior studies, can prove effective in constructing coherent, comprehensive communication campaign theory. This approach will only succeed, however, if the same constructs are consistently measured across evaluations, and if each evaluation meets minimum standards of validity. Thus, Lehmann places special emphasis here on the effort to apply scientific standards to evaluation research, while keeping it useful for programme planners and policymakers.

To demonstrate, this approach, Lehmann analyses a sample of 33 evaluation reports for mass media health communication campaigns conducted in 22 different countries; 32 of these reports had not been published in a scientific journal. The main questions for the empirical research that Lehmann undertook were:

  1. What theoretical concepts or models do the evaluations and/or the evaluated campaigns rely on?
  2. What are typical campaign goals, what are outcomes measured in the evaluation, and how do the goals relate to the outcome measures?
  3. With what methods and designs are the concepts measured and operationalised?
  4. How much attention is paid to questions of validity?


An excerpt from Lehmann's thesis follows:
"The findings suggest that theory is not widely and consequently used to inform health mass communication campaigns or their evaluations – with notable exceptions. While there is a large number of outcomes measured, they seem to be taken out of theoretical context. Neither the campaign goals nor the evaluation measures reflect the large number of possible communication strategies that the various communication or behavior-change models and theories imply. Unintended campaign effects were mostly ignored. In very few cases the campaign designers or evaluators make use of an effects model or program logic model. This is one of the areas where I see the possibility of an important improvement.

The methodology of campaign evaluation is relatively homogenous across the 33 cases in regards to data collection method. Standardized questionnaires are the dominating data collection instrument. Non-reactive observation or tracking methods are very rare. A surprising two thirds of the evaluations did not use multivariate analysis, and the reliance on self-reports raises questions of reliability.

One of the important findings was that reliability was rarely assessed, even though recall and recognition measurements, as well as self-reported behavior, are known to be unreliable. In addition to that, there is room for improvement when it comes to validity: 24 of the evaluations left me with the impression that covariates were not controlled thoroughly and almost half of the campaigns did not control sufficiently for extraneous historical events. The question of timing was rarely discussed, and it seems that an important proportion of the evaluations measured short-term effects only. Social desirability, cuing selectivity or response-bias and missing data are further threats that often go unmentioned.

An evaluation conducted in Zambia, two in South Africa, one in Germany, one in Nigeria, one in Cameroon and one in Canada are mentioned as promising examples, even though they all are not completely apt to be used for theory improvement. With a relatively small additional effort, these evaluations could have provided evidence for the improvement of theory. In other cases, more important changes in the evaluation design and campaign planning would have to be made in order to make them useful for theory.

Valid data from evaluations and a consequential use of sophisticated theory is not only desirable from the point of view of theoreticians. Improved campaign effectiveness and a meaningful decision-base for policy makers also depend on it. In terms of improved theory and practice, and better theory-practice integration, a lot of work remains to be done. On one hand, this work consists in building comprehensive and strong theory, and in consistently translating it to applicable effect models and guidelines. On the other hand, it is the use of such models by practitioners. And last – but not least – valid and theory based evaluation could serve as an important intermediary between the theorist and the practitioner."

To request a copy of this thesis, please contact the author at the email address listed below.

Contact Information: 
Source: 

Emails from Joël Lehmann to The Communication Initiative on June 29 2008, July 9 2008, and August 3 2008.

How valuable is this shared knowledge to your work?
4
Average: 4 (2 votes)
Your rating: None

The Communication Initiative Network and Partnership

Covers all major development issues. Convenes the communication/media development, social/behavioural change community with a social network of 85,000 - please join. . When registered you can search the network to identify relevant support and connections.. Knowledge sharing - 35,000 summaries, 1 million users pa. Critical peer review - ratings, comments, dialogue. Advocacy for this field. Strategic direction/funding by 20 Partners. To discuss partnership please contact Warren