A Practical Guidebook

Laura Schneider
Publication Date
June 1, 2014

DW Akademie

"While press freedom rankings determine national and international media policies, it is not always clear as to how objective these rankings really are."

In the first of a publication series, Edition DW Akademie, media researcher Laura Schneider uses data as well as interviews with experts to examine the methodology used in each of the 5 best known international freedom of the media indices. She takes a closer look at the organisations doing the ranking as well as how they conduct their respective ratings. Schneider also investigates the advantages and disadvantages of each index and points to aspects which could be improved. Schneider's central criticism is that the ratings aren't objective, often being carried out by a handful of academics or media experts who are mostly from Western countries. This inevitably results in a bias, she argues, especially as the majority of the ranking organisations fail to state their exact definition of media freedom. "The aim," says DW Akademie Director Christian Gramsch, "is to increase awareness of how these rankings are carried out so that the information they contain can be used fairly."

According to DW Akademie, every year, organisations publish their respective press freedom rankings, which usually causes "an outcry, especially from countries low on the list. While politicians hotly defend their media policies, opposition parties and NGOs seize the opportunity to criticize the government's approach." It is estimated that there are more than 100 organisations throughout the world that engage in some form of media freedom assessment, evaluation, or promotion. Schneider researched the contact details of more than 200 organisations - mostly non-governmental organisations (NGOs) that, according to their website, focus primarily on the support of media freedom. The majority are based in Eastern European countries, Asia, Africa, and Latin America and follow the mission of promoting free and independent media through activism, monitoring media freedom violations, evaluating media systems through indices and written reports, and defending and protecting journalists working in conflict zones and under repressive governments. While most of these groups have a national scope, there are also numerous organisations that promote media freedom internationally, regionally, or globally. They are either NGOs or intergovernmental organisations such as the United Nations (UN), the United Nations Educational, Scientific and Cultural Organization (UNESCO), the Organization of American States (OAS), the Organization for Security and Cooperation in Europe (OSCE), and the African Union (AU). These organisations deal with and promote media freedom in different ways, for example by producing written reports, organising workshops and training, and/or publishing statistical data and descriptions of physical, psychological, and legal attacks on journalists and media organisations.

The following 5 indices are the subject of the document, with an entire chapter devoted to each:

  1. The Freedom of the Press Index by Freedom House [1 expert scores 23 indicators; regional meetings determine final score]
  2. The Press Freedom Index by Reporters Without Borders [several experts score 87 indicators; staff scores and determines final score]
  3. The Media Sustainability Index by International Research & Exchanges Board (IREX) [10–14 experts score 40 indicators and discuss scores in panel; staff scores all indicators and determines final score]
  4. The African Media Barometer by the Friedrich Ebert Stiftung [following a panel discussion on 39 indicators, 10-12 experts score these, which determine final score]
  5. The Media Development Indicators by UNESCO [UNESCO Research team applies different methods to evaluate 50 indicators; results are presented in report]

These 5 measures are systematically introduced and analysed in the document through questions like these: Which organisations stand behind the respective measures? When were they initiated and why? What is the conceptual background of the different media freedom indices, and what goals do they have? How are they financed? Which methodologies are used to evaluate each country and, if applicable, create the ranking? What are the strengths and weaknesses of each international media freedom index? And which index should be used for which purpose? The guidebook systematically analyses the indices following the same structure for each. Further, in separate text boxes, it provides a brief overview of the most important aspects of each index and indicates for what purpose the index can and should be used. All additional information tools that are published by the different organisations are not only named but also analysed as to what they may be useful for. The extensive table following the 5 index chapters (beginning on page 40) summarises the information about all indices and thus allows for a direct comparison. Based on the table, the chapter "What to Keep in Mind When Using the Indices" summarises the most important aspects we should consider when using one or several of the indices.

Moreover, this guidebook gives an insight into the opinions and aspirations of the indices' authors. Short interviews with the editors of the 5 media freedom measures were conducted; their answers to the following questions are depicted in separate text boxes: What is the unique feature of your media freedom index? What does your media freedom index not supply? What would you change about your media freedom index if you had double the budget available?

The analysis of the 5 media freedom measures shows that all of them were originally designed for different purposes, have different conceptual backgrounds, and apply different methodologies. These differences are reflected in the findings: Diverse instruments produce diverse and sometimes even conflicting results. Therefore, it is not a simple or neutral matter to utilise and rely on one particular measure. This fact, Schneider argues, is not sufficiently taken into account when the indices' findings are used or received. This guidebook points out that the most crucial criticisms of the international media freedom indices are these of bias and subjectivity. With regard to the former: "Although the organizations, at least Freedom House, Reporters Without Borders, IREX and UNESCO claim that their measures are universal, there are always geo-ideological assumptions that underlie them: For example, the weighting of the different aspects (indicators or categories) determined by a few people from the same cultural background will always be highly normative and somewhat ideological, and thus intrinsically carries a cultural bias." The second fundamental criticism, subjectivity, is a difficulty inherent in almost all international comparisons: "[I]t is difficult, if not impossible, to conduct representative surveys in each country in order to avoid subjectivity."

In short, "due to natural limitations, no ideal international, let alone global, media freedom measure is possible. However, it is crucial to deal with the shortcomings of the existing assessments openly and transparently."

Editor's note: You may also be interested in DW Akademie's Media Freedom Navigator, which gives a visual overview of the diverse media freedom indices.


DW Akademie website, December 2 2016.