American Institutes for Research

The American Institutes for Research (AIR), with funding from the United States Agency for International Development (USAID) through the Educational Quality Improvement Program 1 (EQUIP1), launched a comparative study of paper versus digital education data collection methodologies. This study used a deployment of Global Relief Technologies (GRT) digital data collection approach in Nicaragua to compare digital data collection methodology with the paper-based approach that was being used on the broader-based Child Friendly Schools (CFS) evaluation (a data collection process analysed simultaneously by the same research organisation). The GRT model used hand-held personal digital assistants (PDAs) for capturing observable, quantifiable data, a portable scanner, and portable satellite transmitter for capturing and communicating data.

The comparison considered process efficiencies, implementation flexibility, and cost. The paper-based collection required: on average, 20,000 photocopies, which also required 3,000 sharpened pencils. The large volume of paper required additional suitcases or waterproof containers to allow for local transport and security. Completed surveys were either shipped or hand-carried back to the US home office; then, they were manually catalogued, including hardcopies, electronic copies, and digital media into a tracking log; finally, hard copies of the surveys were shipped to a scanning service where they were scanned into a digital format and sent back in electronic data sets for analysis.

Digital data collection was done by PDAs that combined six devices in one: GPS for tagging exact locations of schools, audio recording, cellular, video, and digital photo capabilities, as well as automatic intake cataloguing of media files as they are created. With the digital collection through PDAs, the digitally-equipped data collectors shadowed data collectors deploying the paper-based instruments and sought to capture the same information. At the end of each day, all data captured through the PDAs waere uploaded using a portable satellite transmitter to GRT's Virtual Network Operations Center (VNOC) that automatically catalogued and organised data into a database by school and instrument. Additionally, student surveys that had been captured via paper were scanned into a dual-sided portable scanner and digitised. Once the student surveys had been scanned, they were also uploaded to the VNOC and automatically catalogued and organised into the database.

That study found that the digital system provided flexibility, including the ability to make immediate revisions. Easing the data entry and collection processes, as stated here, reduces the burden on data collectors and increases the accuracy of the data collected. The problems of protecting the condition of paper-based results for transport to the home office for scanning was not an issue with digital management. The digital data collection model required much less paper, but did not remove the need for paper completely. The paper that was used (e.g. surveys of students, teachers, and principals) was scanned and uploaded on a nightly basis. Also, questions could be changed or adapted onsite.

An advantage of digital questionnaires was that they could be set up to require an answer to each question before allowing the collector to ask subsequent questions, avoiding unanswered fields. Media collected, including photos and audio, were automatically tagged to specific correlating questions and/or instruments as part of the automatic cataloguing system and scanned at the end of each day.

The GRT system was found to be more conducive to quantifiable, close-ended questions than to supporting interviews or focus groups. Multiple choice or number formats worked best, for example: capturing the number of students in a classroom or observing conditions of school campus. The number of respondents was limited by the number of PDAs, so surveys of large numbers of respondents were not conducive to the PDA format and, at least in the current iteration, would still need to be captured via hard copy to be scanned/digitised in the field and uploaded.

The paper system took about 4 weeks from data collection through digitisation, while the digital system was almost immediate and provided early analysis for possible changes needed in the field process. Accuracy was nearly equivalent in both systems, whereas security was deemed to be easier in the digital system due to password protection of data.

Costs of labour were higher for the paper process, while equipment rental was higher for the digital process, though rental of cameras and voice recorders, part of the PDA platform, added costs to the paper system. If equipment is purchased and those costs spread over several projects then costs, as stated here, come down further for the digital process than the paper process. This, according to the study's conclusions, makes the cost/benefit ratio of digital very attractive compared to that of paper, particularly in light of lower vulnerability to loss, which would be at significant cost. The paper concludes that while paper is familiar and digital new, "as...information and communication technologies become cheaper and more readily available, it is important to look to the future and experiment with methods that while unfamiliar and require a learning curve, may ultimately prove to be a time-efficient and cost-effective methodology."

The Development Experience Clearinghouse (DEC) Express on May 15 2009.