Harvard Family Research Project (HFRP)

Offered as part of the Harvard Family Research Project (HFRP)'s series of Out-of-School Time (OST) Evaluation Snapshots, this brief provides an overview of the strategy of using technology-focused programming in the nonschool hours to enhance the learning and development of children and youth in the United States. It culls information from the 315 programmes and evaluations in HFRP's Out-of-School Time Program Evaluation Bibliography that focus on using technology. Specifically, the authors examine 30 evaluation reports of 19 OST initiatives, exploring how these technology-focused programmes and initiatives are being evaluated and what implementation and outcomes lessons can be gleaned. In addition, this document illuminates some promising strategies for and challenges to incorporating technology into OST programmes.

The report begins by providing background on one key purpose of OST programmes: to support complementary learning, or various nonschool supports that work with schools and other community-based organisations. As detailed here, there are 4 primary approaches that OST programmes use to support this learning strategy:

1. Using software packages for educational remediation and skill building - for example, in the District of Columbia, the 21st Century Community Learning Centers use two software programmes, one for reading skills and one for math skills, during their summer programme.
2. Integrating technology and multimedia into project-based learning - for example, in the South Bay Project (a collaboration of school and community institutions providing after school services to youth in low-performing San Diego, California schools), youth create simple computer games, electronic portfolios, and multimedia Web pages to learn about computer programming.
3. Creating community technology centres - for example, the Intel Computer Clubhouse Network is an initiative aiming to create over 100 community technology "clubhouses" with 3-D imaging software, digital video recording and editing tools, and music recording and mixing equipment for use by underserved youth.
4. Providing technology-focused mentoring and career development opportunities in science, engineering, and technology-related fields.

The authors then identify a few formative questions common to the evaluation reports being considered in this research, including:

• What were the most successful aspects of using technology?
• What were the major challenges and barriers programmes faced when implementing technology?
• How did youth experience their interactions with technology (e.g., which aspects did they find useful and engaging and which aspects less so)?

The authors highlight one particular evaluation process, that of the San Jose Children's Discovery Museum's Discovery Youth (DY) programme. This process involved 5 rounds of surveys and questionnaires (involving 35 DY youth) to measuring changes in participants' learning and sentiments. Ten youth participated in focus groups, and a youth-focused group evaluation exercise was conducted. In addition, parents were interviewed and staff were surveyed. This example is consistent with the authors' observation that most of the evaluations used nonexperimental research designs that tend to draw on purposive sampling techniques such as case studies, ethnographic studies, participatory approaches, or data collection and reporting for accountability. Though a smaller set of evaluations included a quasi-experimental design, no studies in this review used an experimental design.

In that context, the authors note that the studies generally reported that stakeholders found the programme valuable and that the programme helped youth learn new skills and enhanced their development, including knowledge of and comfort with technology, improved social skills, and development of technological skills. However, there are challenges, which include:

• Technology has the power to engage youth, such as by creating a fun and playful learning environment, but it can be difficult to engage older youth in the programming, keep software-based programming "fresh", and find technology programming that is of interest to diverse groups of youth.
• Staffing for technology-focused programmes is challenging; in particular, it can be difficult to retain staff who are interested in and dedicated to engaging youth with technology.
• Technology-focused programmes require a commitment to infrastructure maintenance; common problems included maintaining internet connectivity and hardware performance and updating and repairing technological equipment.
• Programmes need to determine the balance among technology, enrichment, and academics - such as by getting the right "mix" of programming for youth and balancing technology-related educational goals with fun and enrichment.

The authors conclude that, "to fully understand the impact of the growing use of technology on OST programs and their participants, programs will need to continue to collect implementation information but ramp up their efforts to better understand outcomes. Future evaluations should also take into account the diversity of technology-focused programming in order to provide useable information about meeting implementation challenges and identifying the program characteristics that contribute to the effectiveness of these programs for youth."