Mile High United Way Evaluations

APA is the evaluator for four programs currently funded through Mile High United Way’s Social Innovation Fund (SIF). The SIF program leverages $1.8 million each year in federal funds to create up to $6.6 million in literacy investments in Colorado’s communities. In particular, Mile High United Way supports a series of innovative programs designed to improve literacy outcomes for children from birth to age eight. Each program is required to include a rigorous evaluation component that measures: 1) how consistently and effectively the programs are being implemented; and 2) the impacts of the program on child literacy and learning. Each evaluation plan is required to undergo significant external review, and must obtain approval from national experts in program evaluation design as well as approval of an independent Institutional Review Board (IRB).

The four programs which APA evaluates use different approaches to improve literacy outcomes, and each requires its own unique evaluation design:

This program provides training to preschool providers who serve primarily low-income, Hispanic communities. These providers tend to be family, friends, or neighbors of the children they serve and often have little or no training in early childhood literacy or education. PASO provides such training through a year-long course that includes classwork and in-home visits to preschool providers. APA conducted a formative evaluation of PASO in 2011 designed to inform program leaders and funders about aspects of the program that could be enhanced or improved. APA used a mix of surveys, interviews, observations, focus groups, and reviews of program data to offer a set of objective recommendations. PASO leaders accepted and implemented several recommended changes, such as offering providers the opportunity to earn a nationally-recognized credential for participating in the program. APA is now studying how the program is implemented across several different communities, and has developed an approach to use assessments to measure impacts on children and the providers who serve them, over a three year period.
This program provides tutoring services directly to struggling elementary school readers using a national reading curriculum and trained volunteers. APA worked closely with Reading Partners leaders as well as representatives of Mile High United Way to create an evaluation design that met not only our client’s needs but the rigorous design requirements of external evaluation reviewers. APA’s approach began with an implementation evaluation that allowed for an analysis of how consistently the program operated across multiple school and district sites. This work led to the development of an approved study that will use a statistical approach (propensity score matching) to examine program impacts on students over a three year period based on district reading assessments. The design of this impact study required considerable flexibility in order to accommodate the shifting needs of participating schools, districts, and students, as well as the Reading Partners program. The design will also maintain flexibility in order to recognize the rapidly changing district assessment environment.
This program seeks to reduce summer learning loss in children by providing five weeks of summer academic instruction in reading and math along with enrichment activities and field trips. In consultation with program leaders and Mile High United Way, APA developed an approved evaluation design which led to an analysis in 2013 to determine whether the program was being implemented in schools consistently, and to understand any challenges associated with maintaining program fidelity. This evaluation included site visits, interviews with key program and school district staff and leadership, as well as a review of existing program data and materials. The implementation evaluation also informed the design of APA’s impact evaluation of the program, which will use statistical analysis of data to understand how the program impacts student performance on district reading and math assessments.
APA is conducting an impact evaluation of the Jeffco Summer of Early Learning (JSEL) program.  JSEL is a summer program designed to provide academic support and enrichment to elementary school children in the Jefferson County School District in Colorado. The APA evaluation is part of a multi-year effort to understand the impact of the program on student achievement.