SHORT TITLE: HIGH QUALITY SES/21ST CCLC DEMONSTRATION PROJECT EVALUATION

Investigator: Deborah Lowe Vandell
Researchers: Pilar O'Cadiz, Valerie Hall, Andrea Karsh
Funding: US Department of Education
Dates: October 2008 - September 2011 

Description 

This project is a 3-year evaluation of a blended model of federally funded supplemental educational services (SES) and 21st Century Community Learning Center (CCLC) afterschool programming being implemented in a local school district, in partnership with a community based organization. The model includes both afterschool enrichment activities and an intensive, research-based instructional program aimed at improving student performance in English/language arts and math. Through the collection and analysis of survey data, program attendance data, achievement test scores, program observations, focus groups and individual interviews, the project seeks to identify best practices related to the implementation of the blended model of SES and afterschool programming as well as the effectiveness of the model in recruiting and retaining students to the SES program and the impact of program participation on student achievement in English/language arts and math. 

Both quantitative and qualitative measures and methods are used to evaluate program performance:

• Student enrollment and attendance in the SES program


• Student completion of SES sessions (maximum = 34 sessions)


• Standardized Test Scores (Language Arts and Math) to measure academic improvement


• Student surveys (pre- and post-participation) focusing on student outcomes


• Parent surveys (pre- and post-participation) focusing on program choice and satisfaction


• Interviews with parents, program administrators and staff, and school principals


• Observations of program activities

1. Work Habits
   The Work Habits measure is a 6-item child self-report utilizing a 5-point response scale (1 = not at all true, 5 = really true). It is an adaptation of the Work Habits scale of the Mock Report Card (Pierce, Hamm, & Vandell, 1999. The items were reworded for the 6-item child self-report and the response scale was modified. Cronbach's coefficient alpha for the student report in the Study of Promising After-School Programs was .76. Walker & Arbreton (2004) found that similar student reports of work habits were positively associated with academic performance and school attendance.


2. Reading/English Efficacy & Math Efficacy
   The Reading and Math Efficacy subscales are adapted from measures developed by Eccles and colleagues with excellent psychometric properties (Eccles et al., 1993). The adapted measure consists of four items about reading/English and four items about math. The items are related competence belief (students' perception of how good they are at reading/English and math, their expectation for success in each subject, and their sense of efficacy about learning new material in each academic domain), as well as students' subjective task value (i.e., how important it is for them to be good at reading/English and math) (Eccles et al. 1993). The response scale was modified from the original 7-point Likert scale to a 4-point scale (1 = not at all true, 4 = really true). Jacob et al. (2002) report Cronbach's alphas of 0.83 for reading/literacy and .84 for math.


3. Peer Support
   The Kids in My Class survey is a compilation of 18 items taken from three questionnaires developed by Gary Ladd and his colleagues (1996) and used in the NICHD Study of Early Child Care and Youth Development. It measures the child's social support received from peers at school, engagement in physical and verbal bullying behaviors with school classmates, and perceived victimization. The child answers on a 5-point Likert scale (1=never, 5=always). Cronbach's alphas were high for social support from peer items (.92) and moderate for perceived victimization (.81) and engagement in bullying behaviors (.78).


4. After-School Environment Scale
   The After-School Environment Scale (ASES; Rosenthal and Vandell, 1996) is a measure of the psychosocial climate in after-school programs. It contains 36 items on three subscales: Emotional Support, Autonomy/Privacy, and Peer Affiliation. For this study we used a shortened version used in the Study of Promising After-School Programs. The response option is a 4-point Likert scale (1=never, 4=always). Cronbach's alphas were .88 for Emotional Support, .77 for Autonomy/Privacy, and .83 for Peer Affiliation.