Researched and reported by:
Nancy, B. Reilly, M.A., Moving Forward Pittsburgh, Inc., Research Volunteer
March 16, 2016
A greater Pittsburgh, suburban school board’s citizens’ advisory committee is seeking to determine the feasibility of a return on investment (ROI) formula for internal and marketing use. After considering demographic, economic and student achievement data, two possible school district ROI formulas have been identified. Table 2 Sample ROI summarizes the two formulas. In the first formula, the lower the percentage of budget spent on instruction and the higher the SAT math score, the higher the percentage ROI. In the second formula, the return calculation would indicate a better “return index” with a smaller score, as a district would seek to spend the fewest dollars and the SAT math score to be as high as possible.
METHOD AND DISCUSSION
After some discussion, the types of variables measured included: student achievement, demographic and school district-related. Barbara Shafran, an ROI sub-committee volunteer and Executive Director of Moving Forward Pittsburgh, Inc., compiled statistics from national and state education resources into a spreadsheet and sent the spreadsheet to Nancy Reilly, a Moving Forward Pittsburgh, Inc. research volunteer familiar with statistical methods. The data was analyzed using SPSS (Statistical Package for the Social Sciences) software which generated Pearson correlational coefficients, 2-tailed, for each relationship among all variables.
The school districts selected for the study included 15 districts identified as peer school districts by members of the greater citizens’ advisory committee and included: Altoona, Bethel Park, Butler, East Penn, Fox Chapel, Hampton, Mt. Lebanon, North Allegheny, Pine Richland, Quaker Valley, South Fayette, Tredy Fin East, Upper Darby, Upper St. Clair and West Chester.
After the initial meta-correlation calculations for all variables compared to each other, it was found that there were strong correlations (all at .05 or .01 levels) between SAT and ACT scores, the SPP 2014-2015 scores and the Academic Scores for 2013-2104, so the SAT math score for 2013-2014 for each school district was used as a simplified measurement of Student Achievement.
A second, more refined analysis was done that focused on Student Achievement (as measured by SAT math score) and its correlation with variables measuring student cognition and motivation, poverty, family education level, and student teacher ratio, expenses per pupil, district size and district revenue sources. (See Table 1: Data Analysis.)
The strongest correlation in the more refined data analysis was percentage of households in a school district with a bachelor’s degree or higher (r=.895, p<.01) in relationship to high score on SAT math exam. This would suggest kids who score higher on the SAT math test come from homes in which at least one parent has a bachelor’s degree or higher, in other words, from households that value education. The average percentage from the 15 districts analyzed is 48.3, the highest percentage is 75.2 (Tredy Fin East) and the lowest percentage is 16.3 (Altoona). This suggests that when evaluating the performance of a school district, the education level of the local population is an important influence on student achievement.
The second strongest correlation from this second analysis was a strong, negative correlation between high achievement on the SAT math exam and the percentage of high school students receiving free lunches (r=-.855, p < .01). The negative relationship between percentage of households with reduced lunch (r=-817, p<.01) and percentage of households that are economically disadvantaged (r=-830, p<.01) and student achievement as measured by the SAT math exam all confirm that the higher the poverty level of a school district studied, the lower the SAT math exam score.
The third strongest correlation in the refined analysis was between high student achievement on the SAT math exam and the percentage of revenue that a school district receives from local taxes (r=.721, p<.01). It is worth noting that there is a strong, negative correlation between the percentage of state (r= -.699, p<.01) and federal (r= -.626, p<.05) revenues in a district’s budget and student achievement on the SAT math exam. This also suggests that poverty and income levels of a district are key indicators of student achievement since local revenues are generated from local real estate values. The poverty statistics for a school district should be considered when evaluating a school district’s performance.
The fourth strongest correlation with SAT math scores in the refined analysis were the negative correlations regarding measures of the cognitive abilities of a school district. The percentage of special education children (r=-.601, p<.05) and the percentage of kids with IEPs in a district (r=-555, p<.05) indicate that the higher the percentage of kids enrolled in special education or with IEPs, the lower the SAT math scores are for the district. In the study, the average percentage of kids enrolled in the districts studied is 12.65. The largest percentage was 17.94 (Altoona) and the smallest percentage was 6.04 (Hampton). The average percentage of students with IEPs is 14.46, the highest is 22.42 (Altoona) and the lowest is 9 (Hampton, North Allegheny, and South Fayette). When evaluating the performance of a school district, the number of kids in special education and who have IEPs are influential factors on student achievement.
The instructional expenditure per student is the fifth and final significant correlation (r=.571, p<.05)) to high SAT math exam scores in the study. The average percentage of a district’s budget spent on instruction across the 15 districts studied is 63%. The highest is 66% (Fox Chapel and North Allegheny) and the lowest is 58% (South Fayette). The average SAT math score is 554. The highest is 612 (Tredy Fin East), and the lowest is 478 (Upper Darby) with a standard deviation of 37.
As a result of the significance of expenditure per student related to average math SAT scores, Moving Forward Pittsburgh, Inc. proposes the creation of two measures of school district ROI that can be used to evaluate a school district’s performance.
First is the percentage ROI, calculated by taking a district’s score on the SAT math exam, dividing it by the percentage of its budget spent on instructional expenses per student and multiplying that by 10. (See Table 2: Sample ROI.) The lower the percentage of instructional budget and the higher the SAT math score, then the higher the percentage return on the district’s investment. The lower the percentage of budget spent on instruction and the higher the SAT math score, the higher the ROI.
Second is the ROI index that would use actual instructional expense dollars divided by SAT math score. (See Table 2: Sample ROI.) One example would be the return index for North Allegheny that in the study spent $8,568 per student, divided by 578 SAT math average which equals an index of 14.82. For Tredy Fin East, that would be $8,500 divided by 612 which equals 13.89. This return calculation would indicate a better “return index” with a smaller score, as a district would seek to have the dollars spent to be as little as possible and the SAT math score to be as high as possible.
In conclusion, while standardized tests like the PSSA are important measures of student achievement and school district performance, the education level of the local population, the poverty indicators, the percentage of local revenue in the budget, the cognitive abilities of the students and the amount of money spent on instruction per student are also factors that influence student achievement and should be considered when evaluating school district performance. The adoption of a statistically significant “return on investment” calculation would also enhance comparisons of school districts using public data that is meaningful to both educational and non-educational stakeholders.