What's in a Letter Grade? 2021-22 NC School Performance

Introduction

I have been a teacher at an "A" school and I have been a teacher at a "D" school and there is one thing I know to be true...there is amazing teaching and learning occurring at ALL of these schools, despite the "letter grade" that defines them.

In the 2011-12 school year, the state of North Carolina began using the Education Value-Added Assessment System (EVAAS) data as part of the North Carolina Educator Evaluation System for teachers and school administrators. In 2012-13, the state began reporting on EVAAS data, using it as an accountability model to determine teacher effectiveness and school performance. Part of this EVAAS accountability model is School Performance Scores. "A" through "F" letter grades are assigned to all public schools in the state of North Carolina. You can read more about EVAAS in NC here.

I will never forget sitting in one of my very first staff meetings during my first year of teaching in 2013, learning that my school had been labeled a "D". As a person who had made straight A's practically my whole life, a "D" was just...DEVASTATING. I had worked tirelessly to do everything I possibly could for my students. I worked with an amazing team of innovative teachers who stayed late, arrived early, and always put students first. My students had worked incredibly hard and made me proud! My principal had been the backbone of support the entire journey. I just knew the score had to be wrong...

The next year, teachers in our school were encouraged to go visit and observe other schools in the district who had been rated higher than ours. I took this very seriously, as I am always one who seeks to grow and learn from others. During those visits, there was a noticeable difference though. Our communities were different. The school where I taught had more diverse economic backgrounds, more minority students, and more students with disabilities. I didn't understand why my school would be "compared" to these schools or even graded using the same measurement, but this was a new measurement system, so I wanted to give it time. It's now 2022 and the data seems to have the same trend. A higher Percent of Economically Disadvantaged Students (EDS) tends to suggest a lower school performance grade. Just take a look at the trend of the data below...

As the Percent Economically Disadvantaged Students (EDS) increases, the school performance grades tend to decrease.

How does EVAAS determine School Performance Grades (A-F)?

There are two major indicators: 80% school achievement (SA) and 20% school growth (SG). The formula for calculating School Performance Grades is as follows:

School Achievement(.8) + School Growth(.2) = School Performance Score (SPS)

For example, from the chart above, you can see that A L Stanback Middle scored a 97 for the school growth score (SG), but only 44.5 for their school achievement score (SA), so:

44.5(.8) + 97(.2) = 55(SPS) which was a "C" overall

School Achievement (SA) scores are calculated using an undisclosed algorithm including data from: Academic Assessment Scores (Reading and Mathematics EOG/EOC), English Learner Progress Score, Four-Year Cohort Graduation Rate Score, Biology EOC Score, The ACT/WorkKeys Assessments Score, and Passing NC Math 3 Course Score. School Growth (SG) is determined based on an index that provides a growth score and status:

• Exceeds Growth (>= 2.00)
• Meets Growth (1.99 to -2.00)
• Does Not Meet Growth (< -2.00)

What I Learned About School Letter Grades...

• The dashboard pictured above, 2021-22 NC Performance Grades, can be manipulated and provides a variety of filters. I was particularly interested in the breakdown according to school grade levels (elementary, middle, high, etc.) and the Percent Economically Disadvantaged Students (EDS).
• Performance Grades for ALL schools totaled: 145 "A" schools, 446 "B" schools, 907 "C" schools, 833 "D" schools, and 264 "F" Schools. Schools with insufficient data (I) and alternative schools (ALT) were not given scores.
• Early Colleges account for approximately 78/145 of the "A" schools in the state of NC.
• Correlation: There is a strong negative correlation (Pearson's Correlation Coefficient, r = 0.7167) between the variables: Percent Economically Disadvantaged Students and School Performance Scores.
• 42.3% (1097/2595) of schools were were rated "D or F". In particular, 246/473 middle schools are labeled "D or F", revealing that more than 50% of the middle schools are not effectively growing and preparing students to achieve on-grade-level, based on the current grading system.
• 1040 Schools Met Expected Growth, 757 Schools Did Not Meet Expected Growth, and 735 Schools Exceeded Expected Growth. Of those, 204 Schools EXCEEDED, but still scored a "D or F," based on achievement levels which raises the question: is something wrong with the grading system, or does something just need to change drastically to improve teaching and learning to increase achievement (students on-grade-level)?

The Problem

Teacher and administrator effectiveness and School Performance are being measured based on a system that reveals strong correlation to economic factors. Schools are being graded using a system that yields a high rate of failure based on educational growth and achievement. Over the years, critics have argued that the use of letter grades do not fully quantify the picture of student success and school quality.

Data

The dataset used in this study was the 2021-22 School Performance Grades, which is open-source from the NC Department of Instruction (NCDPI) website.

I imported the data into google sheets and added some variables of interest to my dataset, such as distinction for "Early Colleges" and I grouped all schools that were elementary age ranges (PK-05), intermediate (04-06), elementary to middle (PK-08), elementary to high (PK-12), middle (06-08), middle to high (06-12) and high (09-12). I also used the REGEXMATCH function in Google Sheets to test whether the name of the school matched the title of being an "Early College."

I also used matching to merge the original dataset with a dataset that included the addresses for each school. Geocoding in Google Sheets allowed me to find the Latitudes and Longitudes of every school to put them on the interactive map in Tableau.

Analysis

The percentages for letter grades are alarming and attention needs to be drawn to the fact that 42.2% of NC Public Schools eligible to receive a letter grade scored a "D" or "F", based on the current EVAAS accountability model.

• 145/2595=5.6% scored an "A"
• 446/2595=17.2% scored a "B"
• 907/2595=35% scored a "C"
• 833/2595=32% scored a "D"
• 264/2595=10.2% scored an F

Out of the 145 schools that scored an "A", filtering in Tableau allows us to determine that only 67 of those schools are "traditional" public schools. Early Colleges make up 78 of the 145 "A" schools, because Early colleges are typically schools with high-achieving students.

Further analysis, using a query in SQL, allowed me to determine the top performing schools who scored an "A"AND have higher than 40% Economically Disadvantaged Students.

As you can see, the majority (10/12) of these "A" schools with >40% Economically Disadvantaged Students are "Early Colleges" (including Cleveland ECHS, which was not identified because it did not include the word "Early").

Schools to watch, or perhaps "study" may be a better term, would be Beaver Dam Elementary and Moravian Falls Elementary School. What programs, teaching strategies, community involvement, etc. helped these schools of higher Percent EDS perform better than other schools?

There was NO SCHOOL that had greater than 50% EDS that scored an "A". Thus, we will study the "B" schools, using a slightly different SQL query.

The results for 2021-22 schools that rated a "B" AND had >50% Economically Disadvantaged students were:

Again, the schools appearing on this list would be great schools to study more in depth to discover the strategies, programs, and other factors that are helping them to perform higher than the underperforming schools of similar demographic. In particular, we have ONE middle school that has appeared from this query - Ashe County Middle School has approximately 52% EDS and was rated a B! This would be a school of interest to me if I were a middle school leader who is interested in improving the achievement and growth at my school.

Due to the fact that many middle schools in NC are underperforming according to the current grading system, I chose to dig deeper into the school growth by writing a query that would search for the middle schools that are "Exceeding" growth AND have a high percentage of Economically Disadvantaged Students (>50%). I ordered these descending, according to % EDS, to discover the top 10 schools with a high % EDS "Exceeding" growth:

The results from the query reveal many insights - all of these schools exceeded growth, but scored a "D" or "F" overall rating. Exceeded growth means that the students, on average, grow academically more than expected. Yet, the achievement score which determines whether a student knows what he or she is supposed to know at their particular grade level ranges from 26-46.6, indicating that only about 1/4-1/2 of the middle school students who are "Exceeding" growth AND have a high percentage of Economically Disadvantaged Students (>50%) are achieving on grade-level. Thus, I decided to do one more query, based on the top school achievement scores for schools with a high percentage of Economically Disadvantaged Students. This was the query I wrote to search for schools with a School Achievement score of >60 and a Percent EDS >50%:

The results were as follows:

From the schools with >50% EDS, the highest school achievement scores were from East Robeson Primary, Taylorsville Elementary, and Freedom Trail Elementary. These schools would be great schools to collaborate with and to discover insights of how they are helping their students grow AND achieve. Sadly, most of these schools still received school performance grades of a "C", which just seems wrong considering the amount of growth demonstrated by these students! As you can see from the Tableau Public dashboard screenshot below, when I filtered the schools that exceeded growth, but still scored a "D" there were 125 elementary schools, 47 middle schools, and only 2 high schools. There seems to be a high concentration of schools with D's in the Charlotte area. The low number of high schools scoring D's and F's raises more questions about how high schools measure achievement differently from elementary and middle schools, but this will have to be explored more at a later time!

Conclusion

There is a strong correlation between Percent Economically Disadvantaged Students and the school performance grade. The Middle Schools in NC should be the highest priority for improvement over the next few years. My suggestions would be increased educational resource funding and bonuses for retaining and hiring highly qualified teachers - in particular for those schools with a percent EDS >50%. In addition, further studies should be conducted to determine an improved accountability model for determining teacher effectiveness and school performance - a model that accounts for more variables that are out of the control of school leaders, teachers, and students. These letter grades are only creating a greater divide and are not a true picture of the amazing teaching and learning occurring at all schools, particularly those with more economic disadvantages. Perhaps a model that more equally measures growth AND achievement (50%/50%), rather than measuring achievement (80%) and growth (20%).

My recommendation for schools RIGHT NOW, would be to begin studying schools with similar demographic to your own that are demonstrating high growth and high achievement. Collaborate with these school leaders and analyze the programs, teaching, and learning opportunities that are helping these schools to outperform other schools with similar demographics. Take action on what you can control, because many of the factors that are currently determining your school letter grades are out of your control. My short synopsis here does not even dive into the subgroup data that could be explored related to this topic, and I look forward to studying these more in the future.

I am a math educator and data analyst who is continually trying to learn and expand my data skills, so if you have any?suggestions,?questions, or other?feedback?to provide, please feel free to?message me. Also,?I'm looking for a job?in data science, so if you know of any opportunities please let me know - I'd be open to explore! Feel free to?connect with me?on?LinkedIn, and be on the lookout for more data projects from me in the future! :)