Improving the Effectiveness of Assessment: Part 1 – treat students like cars!

There are many approaches to assessing educational success. Some of them are beneficial in that they provide timely information in a cost effective manner without negatively impacting student success. Too many, particularly when dealing with vulnerable populations amount to little more than saying … Student A is successful and student B is not successful. This is like saying Car A is running; Car B is not running. Clearly, this kind of assessment isn’t enough to help us in improving student success.

Take, for example, a college level retention program with a list of identified measures. To help target improved college level retention a team of very smart, very caring individuals identified a series of measures like pass rates in classes, annual retention rates, by demographic, and graduate rates (4, 5, and 6 year).

Nowhere in the proposed assessment protocol are data that can answer the question … what causes some students to graduate and others not to.

From looking at these measures, this committee just very well may be able to determine that students with low grades are less likely to graduate than students with high grades! Think about how much time and effort is going into getting the data, analyzing and reporting on it just to be able to say, students with higher grades are more likely to graduate.

When I pressed the issue with members of this committee, they said that they also had an exit interview for students leaving the university. This measure had 5 options for students to select on why they left the university, none of the options were particularly insightful. Two options contained information already available to the committee without having to ask students (i.e., academic dismissal; conduct dismissal). Though most of the students never completed the exit interview … the committee did find that students who did complete the exit interview gave one answer more than others for not staying in college. Wait for it … yes, they left college because they couldn’t afford to pay for it.

Hundreds of hours went into the report that came to these two conclusions: students left school because they couldn’t afford it and students with higher grades are more likely to graduate. But what new information does this tell us? And how can we use this information to come up with a plan of attack for improvement? What is CAUSING their low grades? What is causing them to not be retained? If we do not know what is at the root of student failure, or their successes, how can we address it?

In my lab, we work only on variables for which we can change. (I can’t change race and gender … sure I measure them, but they aren’t my focus). So, what do we hypothesize impacts student success? Firstly, it’s an extensive 6 domain theory. Three of the domains are at the organizational level (things like effective pedagogy and curricular implementation and leadership) and three domains are what student brings (like their beliefs, cognition, and behavior). Each domain has 5 to 20 variables. Each variable is changeable. So, if we can figure out what is weak in an organization (e.g., lack of articulated vision or lack of understanding of how decisions are made), we can target that area for improvement, and the organization’s improvement will guide the students toward success. We can also take a look at students, individually. For example, students who lack the ability to regulate their emotion or manage their stress tend to not do as well in school as students who do. If through a measurement you find students lacking in these skills, you can target an intervention (e.g., Mindfulness activities) that could help students, taking them towards a path of success.

Let me walk you through a scenario. In the theory of student success, we hypothesis that students who seek help (e.g., go to tutoring services, meet with mentors or faculty members, make use of counseling services) are more likely to be successful. In fact, we have found that they have higher grades than students who are low in help seeking behaviors. So, if we administered this three item measure to students and find them to be low in help seeking behavior we can utilize an existing peer mentoring program to engage in targeted improvement. By training mentors to guide their mentees towards help seeking behaviors, we could see students make use of the on-campus resources to improve their grades.

In the example at the beginning of this article, the school has a mentoring program, but the mentors are not targeting any specific variables. Mentors are trained in how to talk with students and how to be active listeners. All other information they are to provide to their mentees is based on answering questions their mentees ask them, or what they feel would help them to be successful. Imagine if our auto mechanic just tried what he/she felt might work in fixing our car? Yet, with a change in assessment, we can identify where are the struggling students in need of training. Once that’s identified … train the mentors, and help guide the students. Think about how truly small, no cost change we are talking about. And here’s the beautiful thing, most assessments do not need to be administered every year. Depending upon what you are measuring, you can take a measurement once every three years. (You can find more information about this in the article on Improving the Effectiveness of Assessment: When less means more.)

In another example, we have found that students with a professional future sense of self (they can see themselves in their future career) do better in school, particularly when it comes to taking challenging classes. So, creating a program that helps students develop a professional future sense of self would help assure student success. This is why pedagogical practices like course embedded undergraduate research, service learning, authentic class assignments, or job shadowing/internships help. As students engage in and with activities and people associated with their professional future, it increases how they see themselves, which improves their academic self-efficacy (belief that they can be academically successful), their grit (sustaining effort towards a long term goal, even in the face of failure), and their ability to set appropriate goals.

Nationwide, we are missing a theoretical framework for understanding student success. So instead of assessing how to assure student success, we are assessing students to classify them into successful or not successful. Thus we are missing out on incredibly important information on how to make improvements for students who are struggling.

When a mechanic comes to a car that isn’t functioning, he or she goes through a series of diagnostic measures. Some are informal, “Tell me what your car was doing before it stopped running.” Others are more formal, like looking at the car’s on-board diagnostic computer. Their measures are keenly focused on not just determining if the car is or isn’t running well, but instead … what is the underlying cause of the car’s poor performance.

The time has come in education for us to stop just saying … these students are “successful” and these students are not based on the tests I have given. As with this as our guide, we stop teaching students and start teaching tests. (You can find more information about this in the article on Improving the Effectiveness of Assessment: Campbell’s Law or Stop Teaching to the Test)

We need to define what is underlying student success. We must get a keener insight into how we can guide, coach, scaffold, and mentor success in ALL students. To do this, we need to have a theoretical framework that identifies variables associated with student success. (You can find out more information about this in the article: Improving the Effectiveness of Assessment: Learning from Pasteur https://www.dhirubhai.net/pulse/improving-effectiveness-assessment-part-2-learning-from-bonnie-green/?published=t) That framework becomes the basis of a more effective assessment protocol. Come on … we can do this with cars. Don’t our student deserve at least that? 

[For more information on the Theory of the Science of Success, please see https://osf.io/tk7dp/ for free details and available measures. You can also reach out to www.assessment-evaluation.com for additional information on how to make the most of your assessment and evaluation protocol.]