Improving the Effectiveness of Assessment: Part 3 - When less means more

Improving the Effectiveness of Assessment: Part 3 - When less means more

By Bonnie A. Green, Ph.D. Owner and Principal Analyst at Illumin Analytics

Over the years, I have spoken about assessment at close to 2 dozen different universities and another dozen or so non-educational institutions. I have spoken to thousands of people on effective forms of assessing, particularly in areas of educational success or reducing recidivism. And yet, with each of these interactions, one thing has remained the same. People collect too much data, too often, and not the right kind in order to really understand what is going on.

I have already addressed the issue of what kind of data we should be looking at to get to better answers, sooner and faster in these two articles: Improving Effectiveness of Assessment: Treat Students Like Cars https://www.dhirubhai.net/pulse/improving-effectiveness-assessment-part-1-treat-students-bonnie-green/ and in Improving Effectiveness of Assessment: Learning from Pasteur https://www.dhirubhai.net/pulse/improving-effectiveness-assessment-part-2-learning-from-bonnie-green/ )

In this article, I want to help everyone to understand why we need to collect LESS data, LESS often, and from FEWER people. Instead, let the statistics work for you in estimating what is going on.

There are a few things we need to first acknowledge: most measures used with humans are simply not that perfect. There is a fair amount of error in each of those measures these errors come in several places:

·        Errors from the individual.

o  How does the person completing the assessment protocol feel – are they in a good mood, cranky, stressed? Are they motivated to do well? Could they be motivated to … cheat or lie? Do they have faith in the measure and how the results will be used?

·        Errors from the items themselves.

o  Wording, order, assumptions made about the test taker that may or may not be true.

·        Errors from how the results are analyzed.

o  How many times have you entered a number in wrong in a calculator? How many times have you transposed numbers? Even the best of analysts make mistakes.

·        Errors from how the results are interpreted and used.

o  The results of a test are only as good as the test is designed to meet its intended purpose. And yet, I regularly see people use test results outside of their intended purpose.  

Even in the best of situations, test results give you a glimpse of what is going on, but they never provide you with the kind of detailed, low error information that a physical scale could provide (e.g., weight, speed, distance traveled). We must know … what can evaluations and assessment tell us and what can’t they? I think once we accept that we are getting a good approximation, but it’s an approximation at best, the reasons why we don’t need to collect more data, from more people, more often become clearer.

All we need to do is collect enough data to get a glimpse of what the underlying truth is.

We can do this by:

·        Know what you are trying to understand.

·        Get enough information to have a sense of where you are.

·        Collect it from enough people so we have a sense of where everyone is.

·        Collect it often enough to track change, but not too often that we don’t have time to implement change.

·        If you don’t fully understand what psychometric tools can tell you or what statistics can and cannot reveal … obtain an expert to aid you along your way.

What we don’t need to do:

·        Never waste your time with pre-test post testing … participants change from point A to point B. Participants change with being tested.

o  Instead: If you really want to test the casual impact of a program there is only one way … random assignment. Get a group of volunteers and randomly assign them to one of two (or more conditions). Measure them at the end … and compare. Less data, less hassle, more useful information.

o  If you can’t randomly assign, use a baseline from a similar group who didn’t go through the program.

o  Capitalize on the benefits of quasi-experimental design.

·        Never just assume that self-reported improvement is worthwhile

o  Instead: Try to measure something that goes beyond self-report to truly capture change.

o  Use two or three different measures that are approaching the information differently, to get a fuller picture.

·        Never measure annually unless what you are measuring is changing that quickly.

o  Instead: Most human measures in areas of education only need to be taken every three years or so.

o  Don’t forget to measure mediating variables … variable that pre-tell the change in the variable you are most interested in seeing. (See article 1 for additional details https://www.dhirubhai.net/pulse/improving-effectiveness-assessment-part-1-treat-students-bonnie-green/).

·        Never collect data you don’t intend to use.

o  Instead: collect just enough data to give you a sense of what is going on.

o  Select measures that most people in your organization trust.

o  Come up with a time line that will work so you can actually make use of the data to improve programs, policies, or practices.

I know, various people in charge are insisting you measure every person every year. That’s part of the problem. We have people in positions of power who know just enough about assessment and evaluation to create havoc.

·        Identify what are your goals.

·        Understand what will cause you to reach those goals, what are the underlying mechanisms towards success?

·        Identify measures for those underlying mechanisms. Make sure they have some level of validity … that is they will provide you with accurate and useful information.

·        Think about how often makes the most sense to assess … that will give you enough time to use the data to make improvements. Though some things require more assessment, most areas of education and reducing recidivism could benefit from a 3 year cycle of assessment, with programmatic improvement.

·        And recognize, if you don’t have someone on your team with expertise in sampling, measurement, statistics, research design, and interpretation and visualization consider hiring an expert.

If you are interested in learning more about how to find solutions with less data, so you can move your goals forward, sooner, quicker, and cheaper, please look into Illumin Analytics at https://www.assessment-evaluation.com, or contact me.