What Is The Future of Testing?

Should concerns around legitimacy of SAT scores affect how American universities use them? The SAT Has Been Hacked

I was asked to answer the question above on the website Quora.com

***************************************************************************

I have been monitoring the reaction to the Reuters story over the last two weeks since it was published (March 28, 2016). I think it is too early to say what the ramifications of this story will be, but I will try to give some reactions by educators and then some predictions of how things may or may not change on the test and on the part of colleges and universities.

Before doing so, however, in the interest of full disclosure, I need to mention I talked several times over the past year with two of the authors of the Reuters articles about topics related to students and education in China. In addition, I raised the issue of testing in China in an article I wrote last summer for NACAC’s Journal of Admission.

The Reaction From Educators

While there has not been universal consensus among the educators who have posted remarks about these stories on closed groups across the web, many have placed blame for the problems with the SAT on squarely on the College Board and its policy of recycling old questions on subsequent exams. They argue that in an electronic age all old questions will be recorded and given as study aids by those companies that gather all such information within minutes after each administration of the test.  The College Board has been aware for years that old questions were being gathered by these companies in China (and other places too).  Given that test security has been compromised over many years, The College Board responded.

The Reaction From The College Board

Shortly after the Reuters’ stories were posted, The College Board responded with a “Note”:

The College Board is not perfect; we did our best with the information we had at the time. We can and should do more. In fact, we have recently taken more aggressive actions to secure the exam internationally, including canceling an administration in China. We continue to balance widening access with protecting the exam. We recently refused entry to the first administration of the redesigned SAT to a number of high-risk registrants — many of whom make a living violating security protocols. In the future, we will need to do even more.

We have been working with Reuters on these stories since summer 2015. During that period, we’ve made College Board leadership available for multiple interviews and shared more details about test security than we’ve provided any other news organization. We’ve been transparent with Reuters in an effort to make sure its readers better understand all that we do to protect the integrity of the SAT internationally and ensure the validity of the test scores we report. In the spirit of full transparency, we are sharing here the detailed letter we sent to Reuters with the facts of our decision-making.

Even as we recognize where we need to do more, we are so proud of the new SAT and its impact on students. This is a time of remarkable, positive change for our century-old organization. In early March, more than 460,000 students took the redesigned SAT, which better reflects what students are learning in the classroom and connects them with distinct opportunities such as free practice tools and college application fee waivers. 

The “Note” admits that there have been problems and that their are efforts to address some of the security issues related to test administration. What the Note does not address, and what has many educators worried, is that the College Board does not directly address the use of old questions on new versions of the test.

 What This Means For Future Test Takers And For Colleges And Universities

One reason the College Board did not address this question may have to do with the fact that the most recent testing, in March, was the roll out of the new and updated SAT. The first test administration did have higher security and anyone not of college age was prevented from taking it (whether than can continue to do this may be something courts will decide). The effort to keep out professionals who are part of test preparation companies will not, however, keep these companies from gathering all the questions on the test. Some test prep companies get students to memorize questions and once the test is over all the questions are gathered and are then used to prepare the next set of test takers.

What is important about the Note is not something that is included; rather, it is what is not addressed that is most telling. There is nothing in the Note that would lead me to believe that old questions will no longer get recycled on subsequent tests. As long as old questions are recycled there will be an unfair advantage to those who have had prior access to these questions.

To come up with questions that allow The College Board to distribute scores across a wide range of student ability keeps many psychometricians employed by the Educational Testing Service (ETS). For those not familiar with the test design, the SAT yields responses such that average scores for students are approximately 500 out of 800 on each section of the test. For the test to produce these results each year questions have to be designed so that there is a reliable bell curve on responses by students. There have to be, to put it simply, some questions that are easy and others that are hard. For example, there are some questions that the vast majority of test takers will get right and there are some questions that are designed for almost all students to get wrong. Test designers create a test in which the range of responses always leads to a mean of close to 500. This requires test questions to be vetted before they are used as a part of the official test. The students taking the official SAT are not told which of the sections of the test is experimental and therefore are not scored for the students taking that particular SAT on that particular date. The experimental sections provide the data needed on the test questions. Once the psychometricians have measured the experimental questions in terms of how many will answer the question correctly, these questions can then be used as part of a “real” test given at some future date. Those companies and individuals who have gathered all the questions from the SAT on a particular date including the experimental section will then give students who have signed up for test prep access to these questions when they sign up for their test prep services.  The Reuters stories supports what I have just written. It also raises the question of whether students’ SAT scores should be used for admission purposes.

Colleges and Universities need to trust the SAT. If there is strong evidence that the test security has been so compromised that scores may not be accurate for many individual students, then the whole reason for requiring standardized tests comes into question. It is almost impossible, however, to determine which student has gained an unfair advantage by having access to old questions prior to the test date the student choose to sit.

If getting access to questions that have appeared on previous tests this is “cheating”, then it is a far different kind than bringing in answers hidden in a calculator or some other device. The “cheating” that occurs when old questions are recycled occurs before the actual test is administered and therefore cannot be observed or determined by proctors or anyone else. It is, to sum up, almost impossible to prove if a student has benefitted from having had prior access. And this is when things get ethically murky. Some admission readers assume, at the outset, that certain students’ scores (those from China mostly) are not accurate because they have had access to questions. To assume this, however, is not the way we in the US say things should be done to individuals. Students should be innocent until proven guilty. In addition, a reader who assumes a student had an unfair advantage should permit the student to mount a defense. Students do not even know that their scores are being, if not dismissed, then at least looked upon with suspicion by admission officers. I have had a student write about his getting discriminated against for just these reasons. 

The SAT test does not simply give a score of an individual student; in addition, it demonstrates where the student stands in percentiles compared to all other test takers. Without a reliable score distribution the test would not work well for comparing students, not only on an individual test but over multiple tests a student might take. The SAT is also used by ranking businesses such as the US News to measure the selectivity and reputation of individual universities. The test scores must bear comparison to the test scores over previous and subsequent years for them to be useful. For example, if the average score of students who enroll at a particular college increases by 20 points over 5 years the school’s reputation will go up. This is only true, however, it the test scores can be usefully compared. In addition secondary schools as well as colleges are often measured by the average SAT scores of their students. The use of SAT scores to measure the educational quality of secondary school student will increase in the near future because the new version of the SAT is tied to the Common Core. As a result, SAT scores are now mandatory for all students in some States that use the Common Core. If, however, the scores are be unreliable then parents and educators who are anti-testing may mount lawsuits to halt the use of these tests.

I have been outlining why it should be in the best interest of the College Board to discontinue the use of old questions on the SAT because I wanted to build a case that would prove, if not beyond a reasonable doubt, then at least one worthy of vigorous discussion. Now however, I need to suggest why the use of old questions will not change unless the leadership of the College Board feels they must build all new tests for each future date due to pressure from colleges and the public at large, (and, in a less likely scenario, by the courts.)

The College Board is well aware of the security issues and acknowledges them in their Note, but they are not in a position to create new tests for each subsequent administration. The reason that old questions may (and, in my opinion, almost certainly will) continue to appear is a matter of time and cost. The process of creating new questions that fit with the need to create score outcomes that match closely with previous tests, the need to use experimental questions to assure the questions perform in the way they need to in the aggregate, and the need to do this over multiple test dates each year is, as it stands now, an overwhelming task. While the College Board can tighten security in ways that will prevent conventional cheating the issue of using prior questions still remains unanswered.

The SAT was designed long before the Internet. Test questions could be given across time zones and countries with little fear of questions being forwarded or posted.  Now, however, the questions are sent around the globe in minutes. What should the College Board do? They have already created a new test in which some of the things students are asked to do are not easily subject to memorized answers. This is particularly true of the new writing section. From what I can tell the new section is a huge improvement over its predecessor. It ask student to write based on marshaling evidence--Critical thinking rather than memorization. It seems paradoxical then that most colleges and universities have made the writing section optional for students. The collages and universities do not want to discourage students who do not want to write an essay from applying to their schools. The section that is least susceptible to information sharing is the one that will be used least. I would hope that colleges and universities would strongly encourage students to take the writing section at least in places where students are thought to be given unfair advantages due to the sharing of old questions. I would go so far as to encourage schools get faculty to read the essay from the SAT as they are academic in nature and may help to predict academic success as well or better than the multiple choice questions. At the very least studies should be funded to look at how well the SAT essay predicts academic success. I do not expect schools will follow this proposal.

For the last several years the number of colleges and universities that have gone test optional has increased significantly. There are schools all over the US, many of them quite well known that no longer require applicants to submit SAT/ACT scores. I have written about this phenomenon before so I will not rehearse it all here. What I can predict, however, is that many more schools will go test optional in the coming years. First of all it increases the number of students who will apply to schools. This helps schools look more selective when making offers of admission and this in turn could help the rankings of the schools. More importantly, the schools may have less trust in the scores they receive and will, therefore, not put as much weight on them. For groups like Fairest and others who think the standardized test are pernicious this will comes as welcome news. For those who think that standardized tests, especially at the ends of the spectrum (very high testers and very low testers) are good predictors of academic success will be disappointed. Those who want to quantify students will also not be happy. Given that the new SAT is ties to the Common Core, those who support the Core will want the tests to remain a necessary part of evaluating schools ad students.

At this point it may sound like I think the SAT is in big trouble and that its future is in jeopardy. While this may be true for some of the reasons I‘ve listed and for some others as well. I do not think the issue with old questions being recycled will be the reason for its fall from being a common part of the admission process. In order for students to gain an advantage on the SAT by memorizing questions the students have to be incredibly focused and willing to devote many hours to the task. The answers student have require knowing the full question and the answer from many hundreds if not thousands of old questions. In other words, only incredibly motivated students can hope to get a significant advantage form memorizing old questions. While there is now a new SAT and the questions to memorize may not be as large as in the past, this will change within a year when at least 7 tests will be administered per year.

It should come as no surprise that the places where the issue of test prep and access to prior questions are in Asia. The rote memorization of materials has long has been a part of the educations system. Students spend untold hours memorizing materials for their own national exams. What many may not realize then is that these students are in fact cramming for the tests by memorizing the tests. This in and of itself requires a great deal of time and effort. The other piece of information about many of these students is that they are, in many cases bright students. For example, many students in China who take the SAT could score well above 2000 on the old SAT and above 1400 on the new test. These score while near the top percentages of all testers are low for China. In recent years student who are accepted to top schools in the US typical have 800 math scores and well above 700 on both the critical reading and writing sections of the SAT. There are hundreds of student scoring very well on the Sat and it is not only because they have access to old questions; rather, it is that they have studies hard to for tests all of their lives and are prepared to do well on the test anyway. 

The competition for getting into top US schools –which admit only small numbers of applicants from China and Korea—is so fierce that student are willing to devote tremendous effort to improve their scores. So far as I have seen there are not many students in the US who are ready to devote their summers to attending SAT camps that require 12 or more hours of SAT prep per day.  Some students from Asia see that they need nearly perfect scores to be admitted to top schools and therefore age willing to do this. It may be that there are others around the world and in the US who are willing to put in the effort to memorize everything on tests, but the overall percentage will be small. Will this small percentage undermine the credibility of the whole tests? Not likely. Some admission officers have already said that they look at scores from student in some places with a jaded eye. In other words a student with a great score in China will not be looked at the same as a student with a great score from Iowa. It is unlikely however that most schools will not expect the majority of their students who will be offered admission to submit scores and that most will assume the scores are accurate.

If I were a VC and I was looking for a new investment opportunity that has the potential to solve the problems with test security, I would look around for someone to create a test like the PISA exam or even the Gaokao which would be given once a year. The test would, however, not have a mean score like the SAT/ACT. In other words, the test results would not be compared to other version of the test given in other years. The test would be used t show how students did on a particular test. This test could be used as a substitute for the SAT or in addition to the SAT. I realize I am opening myself to a deluge of critical comments as many educators feel students are living in a standardized test culture and adding one more would only make things worse. It may make things worse for some but for those who need to prove that the results of their tests are accurate it may help the students and the colleges too.

The pragmatist in me believes that in the world of admission most  students and schools will muddle on with the tests that are out there and that the concern for test security will not undermine the use of the tests so along as they are tied to secondary school and college rankings. I have been wrong about a lot of things before so I will see how this looks a few years from now.