Testing Trends: The Product Perspective

In March, I attended the industry’s largest conference, ATP’s Innovations in Testing 2024. As the Director of Product Management at ITS, I’m always excited for this conference because it gives me a glimpse into the collective direction of the testing industry. I look at the patterns in sessions and in conversation around the booth, and I take note of the most popular trends. Today, I want to share those trends, which focus on AI, security, and accessibility.

Artificial Intelligence Trends

AI is still in its infancy in the industry. Solutions have come far in a year but are far from mature. Some have solutions that appear to be bolted onto the side of the existing application and some have solutions that are centered around a generative AI item generation workflow. As the AI space continues to progress, solutions in the assessment space are evolving and new use cases for the technology are rapidly emerging.

Exercising Caution Around AI

The industry is rightfully cautious about AI, specifically items created via generative AI. The industry is, by nature, conservative. It highly values reliability, consistency, validity, and quality and needs to be able to prove out the technology. There are several experiments and studies being done by different organizations to determine how the current technology fares.

One interesting benefit of using AI-generated items is that the review process could go faster. Teams can reject bad items more quickly because the rejection is not going to hurt anyone’s feelings. In addition, the SME can focus on quality instead of generating new items.

Using AI for Data Analytics

There is high interest and less concern regarding using AI for data forensics and data analytics, including keystroke analysis, provided that the AI technology does not decide but instead is used to inform the decisions of investigators. There is more openness to explore capabilities in lower-stakes environments and move to higher-stakes testing as the technology and techniques improve.

Trying Prompt Engineering and Fine Tuning

Many ATP attendees had experience in using generative AI and were looking for ways to go deeper and get better results. Accordingly, there is much interest in prompt engineering and fine-tuned models.

Retrieval-Augmented Generation (RAG) was discussed as a way to improve results, but this is different than some proposed as it is not part of the prompt. When I was confirming my understanding of RAG, I discovered a related concept of “Semantic Search,” which can also provide better context for better responses.

Takeaway

Our industry must continue to research the different generative AI techniques such as fine tuning, RAG, semantic search, and prompt engineering to determine the best way to use each to get better results from generative AI.

Security Trends

Security is a growing concern of the industry. The at-home delivery mode quickly adopted during the Covid-19 pandemic out of necessity shifted the economics of cheating. The proxy testing market exploded. With systems no longer highly managed and controlled by testing centers, cheaters and those helping cheat found ways around security measures that were intended as part of a more trusted environment.

Addressing High-Tech Cheating

High-tech cheating is on the rise. Several methods are used to cheat current systems. These include software and hardware mechanisms to both capture content and assist in answering. In addition, several types of AI cheating approaches exist, including the use of AI Agents, also known as intelligent agents (IA).

There are limits to what can be done to prevent cheating when someone else controls the environment (hardware, software, the room itself), especially when cameras can be hidden in the test taker’s space. Companies also aren’t using available low-tech, low-cost, low-effort methods effectively to deter cheating.

Delivering at Test Centers

As cheating undermines the trustworthiness of test results, the industry is at risk of losing credibility and certifications are at risk of losing value. With renewed fervor, because of the ease of cheating when testing remotely, I am expecting high-value, high-stakes programs to limit testing to test center delivery to preserve their brands. I would not be surprised if some programs offered tiered products where lower-level certification tests can be taken remotely but advanced certifications must be taken in a trusted environment.

Takeaway

Our industry will need to be willing to share their innovations for the common good, like Mercedes with the crumple zone and Volvo with the three-point seatbelt.

Accessibility and Accommodation Trends

Accessibility continues to be an important topic at the conference. It is often discussed in the context of diversity, equity, and inclusion, though primary efforts to improve each of those topics are not typically focused on technology solutions. However, accessibility has a clear technology implication and is intended to allow those with disabilities to access services.

The ADA defines a disability as “a physical or mental impairment that substantially limits one or more of the major life activities.” We typically think of those who need accommodations for visual, auditory, or motor impairment; however, the need is broader and includes other types of disabilities that may be less obvious. For example, neurodivergent individuals such as those with autism and ADHD may also require accommodations. Done well, increasing accessibility for those with disabilities should improve the experience for all people.

Changing Requirements for Accommodations

Several organizations are changing their requirements so that excessive paperwork is not necessary for accommodations that require approval such as extended time. Many other accommodations do not require approval and are available to all test takers because those accommodations do not give an unfair advantage in test-taking scenarios.

In the past, programs may have required documentation of diagnostic testing that was expensive and not easily available. New requirements may allow applicants to identify non-family, non-friend individuals such as instructors, mentors, or managers that can attest to the accommodations the individual needs in their current studies or work. These changes to the approval process are intended to make sure those needing accommodations receive them fairly.

Accommodating for Test-Taking Anxiety

Some in the industry are attempting to mitigate the psychological impacts of testing, specifically anxiety. Anxiety does not usually appear named with the request for accommodations but is often related to several of the neurodivergent types. In attempts to control the environment to reduce error in testing, another source of error is introduced. The effects of test-taking anxiety have a significant impact on the individual and their performance.

Recommendations to reduce test-taking anxiety include having a candidate advocate, taking a candidate-centric approach, and soliciting and implementing test-taker suggestions to improve the test-taker experience. At ITS, our improvements in this area include a new user interface, a timer that allows for toggling the display, unscheduled breaks, and the “sandbox” practice test that allows the test taker to familiarize themselves with the test UI and samples of every item type.?

Takeaway

Our industry’s ongoing efforts to improve accessibility and usability must continue so we can enhance the user experience of all test takers whether they need accommodations or simply have test anxiety.

Conclusion

The emergence of AI, the growing importance of security, and the ongoing efforts to improve accessibility and accommodations are shaping the future of our industry. I’m excited about these developments and the opportunities they present. I look forward to continuing this journey with you, exploring new trends, overcoming challenges, and pushing the boundaries of what’s possible in testing. Together, we can create a future where testing is more efficient, secure, and inclusive for everyone.

About the Author

Jeffrey Spranza is the Director of Product Management at ITS where he leads the team of technical product managers transforming the online assessment experience. Jeffrey joined ITS in 2005 with 15 years of experience in IT consulting and technical services. Since then, he’s played a critical role in the company’s growth and product improvement. Jeffrey holds a BS in Industrial and Systems Engineering from Virginia Tech and is certified as a Project Management Professional (PMP) and Certified Scrum Product Owner (CSPO). He’s also a proud Eagle Scout and enjoys outdoor activities with his wife and two sons. Connect with Jeffrey on LinkedIn.