Identifying & Remediating Rater Drift in CNS Clinical Trials

Successful drug development hinges on the collection of valid data. And in clinical trials related to the central nervous system that involve clinical raters, it becomes vital that raters collect and score interviews the same way throughout the trial and maintain high inter-rater reliability.??

Based on 25+ years in the mental health and clinical trials space, Felice Ockun—Senior Manager of Cogstate’s Clinician Network—recently shared valuable insights on identifying and remedying rater drift, a key challenge in ensuring data reliability.?

Understanding Rater Drift:?

The term rater drift can be thought of as diminishing inter-rater reliability that occurs over the life of a study. Raters can drift in different ways; from how they are administering scales—such as how thoroughly they are conducting interviews—to how they are scoring scales—such as forgetting a particular scoring convention.?

Four Ways to Identify Rater Drift:?

1. Round Robins: A collaborative exercise where raters take turns administering items of an interview.?This is “the lowest touch exercise, requiring the least amount of your organization’s resources,” said Ockun.
2. Gold Standard Calibration Videos: Comparing raters' scores against expert panel consensus.?“The big advantage to this method is that once the upfront work is done of creating the videos and convening an expert panel to reach consensus on the scores, your organization can then use this video for training and calibration purposes on an unlimited number of raters indefinitely,” said Ockun.
3. One-on-One Observed Interviews: Direct observation and feedback sessions with expert trainers.?“This approach does take more of your organization’s resources,” said Ockun, “but it allows you to observe drift occurring in administration, as well as in scoring.”
4. Data Analytics: Using algorithms to detect patterns and outliers in rater data. “Examples of flags might be too large of a score increase between visits, too large of a score decrease between visits, or inconsistent scores on two items that should hang together,” said Ockun.?

Interested in a Case Example??Read an example highlighted here on rater drift within MADRS and HAM-D Assessments?

Remediation Strategies for Rater Drift:?

Remediation depends on the drift type. For issues in conducting interviews, didactic re-training followed by observed interviews is effective. For scoring-related drift, re-training on scoring conventions or anchors, coupled with training video reviews, is recommended. Continuous data analysis post-remediation ensures sustained quality.?

Identifying and remedying rater drift is a critical aspect of maintaining data integrity in clinical trials. For more detailed insights and methodologies, feel free to contact our team or access our free webinar for more information.?

_____?

Felice Ockun | Senior Manager, Clinician Network, Clinical Trials?

Felice is a seasoned mental healthcare professional with more than 25 years of experience in the mental healthcare industry, specializing in global clinical CNS trials, psychiatric diagnosis, rater training, quality oversight, and clinical data analytics. Felice’s diverse background spans management roles in clinical trials, inpatient/outpatient mental healthcare, and emergency mental healthcare settings.?

Felice received an MS in Clinical Psychology from LIU’s Clinical Psychology Doctoral Program, and an MS in Clinical Social Work from Columbia University. She also holds a BA in Psychology and Sociology from Emory University.?

?