NORC Conducts Comprehensive Methodological Review of College Ranking Systems
?? Vanderbilt University asked NORC to examine college ranking systems.?
College rankings profoundly influence how prospective students and their families make educational decisions. Given that influence, Vanderbilt University asked NORC to independently, comprehensively examine how college ranking systems distill multiple abstract qualities about a college or university into easily digestible, numbered lists.?
“We used a construct validity framework—which determines if a measure accurately represents the theoretical concept it's intended to measure—to assess whether the methodologies of five popular college ranking systems accurately measure the qualities that form an overall school ranking,” said Susan Paddock, an executive vice president and chief scientist in NORC’s Research Science division. “We discovered that many aspects of college rankings suffer from a lack of construct validity.”
Specifically, we found that:? ?
领英推荐
“Poor data is particularly problematic because it can undermine the ability to capture even well-defined concepts—like graduation rates, student debt, and value-added earnings—potentially skewing the scores,” said?Soubhik Barari, a research methodologist in NORC’s?Methodology & Quantitative Social Sciences department. “Whenever possible, we suggest implementing processes to improve data quality, such as a review of data sources, more rigorous statistical correction procedures, and institutional audit mechanisms.”??
?? Our report recommends enhancing the clarity and transparency of the ranking processes.
Our report makes several other recommendations to address these validity issues and enhance the clarity and transparency of the ranking processes. They include enriching the rankings by adding visual elements; convening higher education experts and other stakeholders to establish common standards, measures, and definitions for future college rankings; or abandoning rankings altogether in favor of a tiered rating system. (While NORC is an independent affiliate of the University of Chicago, the university was not involved in this research at any stage, including commissioning the analysis, conducting the methodological review, synthesizing findings, or forming recommendations.)
“We hope that our report will better inform students, parents, and institutions of higher education about the strengths and limitations of ranking systems and ultimately lead to changes that will improve the quality of information available to those seeking to make a substantial investment of time and financial resources in their futures,” Paddock said.
Explore the Report: NORC Reviews College Ranking Methodologies
I study Crime Policy, Urban Violence and how Technology — particularly AI and accelerating tech — is changing crime.
4 周A much needed analysis. Rankings are not unimportant but they are ripe for abuse. Greater clarity and transparency on what goes into formulating them would help curb their abuse as misleading marketing tools by universities as well as innaproptiate measures of potential success by donors and funders.