Benchmarking - Measuring the Quality of Research

The development of an House of Quality requires a lot of identification, classification and rating. Rating by the application of quantifiable Key Performance Indices (KPIs) seems at the first glimpse more objective than the usage of qualified supposedly subjective KPIs. Nevertheless, there are certain aspects of academic research quality, which cannot be constituted through quantifiable KPIs only. Therefor the additional implementation of a process translating academic research quality into quantifiable KPIs seems necessary. Through the consideration of multiple prioritized parameters, the process should try to reduce subjectivity in the rating of academic research quality.

As a best practice example and for benchmarking, I want to show case the Research Excellence Framework represented in the publication of Thompson & McKenna (2022).

?“Research quality is a term often bandied around but rarely clearly defined or measured.”

Thompson, D. R., & McKenna, H. P. (2022). Research Quality—Lessons from the UK Research Excellence Framework (REF) 2021. Nursing Reports, 12(3), 510-514.

The national Research Assessment Exercise (RAE) is one way of measuring the quality of research across different higher education institutions and its parity with other disciplines through periodic national research assessment exercises. In 1986, the first national RAE in higher education took place in the United Kingdom. The purpose was to determine the allocation of funding at a time of tight budgetary restrictions. In 2014, the first Research Excellence Framework (REF) replaced the RAE.

The Stern report identified five purposes:

• to provide accountability for public investment in research and produce evidence of the benefits of this investment
• to provide benchmarking information and reputational yardsticks for the higher education sector and for public information
• to provide a rich evidence base to inform strategic decisions about national research priorities
• to create strong performance incentives for higher education institutions/ researchers
• to inform decisions on the selective allocation of non-hypothecated funding for research

As the REF is a discipline-based expert review process, 34 expert sub-panels, working under the guidance of four main panels, reviewed the submissions and made judgements on their quality. The panels comprised 900 academics, including 38 international members, and 220 users of research.

Results were produced as “overall quality profiles”, which show the proportions of submitted activity judged to have met each quality level from 4* to unclassified.

The rigour, significance and originality of research outputs were judged on a 5-point scale:

• 4* (quality that is world-leading in terms of originality, significance and rigour);
• 3* (quality that is internationally excellent in terms of originality, significance and rigour but which falls short of the highest standards);
• 2* (quality that is recognised internationally in terms of originality, significance and rigour);
• 1* (quality that is recognised nationally in terms of originality, significance and rigour);
• Unclassified (below the quality threshold for 1* or does not meet the definition of research used for the REF).

The overall quality profile awarded to each submission is derived from three elements that were assessed the quality of research outputs (contributing 60% of the profile); the social, economic and cultural impact of research (contributing 25% of the profile); and the research environment (contributing 15%) of the overall quality profile.

Get the paper: https://www.mdpi.com/2039-4403/12/3/48/pdf?version=1657195549

The process to translate academic research quality into quantifiable KPIs is for sure not universally applicable, but through its adjustment a must have to complete any academic research rating system based on quantifiable KPIs. Therefore, it is obvious that the process to translate academic research quality into quantifiable KPIs will be a central element of the Academic Research Impact Cockpit (ARIC).