How to measure the performance of a scientist?

The often debated question in the research world is "How do I measure the performance of a scientist?"

What is the impact of the representative work or selected publications that has appeared frequently in recent years on the future prospects and development of scientific researchers? Will the academic system realize any substantial progress from simply counting papers to changing the assessment to reflect the influence of a cluster of work?

With these two key issues in mind, today, I will take you through a discussion on the in-depth understanding of representative work and help you judge the new requirements and new directions for future papers and how you can take your work further.

Papers, the lifeblood of scientific researchers

The results of the scientific paper have been recognized as an output metric by academic peers, and also the metric for universities and scientific research institutes to strive for resources based on the various rankings that have been adopted. As a result, the trend of paper counting was popular for a time. With people trying to strive for more and more papers, leading to a high volume of publications. But a ton of papers is not the best way to evaluate performance as we all now know.

The chase for papers not only leads to the vicious circle of talent evaluation, academic evaluation, project evaluation, to name a few, but also the use of SCI in the overall construction and development evaluation of universities. The so-called scientific research spirit of "life-long learning; search by research; patience and quality" began to degenerate into a public joke.

It appears that instead of volume of papers, one could instead look at the representative work as a method to gauge what the scientist is actually good at. In order to consider someone for a promotion to a higher level, there has been a shift to the concept of representative works. When applying for the evaluation and appointment to a more senior position, the participant would have to submit 1-3 representative works, whose academic value was assessed by experts. If they were unanimously recognized, even if only one representative work could be employed as an evaluation criteria for the senior title.

Will this weaken the dominant position of the paper? There could be other achievements such as patent achievements, project reports, project plans, teaching plans, medical records (how about assessing the health of a person if he/she can take up the higher and potentially more stressful position) and other achievements which can replace the paper output. This could somewhat broaden the use of the representative works to a certain extent. How then can one enhance the write up on representative achievements? The activity and influence of discipline, the position of important academic organizations or journals, the originality of the research achievements, the game-changing influence and translation of technology, and the satisfaction of science and technology services are taken as further important evaluation indexes.

In the evaluation of publicly funded research, applied technology development and other types of scientific research talents, the number of SCI and core journal papers published, paper citation list and impact factor ranking should not be the only metrics used as the evaluation reference.

Applied research should not be bound to papers

In terms of quantity, in the process of project selection and scientific and technological awards, no more than 5 representative works belong to the research field. On the one hand, it is important to break the unhealthy atmosphere of having either only the number of papers, on the other hand, it is also detrimental to have the sole corresponding author criteria in the future papers. These mean nothing if the representative research works are taken in to account.

From the perspective of applicable categories, the differences between basic research and applied research can be distinguished. The former focuses on evaluating the quality, contribution and influence of the landmark achievements such as "new discovery, new viewpoint, new principle and new mechanism" in the paper; The latter focuses on the evaluation of the quality, contribution and impact of such landmark achievements as "new technology, new process, new materials, new equipment, key components, experimental devices / systems, new diagnosis and treatment programs, clinical guidelines / specifications". The paper should not used as the main evaluation basis and assessment index.

In this way, many dimensions closely related to scientific research evaluation, such as talent evaluation, project evaluation, platform evaluation, and science and technology awards, should also be considered. It can be seen that the assessment of representative work will really change from simply referring to papers and affirming all kinds of academic achievements. It will zoom in on what the researcher is really good at. To a certain extent, it liberates the researchers in the field of non basic research.

Controversial point: peer review of organizations

Some people are skeptical about who will evaluate the representative works, but also worried about the fuzzy standard of the combination of quantitative evaluation and qualitative evaluation. As for the subject of evaluation, learning from the best practices of other universities and adopting the way of peer review is an important step forward. Once upon a time, the administrative scientific research management in universities was the main way of evaluating a person, sometimes this only looked at SCI, division and influence factors, and the quantitative management mode of journal level, project level and award-winning level was the output.

Today, the opinions of experts recognized in the industry should be used as the main tool for judgment. In fact, this is a more objective and convincing move, which also shows the original intention of replacing central appraisal with expert evaluation in the system that uses representative works. However, there is a need for a clear answer to the specific rules of peer review and the criteria for collaborative evaluation. On the combination of quantitative and qualitative, the quantitative part is not difficult to understand. How can qualitative evaluation avoid subjective factors? After reading the relevant documents in detail, it is found that the qualitative evaluation advocated by the personal bias of the reviewer, on the one hand, is determined by experts' academic value and influence, on the other hand, is combined with the degree of contribution.

For younger scientists, it can be predicted that the identity of the first author, or even the corresponding author, will occupy a larger proportion in the paper quality evaluation in the future; At present, the controversial "academic value and influence" is more based on the degree of "original innovation" to judge the value, and the incremental research will be gradually eliminated. In order to really convince the scientific community, there has to be a unified and reasonable weighting formula. However it is unclear whether this can be derived. Although some implementation details still need to wait and see, one thing remains unchanged: the original and applied research is something that should be measured not only with papers but with other metrics as well, and the substandard products of opportunism, making up for the number, and shoddy research cannot stand the test under any system.

Future: let papers return to academic essence

There is nothing wrong with aiming papers itself. It is one of the manifestation and communication carriers of scientific and technological innovation achievements; However, it is imperative to oppose the quantitative evaluation of papers and pay attention to the connotation, quality and social contribution of scientific research achievements. It is always the creed that scientific researchers and research institutions should follow to let the papers return to the academic essence and make excellent works which can make a difference to human life or enhance our knowledge.