Update on Improving Fellowship Review: A Request for Information

This blog has been co-authored by Noni Byrnes, Director, Center for Scientific Review, NIH and Michael Lauer, Deputy Director for Extramural Research, NIH. Originally posted on the?Review Matters blog.

NIH is recommending changes to the peer review of?Ruth L. Kirschstein National Research Service Award (NRSA)?fellowship applications by restructuring the review criteria and modifying some sections of the?PHS Fellowship Supplemental Form?that are specific to NRSAs. The goal of this effort is to facilitate the mission of NRSA fellowship peer review – to identify the most promising trainees and the excellent, individualized training programs that will help them become the outstanding scientists of the next generation.?The proposed changes will 1) allow peer reviewers to better evaluate the applicant’s potential and the quality of the scientific training plan without undue influence of the sponsor’s or institution’s reputation; and 2) ensure that the information provided in the application is aligned with the restructured criteria and targeted to the fellowship candidate’s specific training needs.?The RFI requests public input on this proposal. To comment, go to the?RFI, which contains additional background and links to submit your thoughts.?Most of that background follows:

The first stage of NIH peer review serves to provide expert advice to NIH on assessment of the likelihood that the fellowship will enhance the candidate’s potential for, and commitment to, a productive independent scientific research career in a health-related field. The criteria for the review of NRSA fellowship applications derive from the?NRSA regulation 42 CFR 66.106, with four pertinent factors: (1) the scientific, technical, or educational merit of the particular proposal; (2) the availability of resources and facilities to carry it out; (3) the qualifications and experience of the applicant; and (4) the need for personnel in the subject area of the proposed research or training. NIH currently organizes these criteria into the following categorical labels:?Applicant, Sponsors and Collaborators, Research Training Plan, Training Potential, and Institutional Environment and Commitment. By?NIH policy,?peer reviewers are also required to evaluate Training in the Responsible Conduct of Research, Biohazards, Resubmissions, Foreign Organizations, Select Agents, Resource Sharing Plans, Budget and Period of Support, and Authentication of Key Biological and/or Chemical Resources.

NIH gathered input from many sources in forming this proposal. Unsolicited comments over a period of years, reflecting persistent concerns that the NRSA fellowship review process disadvantages some highly-qualified, promising applicants led the Center for Scientific Review (CSR) to form a?working group?to the CSR Advisory Council. To inform that group, CSR published a?Review Matters?blog which was cross-posted on the Office of Extramural Research blog,?Open Mike. The blog received more than 1,500 views by unique individuals and numerous comments. The working group presented?interim report?to the CSR Advisory Council, which adopted the recommendations, at public CSR Advisory Council meetings (March 2022?video,?slides; September 2022?video,?slides). Final recommendations from the CSR Advisory Council (report) were considered by the CSR Director, as well as major NIH extramural program, review, and policy committees that included leadership from across NIH. Recommendations were presented to the NIH Advisory Committee to the Director in December 2022 (video,?slides).

Improving NRSA Fellowship Review

Recommendation 1: Revise the Criteria Used to Evaluate NRSA Fellowship Applications

An Overall Impact Score (scored 1-9) will reflect the scientific and educational merit of the proposal and an assessment of the likelihood that the fellowship will enhance the applicant’s potential for, and commitment to, an independent, productive research career in a health-related field.?Reviewers will take into account their assessments of the three criteria in determining an Overall Impact Score. Each of the three criteria will receive an individual score.?The “additional review criteria” below are unchanged, will not receive individual scores, but will be considered in arriving at the Overall Impact Score. Two “additional review considerations”, also unchanged, will be evaluated but have no effect on the Overall Impact Score. Review the?full text?of the proposed changes.

Review Criteria

I. Scientific Potential, Fellowship Goals, and Preparedness of the Applicant

• criterion emphasizes their accomplishments in the context of their stage of training and the scientific opportunities they have had, as well as other factors that bear on their potential to succeed, such as determination, persistence, and creativity. (scored 1-9)

II. Science and Scientific Resources

• criterion emphasizes the extent to which needed technical, scientific, and clinical resources are specified and are realistically available to the applicant, and whether the scientific expertise of the mentorship team is appropriate for the proposed science and that the role of each mentor is clearly defined. (scored 1-9)

III. Training Plan and Training Resources

• criteria emphasizes whether the necessary training resources are well-specified and available, specifically the practical availability of resources, and to include an evaluation of the training philosophy of the sponsor, their approach to training, time commitments, and their accessibility. (scored 1-9)

Additional Review Criteria?(not scored, but affecting Overall Impact; no changes proposed,?see current language):

? Protections for Human Subjects

? Inclusion of Women, Minorities, and Individuals Across the Lifespan

? Vertebrate Animals

? Biohazards

? Resubmission

Each of the Additional Criteria except the last will be rated as “Appropriate”, with no comments required, or as “Concerns”, which must be briefly justified. Resubmission will be given brief written evaluations.

Additional Review Considerations?(not scored and having no effect on Overall Impact) no changes proposed,?see current language):

? Training in the Responsible Conduct of Research

? Authentication of Key Biological and/or Chemical Resources

? Budget and Period of Support

? Applications from Foreign Organizations

? Select Agents

? Resource Sharing Plans

Recommendation 2: Revise the PHS Fellowship Supplemental Form

Changes to the application instructions are needed to better align the information applicants provide with the revised review criteria. The PHS Fellowship Supplemental Form currently includes the following four sections: Fellowship Applicant; Research Training Plan; Sponsor(s), Collaborator(s), and Consultant(s); Institutional Environment and Commitment to Training.?The NIH proposes to revise the Fellowship Applicant section and the Sponsor(s), Collaborator(s) and Consultant(s) section. NIH also proposes to change the instruction for Letters of Reference. An additional proposed change would allow an?optional?Statement of Special Circumstances from the fellowship applicant.?Review the?full text?of the proposed changes.

Through the?RFI, NIH continues to seek public comment on the proposed changes before moving forward with implementation. The RFI will be open for a 60-day period, until June 23, 2023. We look forward to your comments.