Reviews as Research and Good Professional Practice

by Denise M. Rousseau

A movement is afoot in organization and management to conduct and make use of carefully conducted reviews of research for both decisions and practical guidance. ?Its key feature is use of systematic methods to reduce bias and create trustworthy knowledge (e.g. Kunisch et al., 2023; Rousseau, 2024). ?I treasure the clarity and fresh thinking stimulated by reading and even carrying out a well-specified review and would like to invite more practitioners to share these experiences.

Systematic review methods are relatively new in management research and practice (Chalmers et al., 2002; Kunisch et al., 2023; Rousseau, Denyer & Manning, 2008; Tranfield, Denyer & Smart, 2003). ?They are widely used in the health professions from medicine and nursing to public health. ?It was a systematic review that convinced emergency physicians (also known as “ER doctors” (emergency room) to give heart attack patients aspirin. ?I hope to encourage practitioners to seek out systematic reviews AND, if warranted, conduct their own reviews on important questions. Thus, I describe here the kinds of reviews that fall into the category of “systematic” and indicate their key features should you want to conduct one yourself.

Three Kinds of Reviews Using “Systematic” Methods

• CATs: CATs are “Critically Appraised Topics” and are “short and sweet” summaries of the peer-reviewed literature on a practical question. (“Peer-reviewed” means published in a scholarly journal whose editorial team has judged the research worthy of publication.) ?First used by physicians and medical students to figure out how an unfamiliar patient condition might be treated, CATs are “quick and dirty” searches on a well-specified question. ?In management, this might be, “Is there research evidence on the effects of Unlimited Vacation Time on employees?” ?“What are success factors in onboarding new employees?” ?In neither case, are you looking for a complete picture of the evidence but rather a sense of direction and general trends.? Conducting CATs is increasingly part of the training of schools of business and management (Gubbins, Rousseau & Capezio, 2025).
• REAs (bigger CATS): “Rapid Evidence Assessments” are a thorough search of the published peer-reviewed (scholarly) literature (while CATs are just a sample of published studies). ?REAs are conducted by consultants and information specialists for clients with critical practical questions. ?CEBMa, the Center for Evidence-Based Management, which I founded with Eric Barends, Managing Director, conducts REAs for businesses like Novartis and non-profits like CIPD seeking to make critical decisions in an evidence-based fashion. ?The critical questions addressed by REAs may evaluate a potential organizational change (“Is changing organizational culture an effective way of improving firm performance?”) or identify key features of effective performance management (“What are success factors in performance management?”).
• SRs (aka the “Lion” of the Review world): “Systematic Reviews” are thorough and complete searches of both peer-reviewed and unpublished research on a topic.? They are fundamental to clinical and medical practice worldwide with major international collaborations helping to conduct and publish them. ?The Cochrane Collaboration helps to guide and conduct SRs in medicine, nursing, and public health, while the Campbell Collaboration promotes and publishes SRs in public policy, criminal justice, education, food insecurity, business and management (e.g., Rousseau et al., 2023). ?Such reviews are the bedrock of scientific advances by virtue of the pre-approval of their methods by a board of methodologists, information specialists, and subject matter experts, reducing the likelihood of publication bias and errors in the synthesis process.

If you are looking to see if CATs, REAs, and SRs exist on topics of interest, I suggest two things:

1. Reach out to an academic or other professional working in your domain and ask for recommendations (this gets you “pointer knowledge” regarding your domain of interest, saving time, and easing the search process)
2. Search “Systematic Review” in GoogleScholar for your topic and try a variety of synonyms.? Similarly, you can similarly search “Rapid Evidence Assessment” as many organizations like CIPD and CEBMa, which produce these, also make them public. (CATs tend to be conducted for private use and are less readily accessible.)

(Helpful Hint: Should you have difficulty accessing an article’s full-text, email the author.? This makes my day, and I bet other authors also appreciate knowing people are reading their work. Never ever pay a publisher for a copy of an article that the authors contributed for free.)?

Note I put developing professional contacts first because good evidence-based practice is very much linked to having a network of knowledgeable academics and practitioners (HakemZadeh & Rousseau, 2024). ?Getting in touch with communities-of-practice helps uncover where pertinent work exists in fields from management, sociology, and psychology to medicine, education, and economics, and the myriad research databases, and sometimes books, containing their work.

Using Systematic Review Methods

Careful review practices are essential to producing a trustworthy review and it is the method of conducting a review that makes it “systematic”. ?Whether you conduct a CAT, REA or Systematic Review, your method needs to be structured, transparent, and replicable. ?The trustworthiness of your conclusions reflects the quality of the judgments you make in the review process. ?Since I believe that our readers are more likely to be interested in conducting CATs and REAs, based on a general treatment of “systematic” methods (Rousseau, 2024) I offer here the basic building blocks of a good quality CAT or REA (for more detail see CEBMa (2024; https://cebma.org/resources/guidelines-reas-and-cats/).

Step 1: Specify Clear Question(s)

Start with a clear question or set of questions about a problem or issue. ?In my experience, it can help to first do a quick and dirty scoping review to check out the peer-reviewed research on a topic that interests you and the terms people use to describe it. ?My favorite source for terminology is Wikipedia as it provides helpful overviews of many practice issues and terms (e.g., Balanced Scorecard, Merit Pay, Business Models, etc.). ?Scoping out your initial question or topic can help you identify key terms to better specify and inform your question and find relevant research. You might do an initial pass through GoogleScholar and see what kind of work shows up and then adjust your terms to get more relevant hits. Identifying the terms that make for a useful question can mean deepening your understanding of what you are interested in. ?A well-framed question helps review authors to specify their search strategy and inclusion criteria (i.e., which studies to include).

Step 2: Specify Detailed Search Methodology

A quality CAT or REA uses a pre-specified search strategy on a specific question. ?The goal of systematic approach is to reduce bias and represent the domain reflecting your question. ?For a question about the effect of diversity in groups, a search strategy might specify articles that are empirical, study diversity at the team level, focus on the information diversity provides to a group, and are in English (or whatever languages you can access). ?In my experience, working with a research librarian makes a difference, including the business librarian at your public library.? Librarians in business or social science can help identify relevant research databases. ?A librarian’s skills in searching not only help with syntax, which differs across databases, but also in refining terminology.

A critical judgment in any review is whether relevant research appears largely in peer-reviewed journals and conferences, or whether books are the normal outlet for scholarship as is the case in sociology and political science. ?If books are a major source, I recommend an approach that informs a broad search, that is, getting leads from cross-disciplinary advisors, using Google Scholar as a database, and scanning the reference section of key publications you find useful (this is called “snowballing”). ?More commonly in management, journal articles are the chief outlet for scholarly work, hence research databases tend to be the primary data source reviews rely on.? ?

The choice of databases depends on the bodies of literature that comprise the review domain.? Scientific research is increasingly accessible through numerous on-line databases (e.g., ABInform, Web of Science, PsychInfo, JSTOR), some available at the public library (and often on-line), others accessed through universities, national governments, and NGOs like the Center for Evidence-Based Management. (Access to peer-reviewed literature is provided by CEBMa to its members.)

Peer-reviewed literature tends to be the primary focus in management research, a broad tent including diverse disciplines and communities of practice. ?Published studies vetted for quality by knowledgeable scholars (that is, those peers who are reviewing studies for publication) typically meet at least minimum standards for quality and presentation. ?To search this literature, check peer-review as a filter in your database search when specified.? Note that some popular management sources are not peer-reviewed (e.g., Harvard Business Review).

??????????? Search Terms. Search terms and their syntax vary across databases though basic keywords are similar.? Following the preparations described above, a systematically conducted review details pre-specified search terms that can then be evaluated in relation to the review question. ?Pre-specified search terms typically include keywords in the title and abstract. (You will need to adjust your search syntax depending on the research database: Some permit NOFT-searching (anything “not in full text”) while other databases require that Title and Abstract be specified separately.) ??

??????????? Search date. Search date is a critical filter in many reviews, that is, whether work relevant to a review falls within a certain period (e.g., from the search start date to present). ?Search date may be left open when no clear criteria are identified. ?For practice-oriented reviews the most important reason for setting a search date is whether the review covers a phenomenon with an historical inflection point, focusing your review on a specific era. For example, a review question might address post-Covid employment practices or the effects of video-conferencing, leading a search to specify 2022 in the first case and 2005 in the latter (based on Wikipedia!).

Step 3: Identify Appropriate Study Type ?

For practitioners conducting CATs and REA’s it is useful to recognize that the design of studies your review turns up tend to differ. ?Your question might be better addressed by certain research designs rather than others.? Methodological appropriateness refers to the kind of questions answerable with specific research designs (Barends & Rousseau, 2018). ?Experimental designs might be most appropriate for causal questions, longitudinal studies for prediction, cross-sectional surveys for participant experiences, and qualitative research for issues of meaning and interpretation. ?As such, qualitative designs might be best for an exploratory study or one in which interpretations of meaning are the focus, while experimental designs are suited to test well-defined manipulations of cause and effect. In your table summarizing the studies your review turns up (Synthesis below), I suggest you enter the kind of research design each study employs.? Also you may wish to limit your search to certain kinds of designs (e.g., “experiment” “controlled study” or “interviews”) in line with the nature of your question. (In our study examining the evidence on whether CEO financial incentives predicted firm performance (Rousseau et al., 2023), we included only studies with a time lag between CEO pay and firm performance.)

Step 4: Synthesis—The Review’s Intellectual Product

Synthesis is the process of analyzing and reflecting on included studies and uncovering cross-cutting themes and novel insights. ?Identifying themes and insights involves sensemaking and imagination in the synthesis process with the goal of arriving at a position regarding the review question (Cronin & George, 2023).? As described in the CAT and REA guides published by CEBMa, it is helpful to put the studies you identify in a table, allowing you to compare research designs, populations, and observed effects.? In reviewing the table, you should reflect on what your search has yielded as a whole, noting overall patterns-- and being less swayed by individual studies. ?Then return to your question to consider whether the key information you sought at the outset has been identified. ?Or whether your understanding of the question or issues has changed by the review process.

Conclusion

Learning how to conduct reviews systematically was for me an eye-opening, character-building experience (Rousseau, 2024). ?Eye-opening because I found useful research in areas I hadn’t considered previously (Medicine! Engineering!). ?Character-building because following a structured process differed from my usual free-wheeling approaches to searching. ?But I can attest that conducting rigorous reviews can become second nature. ?Critical and provocative positions are better informed and more persuasive when grounded in rigorous review.? Advice to practice can be offered based on a stronger foundation. Importantly, making use of and conducting trustworthy reviews can become part of the practitioner’s skill set, enhancing their practice and deepening expertise.

?

References

Barends, E., & Rousseau, D. M. (2018).?Evidence-based management: How to use evidence to make better organizational decisions. Kogan Page Publishers.

Campbell Collaboration (2024).? https://www.campbellcollaboration.org [Accessed April 27, 2024].

Chalmers, I., Hedges, L. V., & Cooper, H. (2002). A brief history of research synthesis. Evaluation & the Health Professions, 25(1), 12-37. https://doi.org/10.1177/0163278702025001003

CEBMa (2024) Guidelines for Conducting REAs and CATs. https://cebma.org/resources/guidelines-reas-and-cats/ [Accessed December 15, 2024].

Cochrane Collaboration (2024).?? https://www.cochranelibrary.com [Accessed April 27, 2024].

Cronin, M. A., & George, E. (2023). The why and how of the integrative review. Organizational Research Methods, 26(1), 168-192. https://doi.org/10.1177/1094428120935507

Gubbins, C., Rousseau, D.M. & Capezio, A. (2025). CEBMa Handbook for Teaching Evidence-Based Management. Leiden: Center for Evidence-Based Management.

HakemZadeh, F., & Rousseau, D. M. (2024). Evidence-based decision-making is a social endeavor. Behavioral Science & Policy. https://doi.org/10.1177/23794607241265206

Kunisch, S., Denyer, D., Bartunek, J. M., Menz, M., & Cardinal, L. B. (2023). Review research as scientific inquiry. Organizational Research Methods, 26(1), 3-45. https://doi.org/10.1177/10944281221127292

Rousseau, D. M. (2024). Reviews as research: Steps in developing trustworthy synthesis. Academy of Management Annals, 18(2), 395-402. https://doi.org/10.5465/annals.2024.0132

Rousseau, D. M., Manning, J., & Denyer, D. (2008). Evidence in management and organizational science: Assembling the field's full weight of scientific knowledge through syntheses. Academy of Management Annals, 2, 475-515. https://doi.org/10.1080/19416520802211651

Rousseau, D., Kim, B. J., Splenda, R., Young, S., Lee, J., & Beck, D. (2023). Does chief executive compensation predict financial performance or inaccurate financial reporting in listed companies: A systematic review. Campbell Systematic Reviews, 19(4), e1370. https://doi.org/https://doi.org/10.1002/cl2.1370

Tranfield, D., Denyer, D., & Smart, P. (2003). Towards a methodology for developing evidence-informed management knowledge by means of systematic review. British Journal of Management, 14(3), 207-222. https://doi.org/10.1111/1467-8551.00375