Bikeshedding and Methodological Rigor in Literature Reviews

by Zeki Simsek , Ciaran Heavey and Brian Fox

Cyril N. Parkinson (1909–1993) was a British naval historian and author best known for his satirical observations on bureaucracy and organizational behavior, presented as ‘Parkinson’s Laws.’ In 1955, he introduced the now-famous Parkinson’s Law, which asserts that work expands to fill the time available for completion. A related principle, the Law of Triviality (1958)—often called ‘bikeshedding’—illustrates how complex, high-stakes organizational issues tend to discourage substantive discussion, while trivial and easily understood topics, such as the design of a bicycle shed, often invite comprehensive and sometimes futile debates.

Bikeshedding is a helpful metaphor for highlighting the tendency to focus on specific methodological details while overlooking the broader issue of rigor in conducting literature reviews. Although there has been significant progress in recognizing the value of rigorous reviews (Kunisch et al., 2023), too much emphasis is often placed on narrow methodological decisions and moves. This leads to a misallocation of time and effort, where the targeted focus overshadows more critical considerations, such as how to align research questions, review types, and execution designs. The result is that literature reviews may appear methodologically rigorous yet fail to offer a persuasive synthesis of past research and act as a generative catalyst for future research. Bikeshedding, in this sense, constitutes a form of intellectual distraction, diverting attention from the more substantive methodological issues that literature reviews must address to ensure rigor and impact.

This observation is grounded in our experience within the academic publishing ecosystem, where we have served as authors, reviewers, and editors of review manuscripts. In addition, we conducted two studies to examine how this phenomenon manifests in published review articles in the management field (Simsek, Z., Fox, & Heavey, 2023; Simsek, Fox, Heavey, & Liu, 2024). Together, these studies reveal patterns, strengths, and weaknesses in how literature reviews have been designed and executed.

We examined two interrelated concepts in those studies to make sense of specific practices, tensions, and tradeoffs. The first is systematicity: an encompassing orientation toward applying explicit methods in the practice of literature reviews. The second, an extension of our thinking regarding systematicity, is methodological rigor: the reasoned alignment and explicitness of key methodological choices to make coherent and credible knowledge claims.??

Below, we offer a concise overview of the studies. We will outline each study’s objective, discuss how it advances methodological rigor, and highlight the relevant findings. Although this summary captures the essential takeaways, we encourage readers to consult the original articles for a deeper and more comprehensive understanding.

Study 1: Systematicity

This paper was motivated by the Special Issue at Organizational Research Methods on Rigorous and Relevant Reviews (Kunisch et al., 2023). Inspired by the increasing adoption of systematic review methods in management research, along with some gentle (almost!) nudging by our Editors and reviewers during the review process, ?we introduced the concept of systematicity, observing that “one of the more comprehensive reviews of the idea is found not in literature reviews but in a contribution to the general philosophy of science—Hoyningen-Huene’s (2013) discussion on systematicity and nature of science” (p. 294). We defined systematicity as “an encompassing orientation toward the application of explicit methods in the practice of literature reviews, informed by the principles of transparency, coverage, saturation, connectedness, universalism, and coherence” (p. 294).

We argued that such an encompassing orientation could be witnessed across all seven practices of literature reviews – envisioning, explicating, executing, evaluating, encoding, elaborating, and expositing. We then examined that expectation using a sample of 165 review articles. One of the surprising findings was the variability of systematicity across the practices. In some cases, systematicity in one practice area was not predictive of systematicity in another (e.g., a 0.13 correlation between explicating and elaborating). In contrast, in other cases, they were strongly coupled (e.g., a 0.72 correlation between explicating and elaborating). The results suggested that authors were not being ‘systematically systematic.’ They may have been sweating the detail in some practice areas but not in others. We observed a similar pattern concerning the principles of systematicity. Although we could not conclusively establish the root cause due to the exploratory nature of our examination, the finding may be a bikeshedding manifestation. As authors go deeper and deeper down certain methodological rabbit holes, they lose sight of the broader picture. Or they might feel the pressure (actual or anticipatory) to adhere to specific guidelines or the fear of methodological criticism.

With the benefit of hindsight, it is now more apparent to us that the “ambitiousness” of the review significantly influenced its level of systematicity. Classifying reviews based on their intended contribution, we observed that those seeking to make more ambitious contributions, such as introducing new concepts, frameworks, or theories, also exhibited a higher level of systematicity on average. We could not establish the causality of this association – does theoretical contribution drive attention to systematicity, or does systematicity enable more advanced and ambitious contributions?

The study presents some indirect exploratory insights, though. We found that the number of systematicity practices is associated with scholarly impact, measured as the percentage of downloads achieving a citation. One speculative conjecture we entertained was that reviewers focused on systematicity are more equipped to synthesize knowledge elements, leading to richer insights – an idea inspired by the concept of innovation as a combinatory search process. However, it is equally plausible that reviews that seek to make greater contributions are held to a higher systematicity standard.

Study 2: Methodological Rigor

After completing the study, we had more questions than answers—and a newfound appreciation for coffee’s role in reviewing review research! To better understand the inter-relationship between the systematicity and overall construction of reviews, we, in a second study, performed a scoping review of articles in the Journal of Management ( a final sample of 199 review articles) across five thematic elements: (1)? review purpose—research question and theoretical aim; (2) type—review approach such as systematic, rapid, and mixed method; (3) design—the absence or presence of an overarching methodological approach; (4) execution—manifest review practices; and (5) internal alignment—the extent to which review purpose, type, design, and execution are mutually supportive, reinforcing and tightly coupled.??

The results of our scoping review of 199 articles provide valuable insights for enhancing the rigor and quality of review articles, highlighting several ‘low-hanging fruit’ areas for immediate improvement, too. Specifically, most articles articulated the purpose of their review, but fewer were precise in articulating a question or set of questions the review sought to tackle. An increasing proportion of studies provide a methods section, but few articles explained the underlying reasons for their study design choices. Consistent with Study 1, review articles were uneven in reporting methodological steps performed (e.g., search strategy, how articles were selected for inclusions, and how data was extracted and coded from articles). Most importantly, few papers explicitly connected the purpose of their review to their chosen type or form of the review (e.g., traditional narrative reviews vs. systematic literature reviews) or demonstrated how the reported practices in the review were aligned with this overall methodological approach. Said differently, few papers simultaneously and explicitly demonstrated, through the words written in the final manuscript, that a) their review purpose drove the selection of a review type well suited to achieve that purpose and b) that the actual steps performed followed from this selected design rather than being a potpourri of discrete steps deemed necessary to complete.

For us, the findings of the scoping review, in conjunction with those of Study 1, raised an important question concerning the meaning of rigor in management research reviews – should rigor be measured and evaluated in terms of the transparency of reporting practices; the sophistication of the methodologies employed; or the systematicity with which methodological steps are followed; or something else entirely?

In the study’s discussion, we addressed that question inductively. Our comprehensive review of existing conceptualizations and approaches to rigor in management research led us to conclude that methodological rigor in review research goes beyond merely adhering to prescribed procedures and steps. Without a coherent and integrated approach to rigor, pursuing isolated methodological steps can become trivial while creating an illusion of rigor.

As discussed in the study, we do not attribute the observed aggregate divergences in reported practices to the authors’ carelessness or oversight. Instead, these divergences seem to reflect a poorly defined and specified conception of methodological rigor as a fundamental objective. Thus, based on our experience in two studies and a survey of rigor conceptions in the management discipline, we proposed a conception of methodological rigor for review research as: ?“the reasoned alignment and explicitness of key methodological choices aimed at making coherent and credible knowledge claims” (p.21).

Even as we encourage the reader to peruse the study for greater details regarding the effort and research grounding leading to this conception, we point out here that the reasoned alignment of methodological choices towards coherent and credible knowledge claims is the core consideration. It stems from researchers’ careful and deliberate methodological choices about review purpose, type, design, and execution.

The concept of reasoned alignment is in line with what Edmondson and McManus (2007) termed ‘methodological fit’ in the research design —the “internal consistency among elements of a research project—research question, prior work, research design, and theoretical contribution” (p. 1155). It addresses how the current state of theory and literature influences the appropriateness of hybrid research strategies and other methodological decisions, ensuring coherence among all elements of a research project (Edmondson & McManus, 2007, p. 1157). One fundamental expectation in methodological fit is that the three levels of knowledge (nascent, mature, and intermediate) found in prior research roughly correspond to three types of research methodologies (qualitative, quantitative, and mixed) (Edmondson & McManus, 2007).

Although there are some parallels between reasoned alignment and methodological fit, the concept of fit is fundamentally concerned with enhancing researchers’ ability to “align theory and methods in field research,” particularly when using mixed-method designs (Edmondson & McManus, 2007, p. 1156). In contrast, reasoned alignment focuses on the architecture of review articles. Rather than examining how theory and methods align in field research, reasoned alignment focuses on how review articles might support credible and coherent knowledge claims through careful coordination and interrelation of review purpose, type, design, and execution.

Outro

Bikeshedding, which illustrates the tendency to focus disproportionately on minor matters while neglecting critical ones, is a helpful metaphor for highlighting the challenges of achieving methodological rigor in review research. It prompts researchers to reflect on the pitfalls of mistaking procedural complexity for genuine rigor.

We fear that an overemphasis on methodological steps and fixation on an ever-growing menu of narrow methodological details will crowd out attention to more critical questions, such as the review purpose and what review type/design may best serve that purpose. The drive towards incorporating elements of systematic review methodologies (e.g., Tranfield et al., 2003) has made the review process more transparent and replicable and can inspire increased confidence and trust in the conclusions drawn. However, it is essential to recognize that such efforts alone are insufficient. Much like how tactics are ineffectual without strategy alignment, specific methodological steps and moves, on their own, cannot tell us much about what ultimately matters and what one should be doing to move towards what matters in literature reviews. In our experience, the real bottleneck to producing rigorous reviews is not the ever-increasing sophistication of discrete methodological steps but the overall strategy or conceptual framework that needs to guide and inform the overall review.

“We should treat all the trivial things of life seriously, and all the serious things of life with sincere and studied triviality,” Oscar Wilde famously quipped. Although the rigorous pursuit of new knowledge is of utmost importance, a playful and studied triviality in examining our methodologies can reveal unexpected insights and breakthroughs. And if science is indeed a cumulative endeavor, what could be more fundamental than scrutinizing the processes and practices through which we synthesize knowledge claims? As we have discussed elsewhere (Simsek, Heavey, Fox, & Yu, 2022), review articles can serve as springboards for compelling questions toward that end.

?

References

Edmondson, A. C., & Mcmanus, S. E. (2007). Methodological fit in management field research. Academy of Management Review, 32(4), 1155-1179. https://doi.org/10.5465/AMR.2007.26586086

Hoyningen-Huene, P. (2013). Systematicity: The nature of science. Oxford University Press.

Kunisch, S., Denyer, D., Bartunek, J. M., Menz, M., & Cardinal, L. B. (2023). Review research as scientific inquiry. Organizational Research Methods, 26(1), 3-45. https://doi.org/10.1177/10944281221127292

Parkinson, C. N. 1958. Parkinson’s Law or the Pursuit of Progress. London: John Murray.

Simsek, Z., Fox, B. C., Heavey, C., & Liu, S. (2024). Methodological rigor in management research reviews. Journal of Management. https://doi.org/10.1177/01492063241237222

Simsek, Z., Fox, B., & Heavey, C. (2023). Systematicity in organizational research literature reviews: A framework and assessment.?Organizational Research Methods,?26(2), 292-321. https://doi.org/10.1177/10944281211008652

Simsek, Z., Heavey, C., Fox, B. C., & Yu, T. (2022). Compelling questions in research: Seeing what everybody has seen and thinking what nobody has thought.?Journal of Management,?48(6), 1347-1365. https://doi.org/10.1177/01492063211073068

Tranfield, D., Denyer, D., & Smart, P. (2003). Towards a methodology for developing evidence‐informed management knowledge by means of systematic review.?British Journal of Management,?14(3), 207-222. https://doi.org/10.1111/1467-8551.00375