How to account for Delphi limitations

Although methodologically powerful, the Delphi method is not without limitations. Here are four ways to address Delphi weaknesses.

First, it is important to ensure that Delphi is the right method for your project and that the protocol is informed by the literature review and/or expert input. Remember that you do not need to convene a Delphi panel if your literature review provided strong enough evidence. Using Delphi to “rubber stamp” a way forward is not what the method is supposed to help you do.??

Second, Delphi is a “small N” methodology, which means that you should not oversell your findings. You also need to determine if your sample is adequate. Delphi samples are never random because participants are carefully chosen based on their expertise. However, Delphi samples should be large enough to represent a range of relevant perspectives on the topic; you should also make sure that you have not “lost” a certain group or type of participants during the data collection. If you plan to use any significance tests (and there are different perspectives on whether you should even consider them in Delphi studies), make sure to use the right significance test for the type of data you have.??

Third, quality of Delphi panel results depends on the expertise of your participants. You should ensure that all participants have relevant knowledge, do not have conflicts of interest, and are likely to approach the issue from different perspectives. If your panel is diverse, make sure that you have roughly the same number of representatives of each group.

Finally, Delphi has been criticized for low replicability of its results and for being sensitive to who the panelists are. To address this limitation, I prefer to conduct at least 2 concurrently run panels with similar composition of panelists who are randomly assigned to a given panel. Comparing panel results and moving forward with the items that both panels agreed on increases the confidence in the results. Check out my PCORI report that details how to do this.

You can always consult our report that provides additional information on ways to address Delphi method’s limitations.

Want to know more about the Delphi method? Check out my other articles on:

1) What Delphi Method is. This article describes four main characteristics of the Delphi method, including iteration, anonymity, feedback on group’s responses, and statistical aggregation of responses to determine the existence of consensus.

2) How to design Delphi studies. This article talks about the types of projects that could benefit from the Delphi method, discusses key Delphi project design characteristics, and explains how to choose panelists.

3) How to ask good Delphi questions. This article provides guidance on how to develop Delphi items, questions, and response scales.

4) How to administer Delphi panels. This article explains how to conduct virtual Delphi panels.

5) How to minimize participant attrition in Delphi panels. This article presents strategies for increasing participant engagement and retention in virtual Delphi panels.

6) How to analyze and report Delphi results. In this article. This article explains how to analyze and present quantitative and qualitative data collected in Delphi studies.