How to Design a Delphi Study

Now that you know what #Delphi method is (if you still don’t, check out my previous article) and are ready to design your Delphi study, you should ask yourself 5 important questions.

1.?What are the goals of my Delphi study?

As originally envisioned, the Delphi method was created to help researchers explore the existence of consensus among a small group of handpicked experts on a topic. Asking experts the same questions more than once and allowing them to see how others have answered these questions can help develop consensus. Because experts participate anonymously, they are more likely to provide more objective responses informed by their professional expertise and perspectives of other handpicked experts whose identity is not known to them. #Consensus developed this way is treated as a form of evidence, a reliable prediction, or even “truth” in uncertain situations. Not all Delphi studies, however, include only experts or have consensus as their ultimate goal. Nowadays, Delphi is also used as stakeholder engagement methodology that allows for exploring differences and similarities between their perspectives after they have been exposed to the perspectives of others. ?If a group of invited experts is large and diverse or consists of stakeholders with different perspectives, learning about things they disagree on can be as valuable as knowing about things they agree on. Therefore, Delphi can help explore the existence of consensus as well as dissensus. Read this article if you want to learn more about dissensus Delphi.

2.?Do I really need to use the Delphi method?

The answer to this question may be “No.” Delphi is a forward-looking methodology that helps deal with the unknown. In backward-looking studies, those that measure personal attitudes or focus on things that have already happened, such as those that want to know how stakeholders feel about an already implemented policy, surveys work much better. However, if the goal is to identify ways to improve a policy moving forward, the Delphi method can be useful because it can help prioritize desired changes and identify areas of agreement and disagreement among stakeholders. The Delphi method is also useful if you think that exposing your participants to the perspectives of others can change their perspectives.?

3. Who should I invite to participate?

The Delphi method was developed with experts in mind. However, Olaf Helmer, one of the original developers of the Delphi method, argued that “extreme specialists,” or experts with narrow areas of expertise, may not be the best Delphi panelists. Instead, he advocated for engaging those who have broad expertise and can take a bigger picture into account. That is why diversity of perspectives and experiences is crucial for assembling a good Delphi panel. The number of participants should be determined by the mode of data collection and the way the interaction among participants will take place. In-person Delphi panels are typically limited to 9-18 people. In contrast, virtual panels that do not have a discussion component can include hundreds of participants. Virtual panels with an online discussion component, such as those administered using RAND-developed ExpertLens?, typically include 40-60 participants. Conducting larger panels online allows for expanding the type and number of participants. This means that for a healthcare-focused panel, you can invite clinicians and patients to share their perspectives because their participation can remain strictly (or partially) anonymous. To learn more about panel size and composition in virtual Delphi panels, read this BMC Medical Research Methodology article. If you feel like you want to dig deeper into the impact of panel size and composition on panel results, read this Health Expectations article.

4.?How many Delphi rounds should there be?

The answer to this question is: “It depends.” Some pre-determine the number of Delphi rounds ahead of time, while others tell participants that there will be as many rounds as is needed to reach consensus. I personally prefer to tell participants what they got themselves into when I consent them. Pre-specifying the number of rounds ahead of time helps identify areas of agreement and disagreement in diverse panels without “forcing” consensus. Therefore, I prefer to say that Delphi is a method for exploring the existence of consensus, rather than a consensus-development methodology. Research shows that the more rounds you add, the more people you will lose, and that two rounds of questions is typically enough to determine if consensus exists.

5.?What should I do before the start of the Delphi panel?

Before starting the first round, you should make a number of important decisions about:

• Presenting a summary of existing evidence will be shared with participants to help inform their answers. Literature reviews (or evidence syntheses) are very common in health and medical research fields but are not typically used in Delphi studies by other disciplines. When I do not think there is a need for a literature review, I often ask participants about their thoughts on the study topic in an open-ended manner before the first rating round.
• Determining your panel size and composition. You should consider your study goals, mode of engagement, number of available potential participants, and your budget in deciding who will be invited to serve as your panelists. ?
• Conducting a pilot test with a small group of participants similar to those you plan to invite is important to ensure that your data collection process, instructions, and questions are clear. Administering the pilot several weeks or even months before the start of the actual Delphi study should provide enough time to make necessary changes.

Need more guidance on how to design Delphi panels, check out our Delphi manual, leave a comment below, or DM me.

Also check out my other articles on:

1) What Delphi Method is. This article describes four main characteristics of the Delphi method, including iteration, anonymity, feedback on group’s responses, and statistical aggregation of responses to determine the existence of consensus.

2) How to ask good Delphi questions. This article provides guidance on how to develop Delphi items, questions, and response scales.

3) How to administer Delphi panels. This article explains how to conduct virtual Delphi panels.

4) How to minimize participant attrition in Delphi panels. This article presents strategies for increasing participant engagement and retention in virtual Delphi panels.

5) How to analyze and report Delphi results. In this article. This article explains how to analyze and present quantitative and qualitative data collected in Delphi studies.

6) How to account for Delphi limitations. This article describes 4 strategies for addressing Delphi limitations as you design your Delphi panels.

?