Which study design should I choose for my evaluation study?

In theory

The Federal Institute for Drugs and Medical Devices (BfArM) requires a quantitative study to prove a positive healthcare effect of a DiGA; qualitative studies are not considered sufficient. At least one retrospective, comparative study (e.g. case-control study, retrospective cohort study, intra-individual comparison) must be conducted. "Retrospective" means that data already collected in the past (e.g. disease-related registry data, billing data from health insurance companies) are analyzed. "Comparative" implies that a before-and-after comparison within a patient group or a comparison against a control group is carried out. Instead of a retrospective approach, DiGA manufacturers can also conduct prospective, comparative studies (e.g. non-randomized or randomized controlled studies) and thus studies with a higher level of evidence. "Prospective" means that targeted data is collected to answer hypotheses established in advance. The use of the study design depends on the type of DiGA and the intended positive healthcare effect and is freely selectable by the manufacturer— providing a quantitative, comparative study is conducted.

In practice

As of June 20, 2024, 56 applications are listed in the DiGA directory— 35 of them permanently. All permanently listed DiGAs (n=35) have conducted a randomized controlled trial (RCT) to provide evidence of the positive healthcare effect. All provisionally listed DiGA (n=21) are also currently conducting an RCT.

Our experience

The feasibility of retrospective studies for the demonstration of a positive healthcare effect is made more difficult by the fact that often not all the necessary data are available or, in some cases, not of sufficient quality. In DiGA studies, it is particularly important that all relevant characteristics of the study population are recorded and documented to be able to exclude possible influences on the effects achieved (e.g. through co-interventions such as concomitant therapy or additionally used medication) (see DiGA FAQ question 5). Existing data sets often do not contain all these characteristics, which is why manufacturers rely on prospectively collected data.

The advantage of prospective DiGA studies is that the exact data required to prove the positive healthcare effect can be collected. The basic prerequisite for this is thorough planning of the study, which includes systematic research of all relevant patient characteristics, (influencing) variables, and relevant endpoints. Should it become apparent during the study that certain variables (e.g. additional support programs) could influence the positive healthcare effect, but were not collected, it is necessary to collect these variables retrospectively, evaluate them in subgroup analyses, and analyze their effect on the positive healthcare effect.

RCTs have the advantage over non-randomized studies. Randomization leads to better comparability between the two study groups investigated regarding known and unknown variables. RCTs are the best way to prove causal relationships between an intervention and an outcome. So, RCTs are considered the gold standard for evaluating the effectiveness of (digital) interventions. Challenges of DiGA-RCTs in proving a positive healthcare effect lie in the narrow definition of inclusion and exclusion criteria, which can make recruitment more difficult and limit the transferability of the results. The lack of subject blinding in studies with digital interventions can also pose a challenge, as the subjects know their group membership and thus also learn whether they have received the digital intervention or not; this in turn can lead to a loss of motivation and to premature withdrawal from the study (drop-out). One way to avoid this challenge is to design a placebo application. However, this increases the (financial) expense and is not the standard of care, which is why most DiGA manufacturers accept the lack of blinding.

The WIG2 Institute's DiGA-FAQ addresses the questions we are frequently asked by DiGA manufacturers.

As a scientific institute, we support DiGA manufacturers in planning, conducting, and analyzing the trial study. We ensure that the processing of the results meets the formal requirements of the Federal Institute for Drugs and Medical Devices (BfArM) and that it contributes to the evaluation concept for the DiGA-Fast-Trackin the best possible way. Our DiGA expert Dr. Tonio Sch?nfelder will be happy to answer any questions you may have.

DiGA-FAQ // Evaluation study // Question 6: Which study design should I choose for my evaluation study?

? zur deutschsprachigen Version des Artikels