Selection of Effect Modifiers and Prognostic Factors for Indirect Treatment Comparisons

In the world of comparative effectiveness research, indirect treatment comparisons (ITCs) are essential when head-to-head randomized controlled trials (RCTs) are unavailable. ITCs, including methods such as network meta-analysis (NMA), matching-adjusted indirect comparisons (MAIC), and simulated treatment comparisons (STC), help assess the relative efficacy and safety of competing treatments. A key challenge in performing ITCs is selecting the right effect modifiers and prognostic factors that can significantly impact the comparability of patient populations across studies.

This article explores different approaches that can be used to identify and select these critical variables.

Understanding Effect Modifiers and Prognostic Factors

Before diving into strategies for selecting these variables, it’s important to clarify the difference between effect modifiers and prognostic factors:

• Effect Modifiers are variables that influence the magnitude of the treatment effect. They cause treatment effects to differ across subgroups (e.g., age, gender, disease severity).
• Prognostic Factors are variables that affect outcomes regardless of treatment. These factors influence patients’ baseline prognosis (e.g., comorbidities, baseline disease characteristics) but do not directly modify the treatment effect.

Failing to properly account for these factors can introduce bias into ITCs, leading to inaccurate conclusions about a treatment’s relative efficacy and safety. Below, let's review some considerations for selecting these variables.

Approach 1: Systematic Literature Review

A systematic literature review (SLR) is often the starting point for identifying potential effect modifiers and prognostic factors. This approach helps researchers gather data from previously published trials, observational studies, and clinical guidelines.

• Steps: Conduct a comprehensive search of relevant RCTs and observational studies on the treatments of interest. Identify common baseline characteristics and subgroup analyses performed in previous studies. Look for consistent effect modifiers in past studies where treatment effects differed across subgroups (e.g., older vs. younger patients, varying disease stages). Document prognostic factors that consistently influence baseline outcomes, regardless of treatment.

How do SLRs help?

• They provide a broad overview of potential factors from established evidence.
• They enable identification of commonly agreed-upon effect modifiers and prognostic factors used in clinical practice.
• Inspection of prior ITCs in the clinical space may provide valuable information.

SLR Challenges?

• Existing literature may not always align with the exact populations or treatments under investigation.
• Limited availability of subgroup data or inadequate subgroup reporting within trial publications may obscure important modifiers.
• They're not short to perform! However, as large language models rapidly enhance our ability to reduce the time to complete SLRs, this barrier will become smaller in size.

Approach 2: Expert Consultation and Delphi Method

When the literature is unclear or insufficient, consulting clinical experts in the therapeutic area can help identify key effect modifiers and prognostic factors. The Delphi method—a structured communication technique—can also be used to reach consensus among a panel of experts.

• Steps: Assemble a panel of clinical and methodological experts. Conduct iterative rounds of surveys, asking experts to list and rank potential effect modifiers and prognostic factors based on their experience. Use feedback and revise lists, arriving at a consensus on the most important variables.

Advantages of a Delphi Approach?

• Leverages expert clinical judgment, especially when data is scarce or ambiguous.
• Provides insight into real-world factors that may not be well-documented in trials but are significant in clinical practice.

Delphi Challenges?

• Subjectivity: Experts may have differing opinions based on their clinical experience, potentially leading to variability in both factor selection as well as perspectives on the importance across experts.
• Time-intensive: The Delphi process may require multiple rounds to reach consensus, which can pose a challenge when liaising with busy clinical experts (whether via an online survey or virtual/face-to-face meetings

Approach 3: Data-Driven Selection Using Individual Patient Data (IPD)

Individual patient data (IPD) allows for more rigorous and data-driven selection of effect modifiers and prognostic factors. This approach is often used when IPD are available from the clinical trials involved in the ITC.

• Steps: Access IPD from trials evaluating the treatments of interest. Perform subgroup analyses to identify interactions between baseline characteristics and treatment effects (for effect modifiers). Use regression models to assess how prognostic factors impact baseline outcomes across all patients. Identify interactions that significantly modify the treatment effect (e.g., age-treatment interaction) and baseline predictors of outcome.

Advantages of data-driven approach?

• Allows for precise identification of effect modifiers and prognostic factors based on actual trial data.
• IPD enables more accurate adjustment and harmonization of patient characteristics across studies, minimizing bias.

Challenges of doing so?

• IPD is often difficult to obtain. IPD analyses require advanced statistical expertise, and models may become complex when many variables are involved.
• Some may flag concerns associated with potential over-fitting if based on the sponsor’s own trial data. Concerns of bias may arise if based upon the sponsor’s trial population of narrow enrollment criteria. Concerns of limited covariate scope may also be an issue dependent upon the diversity of traits captured within the trial.

Approach 4: Pre-Specified Lists Based on Clinical Guidelines and Regulatory Recommendations

For common therapeutic areas, clinical guidelines or regulatory guidance documents (e.g., by the FDA or EMA) may already outline pre-specified effect modifiers and prognostic factors that should be considered.

• Steps: Consult clinical guidelines specific to the disease area (e.g., oncology, cardiovascular, or respiratory) that recommend key factors for stratifying patient populations. Review regulatory guidelines from bodies like the FDA or EMA, which may provide specific recommendations on patient characteristics to consider in trials and comparative studies.

Advantages?

• Ensures compliance with regulatory expectations and clinical practice standards.
• Saves time, as established guidelines often provide clear recommendations on relevant factors.

Challenges?

• Guidelines may not cover all relevant treatments or novel therapies.
• Pre-specified lists may not be exhaustive or applicable to the specific context of the ITC, leading to oversights

Approach 5: Real-World Data and Pragmatic Trials

When RCT data is sparse or populations in trials do not reflect the real-world population, researchers may consider turning to real-world data (RWD) and pragmatic trials to explore effect modifiers and prognostic factors.

• Steps: Access real-world datasets (e.g., electronic health records, insurance claims, or disease registries) for the population of interest. Identify baseline characteristics from these datasets that predict outcomes or modify treatment effects (e.g., age, gender, comorbidities). Use statistical models to validate which factors are significant as effect modifiers or prognostic variables.

Advantages?

• RWD allows for broader insights into how treatments work across diverse populations, not just the narrow populations enrolled in RCTs.
• Pragmatic trials provide insights into how factors play out in everyday clinical settings.

Challenges?

• RWD is often subject to bias, missing data, or confounding variables, making interpretation of results more complex.
• Adjustments for confounders and biases are necessary, and this can complicate analyses.

Key Considerations for Choosing Effect Modifiers and Prognostic Factors

When undertaking ITCs, it’s important to keep the following considerations in mind when selecting effect modifiers and prognostic factors:

1. Statistical Power: Ensure that there is sufficient data to reliably assess the chosen variables. Selecting too many effect modifiers can reduce statistical power and introduce noise.
2. Clinical Relevance: Focus on variables that are clinically meaningful and aligned with treatment goals. Factors like age, disease severity, or comorbidities often make the most impact on treatment outcomes.
3. Consistency Across Studies: Look for variables that are consistently reported across the studies involved in the ITC. Inconsistency can complicate the ability to harmonize data and compare treatment effects.
4. Regulatory Compliance: Align your approach with regulatory guidance to ensure that the analysis meets the standards for submission to regulatory bodies like the FDA or EMA.

Should I Use Multiple Approaches to Identification?

Using multiple strategies to covariate selection can lend strength and improve buy-in to the subsequent ITCs that are performed. This is due to:

1. A Broader Perspective: By utilizing systematic reviews, the ITC producer ensures they capture the full range of characteristics previously reported in clinical trials. This guarantees that their set is grounded in existing evidence and reflects what’s already deemed essential in the field. Engaging expert clinicians via a Delphi process helps refine and prioritize characteristics that are relevant in clinical practice. This consensus method can ensure buy-in from the scientific community and clinical stakeholders. If pursued, patient and caregiver engagement adds an essential real-world dimension. Patients often highlight variables that may be overlooked by researchers but are crucial to understanding treatment impact, enhancing the real-world applicability of the CPCS.
2. Demonstration of Rigor and Inclusivity: Combining methodologies signals to regulators, healthcare providers, and payers that the company has taken a rigorous, inclusive approach. For example, demonstrating that you’ve consulted clinical experts, patients, and systematically reviewed literature gives more credibility to your CPCS, making it more likely to be accepted by healthcare decision-makers and regulators. Additionally, using multiple approaches can lead to a more comprehensive set of patient characteristics, improving the generalizability of trial findings and ensuring that a wide variety of patient populations are represented.

What Can Be Done to Maximize the Awareness of all this Hard Work?

Undertaking the decision to invest in any of these approaches to covariate identification represents a significant time commitment, and even more so if multiple strategies are used. To maximize return on this investment of time, a variety of strategies might be considered.

1. Produce Transparent Reporting and Publication of the Effort: Publicly sharing the methodology through peer-reviewed publications, conference presentations, and/or open-access repositories ?will demonstrate the thoroughness of the approach. This can also ensure wider acceptance and use of a core patient characteristic set across future trials, real-world studies and ITCs. Documenting each stage of the process—from literature review and expert panels to patient engagement—capture and share the processes followed to show that every stakeholder’s perspective has been considered. This transparency can help build trust among regulatory bodies, payers, and clinicians.
2. Incorporate Activities into Clinical Trial Protocols: Including the core patient characteristic set in trial protocols for regulatory submissions can further demonstrate commitment to robust, patient-centered evidence. Regulatory agencies are increasingly focusing on patient-centric measures, so incorporating these into submissions can lead to better interactions with agencies like the FDA and EMA. Highlight how the CPCS improves trial reproducibility and comparability across studies to increase appeal to healthcare decision-makers who need to synthesize evidence across trials for health technology assessments.
3. Collaboration with Patient Advocacy Groups: Building partnerships with patient advocacy groups during development of core patient characteristic sets and reporting their involvement in communications or marketing can help enhance patient buy-in. Such collaborations can demonstrate commitment to patient-centered research and can lead to broader acceptance and use in clinical practice.
4. Leverage Digital Platforms for Engagement to Spread Word: Leveraging of digital platforms such as LinkedIn to share insights and updates about the development of core patient characteristic sets can increase awareness and interest. Engaging healthcare professionals, HTA researchers, and clinicians through these platforms by publishing updates about the methodology, findings, and potential impact of the work can further establish credibility.

Conclusion

The selection of effect modifiers and prognostic factors is vital to the success of indirect treatment comparisons. Whether relying on systematic literature reviews, expert opinion, real-world data, or individual patient data, researchers and drug manufacturers must carefully balance the need for robust, clinically relevant data while minimizing bias.

By thoughtfully selecting these variables, researchers and manufacturers alike can improve the credibility and reliability of their ITCs, leading to more informed treatment decisions and better outcomes for patients.

References Of Potential Interest

A methodological review of population-adjusted indirect comparisons reveals inconsistent reporting and suggests publication bias. J Clin Epidemiol. 2023 Nov:163:1-10. doi: 10.1016/j.jclinepi.2023.09.004. Epub 2023 Sep 16. Arnaud Serret-Larmande, Belkacem Zenati, Agnès Dechartres, Jér?me Lambert, David Hajage.

Development of minimum reporting sets of patient characteristics in epidemiological research: A methodological systematic review. Research Methods in Medicine & Health Sciences (2023). Volume 5, Issue 2. https://doi.org/10.1177/26320843231191777. My Luong Vuong, Pham Hien Trang Tu, Tat-Thang Vo.

Increasing transparency in indirect treatment comparisons: is selecting effect modifiers the missing part of the puzzle? A review of methodological approaches and critical considerations. Journal of Comparative Effectiveness Research (2023); 12(10). https://doi.org/10.57264/cer-2023-0046. : Andreas Freitag, Laura Gurskyte, Grammati Sarri.