Overcoming the discrepancies between clinical trials and real-world data of small cell lung cancer chemotherapy. A data-driven approach to learn across real-world evidence studies
Luca Marzano(1), Adam Darwich(1), Asaf Dan(2), Salomon Tendler(2,3), Jayanth Raghotama(1), Rolf Lewhenson(2), Luigi De Petris(2), Sebastiaan Meijer(1)
(1) Division of Health Informatics and Logistics, School of Engineering Sciences in Chemistry, Biotechnology and Health (CBH), KTH Royal Institute of Technology, Huddinge, Sweden, (2) Dept. of Oncology-Pathology, Karolinska Institutet and the lung oncology center, Karolinska University hospital, Stockholm, Sweden, (3) K7 Oncology- Pathology Department, Memorial Sloan Kettering Cancer Center, New York, United States of America
Objectives: Since the 21st Century Cures Act in 2015, the potential for translating between real-world data (RWD) and clinical trials (RCT) to improve regulatory decisions has gained attention [1, 2]. A growing body of research has focused on the so-called emulation of RCT control arms using RWD [3, 4], with the purpose of reproducing RCTs outcomes [5, 6]. The most common approach has been adjusting for confounders by matching real-world patients with RCT populations (e.g., using propensity score [7]). However, it has repeatedly been pointed out that there is a lack of replicability due to the high sensibility to undetected confounders [3, 5, 6]. Indeed, the simple alignment of patient characteristics has not resulted in similar outcomes between populations [5]. Furthermore, the bias of the potential differences between clinical practice and RCT protocols have been mentioned but not fully explored [5, 8].
Methods: We designed an approach that aims to explore how RWD could inform RCT design by systematically accounting for the known differences in population samples (randomisation) and operation (protocols and clinical practice). Thus, estimating the factors contributes on the translational gap, providing valuable insights for improving future study design.
We called the approach as SOMO as it is based on exploring the effects of Selection criteria (S), and Operations (O), on the replication of the Measurements of Outcome (MO) [9]. The SOMO pipeline can then be framed in the following steps:
- SOMO components are defined through an explorative analysis of the data, and the documentation with the involvement of clinical experts
- The translation gap for the MO is compared with the one after accounting for the detected S and O analysing the whole cohort
- The results from step 2 informs a mechanistic simulation approach of the randomisation to generate a synthetic control arm cohort formed by RWD and RCT patients. Different scenarios are simulated by intervening on the sampling by accounting for the aspects detected from the SOMO approach.
- The variation of outcomes of step 3 are compared and discussed, thus allowing the estimation of the robustness of eventual interventions in the trial design
SOMO was tested using three Phase III (n=872) and three Phase I-II (n=124) RCTs control arms shared on Project Data Sphere platform [10] and a Swedish RWD cohort of patients treated at Karolinska University Hospital in Stockholm (n=228) on survival of extensive-disease small cell lung cancer patients receiving platinum etoposide chemotherapy (n=1,224). The common patient variables were age, sex, brain metastasis (BM), Eastern Cooperative Oncology Group performance status (ECOG), the cohort from which the data were retrieved (STUDY), progression-free survival (PFS), Overall Survival (OS), and eventual censoring. For the real-world patient cohort, the 8th version of the TNM staging was available (STAGE) [11].
The randomisation was simulated by sampling patients from the RWD and the examined RCT to create the surrogated control arm. Then, the hazard ratios (HR) for OS and PFS between real-world and clinical trials cohorts were computed. The HR reference was the RWD cohort. The randomisation was repeated 100 times to estimate the variability of the outcomes.
The analysis was carried out accounting for eight selection criteria (S) and six operations and study protocol factors (O). The main techniques used during the analysis to create the scenarios were: sub-cohort stratification, logistic propensity score [7], and patients synthetic generation using the SMOTENC machine learning algorithm [12].
Results: We observed a significant difference in OS (HR: 0.65[0.55, 0.75]) and PFS (HR: 0.70[0.58, 0.85]) when comparing RCT with RWD cohorts. Well-known overrepresented patient groups (older age, females and ECOG 2) were also found in the RWD cohort.
The most impactful aspects for leveraging the discrepancy were ECOG, STAGE, and the survival overestimation due to the censoring of RCT patients. The translational gap was leveraged for groups of patients with ECOG of 2 for OS, and for all ECOG values for PFS. The RWD patients with TNM stage IVA matched with OS and PFS of RCTs when using traditional sub-stratification or generating synthetic sub-cohorts with machine learning. This was a novel result since the TNM staging was not yet developed when the trials were executed.
Adjusting the censoring led to interesting results. For what concerns OS, the translation gap limitation of traditional propensity score matching was leveraged when the method was adjusted by including the censoring as a covariate (HR: 1.1[0.87, 1.30]). Surprisingly, higher PFS for RWD was achieved when adjusting the propensity score, excluding censoring from the analysis, or generating synthetic RWD cohorts with same censoring mechanism (HR: 1.5[1.2, 1.8]).
Estimating the variability of the interventions allowed to explore eventual differences between the trials, and how these are closer to the RWD population. The STAGE contribute was high and the most common for all trials. For the trials with a relevant number of censored patients (more than 20% of the single control arm) this aspect was the most impactful one, and it was associated with a higher baseline survival gap.
Conclusions: The increasing presence of RWD in clinical trials constitute a natural development of study designs for regulatory decision-making. This work builds on current approaches and suggests areas of improvement for systematically accounting for differences in outcomes between study cohorts.
The SOMO approach aims to suggest the direction from which we can fill the current lack of consensus on the analytical process, and to pave the way to understand how we can use RWD to close the gap between internal validity and generalizability of clinical trials.
In the past, the added value of a mechanistic systems view of translation in drug development has been beneficial for other areas of model-informed drug development, such as quantitative in vitro-in vivo extrapolation and physiologically based pharmacokinetics of metabolic drug-drug interactions [13]. Developing systems approaches for real-world evidence could enable a similar learn-confirm cycle and learning across studies, where the represented systems include not only the patient, disease, and treatment, but also the operational context.
References:
[1] Schurman, B. The Framework for FDA’s Real-World Evidence Program. Applied Clinical Trials 28, 15–17 (2019).
[2] Dagenais, S., Russo, L., Madsen, A., Webster, J. & Becnel, L. Use of Real-World Evidence to Drive Drug Development Strategy and Inform Clinical Trial Design. Clinical Pharmacology & Therapeutics 111, 77–89 (2022).
[3] Wang, C. Y. et al. Uncontrolled Extensions of Clinical Trials and the Use of External Controls—Scoping Opportunities and Methods. Clinical Pharmacology & Therapeutics 111, 187–199 (2022).
[4] Wang, S. V. et al. Reproducibility of real-world evidence studies using clinical practice data to inform regulatory and coverage decisions. Nature Communications 13, 1–11 (2022).
[5] Jemielita, T. et al. Replication of Oncology Randomized Trial Results using Swedish Registry Real World-Data: A Feasibility Study. Clinical Pharmacology & Therapeutics 110, 1613–1621 (2021).
[6] Sreeram V. Ramagopalan, Ramagopalan, S. V., Ramagopalan, S. V., Simpson, A. & Sammon, C. J. Can real-world data really replace randomised clinical trials? BMC Medicine 18, 13–13 (2020).
[7] Webster-Clark, M. et al. Using propensity scores to estimate effects of treatment initiation decisions: State of the science. Statistics in Medicine 40, 1718–1735 (2021).
[8] Beaulieu-Jones, B. K. et al. Examining the Use of Real-World Evidence in the Regulatory Process. Clinical Pharmacology & Therapeutics 107, 843–852 (2020).
[9] Marzano, Luca, et al. "Overcoming the discrepancies between RCTs and real-world data by accounting for Selection criteria, Operations, and Measurements of Outcome (SOMO)." medRxiv(2024): 2024-01.
[10] Green, A. K. et al. The Project Data Sphere Initiative: Accelerating Cancer Research by Sharing Data. The Oncologist 20, 464-e20 (2015).
[11] Tendler, S. et al. Validation of the 8th TNM classification for small-cell lung cancer in a retrospective material from Sweden. Lung Cancer 120, 75–81 (2018).
[12] Chawla, N. V., Bowyer, K. W., Hall, L. O. & Kegelmeyer, W. P. SMOTE: Synthetic Minority Over-sampling Technique. Journal of Artificial Intelligence Research 16, 321–357 (2002).
[13] Wang, Y. et al. Model-Informed Drug Development: Current US Regulatory Practice and Future Considerations. Clinical Pharmacology & Therapeutics 105, 899–911 (2019).