Partial residual plots (PRPs) as an integrated model diagnostic tool in model-based meta-analysis (MBMA)
John Maringwa (1), Paul Matthias Diderichsen (2) and Chandni Valiathan (3)
(1) Johnson & Johnson, The Netherlands, (2) Certara USA, Inc., Princeton, NJ, USA, (3) Johnson & Johnson, United States
Objectives:
Diagnostic plots are often used to assess model fit, compare models, and elucidate correlations between model predictions and covariates. For Model-based Meta-analysis (MBMA), forest plots comparing observed data vs model predictions per study arm [1, 2], posterior and visual predictive checks (VPC) [3, 4], and plots of residuals vs covariates [5] are used. Methods comparing model predictions to observed data, stratified by levels of covariates, offer limited insights when strata are small. Residual-based plots typically reflect model misspecification rather than effect of covariates.
To address some of these shortcomings, the use of partial residuals plots (PRPs) [6, 7] as a diagnostic in MBMA is explored. PRPs show the effect of one covariate on response after normalizing for all other covariates [8]. Data can be shown with appropriate normalization without stratifying. Placebo response is often not of main interest in MBMA and may be described by an unstructured model. In these scenarios, PRPs become convenient to normalize data to a reference placebo response used for model predictions and have appropriate comparisons. Conceptually, this approach is similar to the prediction-corrected VPC in pharmacometrics [9].
This work aims to illustrate the use and provide a better understanding of PRPs as applied in MBMA. Mathematical derivations based on a routinely used dose-response model were used to illustrate ideas, followed by an example application.
Objectives are:
- Provide mathematical derivations based on a known model structure to illustrate the concept of PRPs as a model diagnostic tool
- Illustrate practical application of PRPs
Methods:
Literature data from 15 placebo-controlled studies of anti-depression treatments venlafaxine (ven, 9 trials, 1221 pts), and fluoxetine (flu, 6 trials, 882 pts), and 1123 placebo-treated pts was used. Dose-ranging studies enabled assessment of dose-response. Change from baseline in the Hamilton Depression Rating (HAMD) scale, the clinical endpoint of interest, was analyzed at the respective primary study timepoints.
The model is: Yij=f(eo,d,B)+eij
where f(eo,d,B)=eoi+Emaxk*{1+a*(B-b)}*dk/(ED50k+dk)
and Yij:outcome in arm j of trial i, eoi:non-parametric trial-specific placebo response, Emaxk: drug k maximal effect, dk:dose for drug k with potency ED50k, B:HAMD baseline, a:effect of centered B and eij:residuals.
Outcome-related reported standard errors were used as weights. A constant dose effect for flu and a dose-response for ven were identified.
The root mean square error (RMSE) was used to assess the discrepancy between model predictions and observations.
Results:
Let Y*ij=f*(eo,d,B) be the full model prediction and e*ij be the residuals, based on estimated parameters: e*ij=Yij-f*(eo,d,B).
To isolate the relationship between the response and dose, Y*(d), independently of the effect of covariates, eo and Bare fixed to their respective typical values eofix and Bfix . Comparing Y*(d) to observed data or summaries derived thereof may not be appropriate since each point inherently reflects a different eo and B, while Y*(d) reflects typical values. An appealing option is to create a “normalized” observation reflecting properties of Y*(d) via e*ij. Prediction on the actual data yields f*(eofix,d,Bfix).
Adding back full model residuals e*ij to f* creates a “normalized observation” Ynij. Intuitively, e*ij express what we do not know about Yij given the full model. The following shows that Ynij is indeed an observation normalized for other effects in the model, except dose.
Ynij=f*(eofix,d,Bfix)+ e*ij
Ynij=f*(eofix,d,Bfix)+{Yij-f*(eo,d,B)}
Ynij=Yij-{f*(eo,d,B)-f*(eofix,d,Bfix)}
Ynij=Yij-f*(eo,B)
where f*(eo,B) reflects the difference between actual and fixed eo and B.
Fixing eo and B to typical values, predictions Y*(d) resulted in RMSEs of flu (2.74) and ven (2.21) compared to observed data and 1.16 and 1.10 when compared to normalized points, suggesting a better fit in favor of normalized data. This is expected. Some observed data may exhibit properties that differ from those of the model prediction. PRPs normalize for this.
Conclusions:
Results showed that normalized data provide a “like-to-like” comparison with model predictions when assessing the effect of one variable (d) normalizing for other covariates (eo and B), unlike using observed data. PRPs provide a robust integrated diagnostic MBMA tool that uses all data to show the correlation between response and any variable while controlling for covariates included in the model.
References:
[1] Gross, J. L., Rogers, J., Polhamus, D., Gillespie, W., Friedrich, C., Gong, Y., Monz, B. U., Patel, S., Staab, A., & Retlich, S. (2013). A novel model-based meta-analysis to indirectly estimate the comparative efficacy of two medications: an example using DPP-4 inhibitors, sitagliptin and linagliptin, in treatment of type 2 diabetes mellitus. BMJ open, 3(3), e001844. https://doi.org/10.1136/bmjopen-2012-001844
[2] Li, H. Q., Xu, J. Y., Jin, L., & Xin, J. L. (2015). Utilization of model-based meta-analysis to delineate the net efficacy of taspoglutide from the response of placebo in clinical trials. Saudi pharmaceutical journal : SPJ : the official publication of the Saudi Pharmaceutical Society, 23(3), 241–249. https://doi.org/10.1016/j.jsps.2014.11.008
[3] Chan, P., Peskov, K., & Song, X. (2022). Applications of Model-Based Meta-Analysis in Drug Development. Pharmaceutical research, 39(8), 1761–1777. https://doi.org/10.1007/s11095-022-03201-5
[4] Berg, A., Clary, J., Hanna, D., Nuermberger, E., Lenaerts, A., Ammerman, N., Ramey, M., Hartley, D., & Hermann, D. (2022). Model-Based Meta-Analysis of Relapsing Mouse Model Studies from the Critical Path to Tuberculosis Drug Regimens Initiative Database. Antimicrobial agents and chemotherapy, 66(3), e0179321. https://doi.org/10.1128/AAC.01793-21
[5] Witjes, H., Khatri, A., Diderichsen, P. M., Mandema, J., & Othman, A. A. (2020). Meta-Analyses of Clinical Efficacy of Risankizumab and Adalimumab in Chronic Plaque Psoriasis: Supporting Evidence of Risankizumab Superiority. Clinical pharmacology and therapeutics, 107(2), 435–442. https://doi.org/10.1002/cpt.1624
[6] Cook and Weisberg (1982). Residuals and Influence in Regression. Chapman and Hall.
[7] Draper and Smith (1998). Applied Regression Analysis (3rd ed.). John Wiley.
[8] Maringwa, J., Sardu, M. L., Hang, Y., Czerniak, R., Vishnubhotla, M., Vakilynejad, M., & Pfister, M. (2021). Characterizing Effects of Antidiabetic Drugs on Heart Rate, Systolic and Diastolic Blood Pressure. Clinical pharmacology and therapeutics, 109(6), 1583–1592. https://doi.org/10.1002/cpt.2130
[9] Bergstrand, M., Hooker, A. C., Wallin, J. E., & Karlsson, M. O. (2011). Prediction-corrected visual predictive checks for diagnosing nonlinear mixed-effects models. The AAPS journal, 13(2), 143–151. https://doi.org/10.1208/s12248-011-9255-z