Evaluation in model averaging: an application to viral dynamic models
Selma El Messaoudi[1], Jérémie Guedj[1], Annabelle Lemenuel-Diot[2], Emmanuelle Comets[1,3]
[1] IAME, Université Paris Cité, Inserm, F-75018 Paris, France. [2] Roche Pharmaceutical Research and Early Development, Pharmaceutical Sciences, Roche Innovation Center Basel. [3] Irset-UMRS 1085,Université de Rennes, Inserm, EHESP, F-35000 Rennes, France.
Objectives:
The use of model averaging (MA) over model selection (MS) has already shown its advantages as it accounts for model uncertainty, by combining several candidate models with different biological assumptions to describe the data1. These aspects are particularly important in the case of mechanistic models, and more specifically in viral dynamic models, where parameter identifiability can be challenging and the different models may focus on different aspects of the disease.
Model evaluation is a step of model building which assesses the performance of a model, and simulation-based diagnostics have been proposed as gold standards2. We aim to extend one of these tools, the npde3 in the context of model averaging. We apply our method to a simulation-based study, using viral dynamic models to i) simulate building datasets ii) estimate parameters under each candidate model iii) computate npdeMA.
Methods:
In the context of model evaluation, the null hypothesis H0 is that observations in a dataset B can be described by a model M, defined with a set of equations and associated parameters Ψ. Prediction discrepancies (pd) are defined as the quantile of the observation within its predictive distribution, which is approximated through Monte-Carlo simulations. pd follow a uniform distribution U(0,1) under H0 and can be then decorrelated into pde using the inverse function of the cumulative density function (cdf). A final step of transformation to npde is then performed, which follow a normal distribution N(0,1).
In the model averaging approach, we define wm, the weights attributed to a candidate model m, and based on the likelihood. We can combine the predictive distributions under the model averaging framework using the same weights, and we then compute pdMA, empirical pdeMA and npdeMA as in the case of a single model. The null hypothesis H0 is here defined as an acceptable description of B by MA and therefore a global test was used testing the normality, the variance and mean of the npdeMA.
Five viral dynamic models often used to characterize acute viral infections were used to showcase the diagnostic4-8. For each, we used a set of true parameters Ψ0m and a design defined in Gonçalves et al. (2020)9 to sample 100 building datasets B from the population parameters. Similarly, 100 datasets V were simulated for external validation.
Each building dataset was used under each model to estimate Ψbm. npde were computed using internal and external datasets, using 1000 Monte-Carlo simulations. Model selection and computation of the weights for MA were based on AIC. We compared the rejection of npde under the true structural model (true model), the selected model, and the MA approach.
Results:
npdeMA can be used with a model resulting from model averaging approaches. npde were plotted for the true model, the selected model, and for the averaged predictions, showing similar graphical reprensentations with MA in comparison to the true models. Regarding model selection, the true model was selected in most of the cases (>90% for the majority of models tested), and npdeMA performed well even when another model was selected.
In the external validation, the variance of the npde was larger than 1 in all models tested,
including the true model. This was responsible for an inflation of the type 1 error to approximately 20% with all models. In most cases, npde of the true model were however normally distributed (90-96%), and centered on 0 (82-88%). Moreover, the rejection of npde was similar between the true model and the other models. This lack of power was explained by an overlaying predictive distributions from the different models, showing that the models tested were difficult to discriminate from the true model. Rejection rates with MA were similar to the true model.
Conclusions: We extended the npde to the evaluation of model averaging. The type I error of the npde have previously been shown to be close to the
theoretical value in different settings. The inflation we observed in our
simulation study was due to the additional estimation step, as the theoretical
distribution is obtained under the true model. In the future, we aim to investigate new approaches to account for uncertainty10 in order to reduce the type I error.
1. Buatois, S., Ueckert, S., Frey, N., Retout, S. & Mentré, F. Comparison of Model Averaging and Model Selection in Dose Finding Trials Analyzed by Nonlinear Mixed Effect Models. AAPS J 20, 56 (2018).
2. Nguyen, T. H. T., M-S Mouksassi, N Holford, N Al-Huniti, I Freedman, A. C. Hooker, J.
John, et al. « Model Evaluation of Continuous Data Pharmacometric Models: Metrics and
Graphics ». CPT: Pharmacometrics & Systems Pharmacology 6, no 2 (2017): 87 109
3. Guedj, J., Thiébaut, R. & Commenges, D. Practical identifiability of HIV dynamics models. Bull Math Biol 69, 2493–2513 (2007).
4. Comets, E. & Mentré, F. Developing Tools to Evaluate Non-linear Mixed Effect Models: 20 Years on the npde Adventure. AAPS J 23, 75 (2021).
5. Madelain, V. et al. Modeling Favipiravir Antiviral Efficacy Against Emerging Viruses: From Animal Studies to Clinical Trials. CPT Pharmacometrics Syst Pharmacol 9, 258–271 (2020).
6. Baccam, P., Beauchemin, C., Macken, C. A., Hayden, F. G. & Perelson, A. S. Kinetics of Influenza A Virus Infection in Humans. Journal of Virology 80, 7590–7599 (2006).
7. Pawelek, K. A. et al. Modeling Within-Host Dynamics of Influenza Virus Infection Including Immune Responses. PLOS Computational Biology 8, e1002588 (2012).
8. Li, Y. & Handel, A. Modeling inoculum dose dependent patterns of acute virus infections. Journal of Theoretical Biology 347, 63–73 (2014).
9. Gonçalves, A., Mentré, F., Lemenuel-Diot, A. & Guedj, J. Model Averaging in Viral Dynamic Models. AAPS J 22, 48 (2020).
10. Yano, Y., Beal, S. L. & Sheiner, L. B. Evaluating Pharmacokinetic/Pharmacodynamic Models Using the Posterior Predictive Check. J Pharmacokinet Pharmacodyn 28, 171–192 (2001).
