Model Averaging and Selection methods for model structure and parameter uncertainty quantification
Yasunori Aoki and Andrew C. Hooker
Department of Pharmaceutical Biosciences, Uppsala University, Uppsala, Sweden
Objectives: There have been several attempts through model averaging and model selection to weaken the assumptions around model structure by pre-specifying multiple possible model candidates prior to the analysis and automatically averaging or selecting the model structures [1]. However, to the best of authors’ knowledge, a methodology to combine the model structure and parameter estimation uncertainties to quantify the overall modelling uncertainty is not available. In this poster we extend our work in [2] and introduce a few possible model averaging/selection methodologies and compare the accuracy of these methods in the prediction of the minimum effective dose using a simulated dataset of FEV1 mimicking Phase IIb clinical trial used in [2].
Methods:
Step1: Create bootstrap datasets.
Step2: Estimate parameters for each model for each bootstrap dataset.
Step3: Simulate dose-endpoint relationships using each set of parameters estimated.
Step4: Construct probability of success vs. dose relationship using one of the following model schemes.
1): model selection using Akaike Information Criterion (AIC) based on OFV from the original dataset.
2): model selection using AIC based on the OFVs from the bootstrap datasets.
3): model averaging weighted by AIC based on the OFV from the original dataset.
4): model averaging weighted by AIC based on the OFVs from the bootstrap datasets.
In addition, we have conducted a numerical identifiability test using preconditioning [3,4] to weight out the models that are not locally practically identifiable.
Step5: Choose one of the candidate doses based on the probability of success vs. dose relationship.
Results: The proposed methods are made available in an open-source GUI based software at www.bluetree.me (also available as an r-script).
We have observed that model selection (schemes 1, 2) have done consistently better than model averaging (schemes 3, 4) when predicting the minimum effective dose. In addition, the confidence level for the correctly chosen minimum effective dose for scheme 2 was marginally but consistently higher than scheme 1. The identifiability test did not influence the results of the model selection; however, did marginally improve the results of model averaging.
Conclusion: Based on our numerical experiment we recommend model selection scheme 2. This method can be used as a way to pre-specify the possible model structures before obtaining the data so as to increase the objectivity of the model based analysis using nonlinear mixed effect models.
References:
[1] Frank Bretz, José C. Pinheiro, and Michael Branson. "Combining Multiple Comparisons and Modeling Techniques in Dose-Response Studies." Biometrics 61.3 (2005): 738-748.
[2] Yasunori Aoki, Bengt Hamrén, Daniel Röshammar, and Andrew C. Hooker, “Averaged Model Based Decision Making for Dose Selection Studies”, PAGE 23 (2014) Abstr 3121 [www.page-meeting.org/?abstract=3121]
[3] Yasunori Aoki, Rikard Nordgren, and Andrew C. Hooker, “Preconditioning of Nonlinear Mixed Effect models for Stabilization of the Covariance Matrix Computation”, PAGE 24 (2015) Abstr 3586 [www.page-meeting.org/?abstract=3586]
[4] Yasunori Aoki, Rikard Nordgren, and Andrew C. Hooker. "Preconditioning of Nonlinear Mixed Effects Models for Stabilisation of Variance-Covariance Matrix Computations." The AAPS journal (2016): 1-14.
