Characterizing Bootstrap Model Selection Variability Through an Automated Model Building Approach
Yersultan Mirasbekov (1), Xiaomei Chen (1), Andrew C. Hooker (1), Mats O. Karlsson (1)
Pharmacometrics group, Department of Pharmacy, Uppsala University, Sweden
Introduction and Objectives:
The development of a population pharmacokinetic (PK) model is a challenging and time-consuming procedure. There are several automatic approaches for model building, but the whole model development pipeline wasn’t addressed [1-4]. A fully automatic model development (AMD) tool has been developed to cover all the components of PK modeling [5]. The AMD tool can potentially automatize model development workflow, and this approach is integrated into Pharmpy/Pharmr software [6]. The aim of this work is to learn about model selection variability and its consequences using the AMD tool. Three search modules of the AMD tool were investigated: model selection of structural, inter-individual variability (IIV), and residual unexplained variability (RUV) models. The structural model search tests all possible structural models within the given search space and selects the model based on the BIC [6, 7]. IIV model search uses a two-step algorithm to select the best model [5]. Finally, the RUV model search selects the residual variability model [8].
Methods:
The AMD tool was applied to four clinical PK datasets of intravenously administered drugs, namely Daunorubicin, Gentamicin, Pefloxacin, and Tobramycin [9-11]. These original datasets were used to generate 10 bootstrapped datasets each. The tool was applied with the model selection modules in the order of structural model, IIV model, and RUV model on the original data and the bootstrapped datasets. This enabled the assessment of how the variability of input data affects the selection of final models (“original model” and 10 “bootstrap-generated models”, respectively). The change in BIC and parameter estimates in each step were tracked to assess each module. Then, the original model was applied for parameter estimation for all bootstrap datasets to learn about parameter variability in the presence and absence of model selection variability. Finally, bootstrap-generated models were assessed on the original full dataset to investigate the possibility to improve on BIC of the original model.
Results:
The variability of bootstrap models was evaluated by the scoring of differences in selected models compared to the original model. The final models based on bootstrap datasets usually had at least one difference in model selection. When modules were evaluated separately, the consistency of the structural model selection (in 85% of cases the bootstrap was the same model as the original model) was higher than IIV (30%) and RUV (48%) modules. Parameter estimates were used to calculate the coefficient of variability (CV), and the estimation of typical values of the clearance (CL) showed a similar CV regardless of the inclusion of model selection or not. For the typical value of the volume of distribution (Vss), CV in Tobramycin and Gentamicin datasets was considerably higher with model selection than without, 28% vs 14% and 64% vs 20%, respectively. In a second evaluation, the bootstrap-generated model and the original model were compared in terms of BIC for each bootstrap dataset. Neither of these two models showed a clear advantage over the other. When bootstrap-generated models were applied to original data, with re-estimation, they were found to improve on the original model in 3 cases out of 40, all for the Tobramycin dataset. The largest improvement in these cases was a decrease in BIC of 11.1.
Conclusion:
The results of the experiments showed that the consistency in model selection was highest in the step of structural model selection, and lowest in IIV model selection. It is important to highlight that the selection of the structural model was the first in a decision tree of final model selection. The contribution of model selection to overall parameter uncertainty, varied between parameters and was sometimes substantial. No significant preference was found when the original model and bootstrap-generated model were fit on bootstrap datasets. Finally, the results of model fitting of all final models on original data showed that the bootstrap-generated model for one dataset had lower BIC than the original model. This pipeline of generating final models from bootstrapped datasets can serve multiple purposes in understanding the model development and the properties of the final model.
References:
[1] Jonsson, E. N., & Karlsson, M. O. (1998). Automated covariate model building within NONMEM. Pharmaceutical research, 15(9), 1463-1468.
[2] Sale, M., & Sherer, E. A. (2015). A genetic algorithm based global search strategy for population pharmacokinetic/pharmacodynamic model selection. British journal of clinical pharmacology, 79(1), 28-39.
[3] Prague, M., & Lavielle, M. (2022). SAMBA: A novel method for fast automatic model building in nonlinear mixed‐effects models. CPT: Pharmacometrics & Systems Pharmacology, 11(2), 161-172.
[4] Ayral, G., Si Abdallah, J. F., Magnard, C., & Chauvin, J. (2021). A novel method based on unbiased correlations tests for covariate selection in nonlinear mixed effects models: The COSSAC approach. CPT: pharmacometrics & systems pharmacology, 10(4), 318-329.
[5] Chen, X., Hamdan, A., Wang, S., Yang, T., Nordgren, R., Belin, S., Huang, Z., Carter, S. J., Buatois, S., Abrantes, J. A., Hooker, A. C. & Karlsson, M. O. “Development of a tool for fully automatic model development (AMD)”. PAGE, 30:10091 www.page-meeting.org/?abstract=10091.
[6] Nordgren, R., Belin, S., Yngman, G., Huang, Z., Carter, S. J., Chen, X., Qutishat, O., Hamdan, A., Wang, S., Yang, T., Assawasuwannakit, P., Buatois, S., Abrantes, J. A., Hooker, A. C. & Karlsson, M. O. (2022). Pharmpy: a versatile open-source library for pharmacometrics. PAGE, 30:10096. www.page-meeting.org/?abstract=10096
[7] Delattre, M., Lavielle, M., & Poursat, M. A. (2014). A note on BIC in mixed-effects models
[8] Ibrahim, M., Nordgren, R., Kjellsson, M. C., & Karlsson, M. O. (2018). Model-based residual post-processing for residual model identification. The AAPS Journal, 20(5), 1-9.
[9] Aarons, L., Vozeh, S., Wenk, M., Weiss, P. H., & Follath, F. (1989). Population pharmacokinetics of tobramycin. British journal of clinical pharmacology, 28(3), 305-314.
[10] Wählby, U., Thomson, A. H., Milligan, P. A., & Karlsson, M. O. (2004). Models for time‐varying covariates in population pharmacokinetic‐pharmacodynamic analysis. British journal of clinical pharmacology, 58(4), 367-377.
[11] Bogason, A., Quartino, A. L., Lafolie, P., Masquelier, M., Karlsson, M. O., Paul, C., ... & Vitols, S. (2011). Inverse relationship between leukaemic cell burden and plasma concentrations of daunorubicin in patients with acute myeloid leukaemia. British journal of clinical pharmacology, 71(4), 514-521.
