Extensive and automatic assumption assessment of pharmacometric models
Mats O Karlsson, Svetlana Freiberga, Gunnar Yngman, Rikard Nordgren, Sebastian Ueckert
Dept Pharmaceutical Bioscences, Uppsala University, Uppsala, Sweden
Objectives: The reliable use of Nonlinear Mixed Effects Models (NLMEM) in decision-making is based on the appropriateness of the assumptions underpinning the model (1-3). Therefore, we develop methodology that involves (i) comprehensive and exhaustive assessment of the probability of assumption violations, (ii) metrics for the expected impact of any assumption violation, and (iii) specific advice on action(s) likely to avoid or rectify a particular assumption violation. In particular, we develop for this purpose “model-proxy analyses”, i.e. analyses that provide the same information as testing a new model (change in goodness-of-fit/OFV, parameters, predictions, etc.) but in a robust and time-efficient manner.
Methods: The scope of assumption assessments we consider includes most aspects testable on data and include six sections: (i) structural, (ii) covariate, (iii) parameter and (iv) residual model components as well as (v) influential individual and (vi) outlier assessments. For this purpose recent (4,5) and new (6-9) methodology based on analyzes of linear(ized) mixed effects models (LMEM) has been compiled and integrated. The methodology is implemented as the “QA” package in the PsN software. Basic arguments to the assessment are the model to be evaluated together with the parameters and covariates of interest. Both covariates already included in the model as well as those not included can be elements of the investigation. The output of the QA routine is a structured pdf-report providing top-level information about what aspects of the model which may or may not be associated with assumption violations and directions to further in-depth results. The performance of the QA routine was evaluated on 30 previously published models available in-house, from collaborators or retrieved from the DDMoRe model repository.
Results:
Average characteristics of the 30 models were: 139 subjects (range 8-644), 3.9 parameters (1-8) and 3.8 covariates (1-10) resulting in an average of 207 (54-700) proxy models that on average completed execution in a total of 12 (2-90) min. While limitations in scope and other shortcomings (i.e. bugs) resulted in some failed runs, the overall success rate was at least 90% for all 6 sections and 100% for the structural, parameter variability and residual model sections.
The structural model section assesses the assumption of lack of bias in predictions. This is evaluated with respect to independent variables such as time and prediction. The output is presented as (i) potential improvement in goodness-of-fit (OFV) to be gained by avoiding structural model bias, (ii) the magnitude of bias (back-calculated from CWRES using the FOCE approximation), and (iii) a graphical (VPC) representation of the original and bias-corrected model.
The parameter variability section assesses extensions exploring full covariance matrix models, allows skewed or heavy-tailed distributions for the random effects and the need for additional interindividual or interoccasion random effects. For each component the improvement in OFV, the change in parameter values and in parameter distribution shape is provided.
The residual variability section provide extensions with respect to serial correlation, distribution skewness, t-distributed errors, time-varying or interindividual error magnitudes and presents improvement in OFV, change in parameter estimates and expected change in precision in structural parameter estimates.
The covariate section present both univariate (SCM) and full model assessments of covariates providing improvement in OFV, magnitude of improvement, Forest plots, variability explained and extreme individual information.
The influential individual section focuses on individuals for which the omission importantly change the description of the model for the remaining individuals.
The outlier analysis differentiates between outliers due to individual observations and due to outlying parameter value, where possible.
Conclusions: A comprehensive evaluation of assumptions that are to be assessed based on data has been presented, implemented and evaluated. Advantages over graphical assessment of a final model is that it is objective, specific on what assumption is violated and predictive of the consequence of a model change. Compared to running new models it is fast, robust and automated. It is easy to use and while of particular value at the final stages of model development, it may improve modelling at any stage of refinement.
References:
[1] Karlsson MO, Jonsson EN, Wiltse CG, Wade JR. Assumption testing in population pharmacokinetic models: illustrated with an analysis of moxonidine data from congestive heart failure patients. J Pharmacokinet Biopharm. 1998 Apr;26(2):207-46
[2] EFPIA MID3 Workgroup, Marshall SF, Burghaus R, Cosson V, Cheung SY, Chenel M, DellaPasqua O, Frey N, Hamrén B, Harnisch L, Ivanow F, Kerbusch T, Lippert J, Milligan PA, Rohou S, Staab A, Steimer JL, Tornøe C, Visser SA. Good Practices in Model-Informed Drug Discovery and Development: Practice, Application, and Documentation. CPT Pharmacometrics Syst Pharmacol. 2016 Mar;5(3):93-122. doi: 10.1002/psp4.12049. Epub 2016 Mar 14.
[3] Ooi QX, Wright DFB, Isbister GK, Duffull SB. Evaluation of assumptions underpinning pharmacometric models. PAGANZ 2018.
[4] Largajolli A, Jönsson S, Karlsson MO. The OFVPPC: A simulation objective function based diagnostic PAGE 23 (2014) Abstr 3208 [www.page-meeting.org/?abstract=3208]
[5] Ibrahim MMA, Nordgren R, Kjellsson MC, Karlsson MO. Model-based diagnostics post-processing for fast automated model building; show-cased with residual error models and CWRES. PAGE 26 (2017) Abstr 7276 [www.page-meeting.org/?abstract=7276]
[6] Nordgren R, Ueckert S, Freiberga S, Karlsson MO. Faster methods for case deletion diagnostics: dOFV and linearized dOFV. PAGE 2018
[7] Ueckert S, Freiberga S, Yngman G, Karlsson MO. Understanding sources of variability: The variability attribution plot. PAGE 2018.
[8] Ibrahim MMA, Ueckert S, Freiberga S, Kjellsson MC, Karlsson MO Model-based post-processing of CWRES for assessment of prediction bias. PAGE 2018
[9] Yngman G, Nordgren R, Freiberga S, Karlsson MO Linearization of full random effects modeling (FREM) for time-efficient automatic covariate assessment. PAGE 2018.
