Faster methods for case deletion diagnostics: dOFV and linearized dOFV
Rikard Nordgren (1), Sebastian Ueckert (1), Svetlana Freiberga (1) and Mats O. Karlsson (1)
(1) Department of Pharmaceutical Biosciences, Uppsala University, Sweden
Objectives: Case deletion diagnostics (cdd) is a method for finding individuals that are especially influential on a model estimation. The standard way of performing a cdd for non-linear mixed effect models (NLMEM) is to exclude one individual at a time and reestimate the model. Typically the cook score and the covariance ratio have been used as metrics for the individual influence on the parameter estimates [1, 2]. Both these metrics require the covariance matrix of the parameter estimates which might be difficult to obtain. More importantly, the usefulness of the cdd is diminished by the fact that it is relatively slow as it requires one full NLMEM estimation per individual. Here we propose a new metric, delta-OFV (dOFV), to assess the influence on the total model fit and evaluate it for a set of models. This metric does not rely on the covariance matrix. Furthermore, the switch to the dOFV metric also allows to use linearized models which can substantially reduce runtime.
Methods: The dOFV metric was calculated for each case deleted run separately and was defined as the difference between the OFV of the full run adjusted for the removed individual and the case deleted run. Let OFVall be the OFV for the run with all individuals and iOFVk be the individual OFV for subject k in that run. Furthermore, let OFVk be the OFV from the run with individual k removed, then dOFV was calculated as dOFV = OFVall - iOFVk - OFVk.
The dOFV based cdd was performed both using the NLMEM as well as using a linearization approximation of the model [3,4] and compared to a cook score based cdd. Additionally, the runtimes of the non-linear dOFV cdd, the linearized dOFV cdd and the cook score based cdd were compared.
All comparisons of the different metrics and runtime were done for 14 different pharmacometric models using PsN 4.7.16 and NONMEM 7.4.2.
Results: The graphic comparison between dOFV and cook score showed a slight non-linear relation and the Pearson (linear) correlation coefficient ranged between 0.40 and 0.97 for the tested models.
The comparison of dOFV with linearized dOFV for the 14 models shows a correlation between 0.52 and 0.99 with an average of 0.92. Setting a cutoff for influential individuals to 3.84 displayed only one missed influential individual in the linearized cdds and no false influential individuals.
Speedup for non-linear dOFV cdd versus the cook score based cdd was between slightly above 1 up to 1.8 for the tested models. The average speedup for the linearized models with short runtimes (below 1000 s) was, due to a high proportion of overhead, only 2.4, but was for the models with long runtimes (above 1000 s) 973.
Conclusions: The dOFV based cdd showed promising performance with large agreement with the traditional cook score based cdd but shorter runtime and without relying on the covariance matrix. Model linearization led to large reductions in runtime without a big impact on the dOFV cdd results. Given our experiments the risk of missing an influential individual exists but is low.
References:
[1] Cook RD, Detection of influential observation in linear regression, Technometrics 19 (1977) 15-18
[2] Lindbom L, Pihlgren P and Jonsson N, PsN-Toolkit – A collection of computer intensive statistical methods for non-linear mixed effect modeling using NONMEM, Computer Methods and Programs in Biomedicine (2005) 79, 241-257
[3] Khandelwal A, Harling K, Jonsson EN, Hooker AC and Karlsson MO, A fast method for testing covariates in population PK/PD Models, 2011, AAPS J 13:464-472
[4] Svensson EM and Karlsson MO, Use of a linearization approximation facilitating stochastic model building, 2014, J PKPD, 41:153-158
