Comparing the objective function value in NONMEM is valid for model selection when fixing THETA or OMEGA but not when fixing ETAs
Lena M. Appel (1), Bernhard Steiert (1)
(1) Roche Innovation Center Basel, Roche Pharma Research and Early Development, Basel, Switzerland.
Introduction: In population PKPD modeling a common aim is to use models to make predictions on quantities of interest of a group of individuals. To select appropriate structural models, parameters and covariates, different assumptions are compared using a goodness of fit metric, such as the objective function value (OFV) in NONMEM. Statistical testing theory provides requirements for when differences in OFV is a valid metric for model comparison. Due to the population approach, which accounts for the distribution of individual parameters, the OFV includes more components than the classical likelihood function used for regression analysis [1,2,3]. These additional components may affect the comparability of the OFV when parameters are fixed, which poses a risk of false rejections of models.
Objectives: The objective of this work was to examine the effect of fixing different types of model parameters on the OFV and thus determine the cases in which the difference in the OFV is a valid metric for model selection.
Methods: A literature review was performed to outline the approximations used to calculate the OFV in NONMEM and to decompose the OFV into its components [1,2]. The analysis was focussed on the FOCE approximation without interaction [3] and an indirect response model was used to generate a simulated data set with 100 individuals. This indirect response model was then fitted to the simulated data and the following scenarios were evaluated: (1) all parameters being estimated, (2) one OMEGA or THETA being fixed to the estimated value, (3) one OMEGA being fixed to various smaller values including zero and finally, (4) one OMEGA being removed from the model and the corresponding individual parameters (ETAs) being set to the estimated values by including them in the data set. A required condition to consider differences in OFV as valid metric for model selection was that a smooth transition of the OFV between the investigated scenarios can be established.
Results: We found that fixing population parameters (THETA) and the interindividual variance (OMEGA) allows for valid model selection based on the differences of the OFV: estimating THETA or OMEGA parameters results in the same OFV as fixing them to the estimated value. Furthermore, we observed that (1) some components of the OFV have a singular behavior when one OMEGA approaches zero asymptotically due to the divergent behavior of the corresponding entry in the OMEGA matrix, (2) if OMEGA is set to zero all corresponding terms - entries in OMEGA and derivatives with respect to the ETA belonging to this OMEGA - are omitted in the calculation of the OFV and (3) the transition from an OMEGA asymptotically approaching zero to omitting all the corresponding entries in the OFV, when setting an OMEGA to zero, is smooth with respect to the OFV value. For diagonal OMEGA matrices we prove the equivalence of the limit of an OMEGA approaching zero to the case where terms are omitted when this OMEGA is set to zero. In consequence when fixing OMEGA to zero the change in OFV is still a valid metric for model selection. In the setting of fixing the individual parameters to the estimated values by introducing the estimated ETAs as covariates, the corresponding OMEGA and its contribution to the OFV components are also removed, but the OMEGA in this case is usually finite-valued. Thus, the removed terms are not divergent and do not dominate the OFV components such that the value of the removed components does not cancel out anymore. In consequence, when fixing ETAs, the removal of the corresponding OMEGA will lead to an abrupt change in the OFV, which renders the OFV unsuitable as a metric for model selection. Despite using only a certain model and variability/noise realization, due to our understanding of the OFV components this result is expected to be generally true.
Conclusion: In conclusion, this work examines the OFV component behaviours in different scenarios and provides guidance on when model selection based on the difference in OFV is meaningful. Although fixing population parameters (THETA) and interindividual variances (OMEGA) leads to valid comparisons between different OFVs when compared to the model fit where no parameters are fixed, other goodness of fit metrics are needed for selecting between models where the individual parameters (ETAs) are fixed.
References:
[1] Wang Y., J Pharmacokinet Pharmacodyn, 34:575–593, 2007
[2] Bae K.-S. and D.-S. Yim, Transl Clin Pharmacol, 24(4):161-168, 2016
[3] Beal S. L. and Sheiner L. B., NONMEM Users Guide - Part VII., NONMEM Project Group C255, University of California at San Francisco, San Francisco, USA, 1998