The development of artificial neural networks for the prediction of influential individuals and outlying individuals and their application during the model building process
Osama Qutishat (1), Simon J. Carter (1), Rikard Nordgren (1), Alzahra Hamdan (1), Shijun Wang (1), Tianwu Yang (1), Xiaomei Chen (1), Simon Buatois (2), João A. Abrantes (2), Andrew C. Hooker (1) and Mats O. Karlsson (1)
(1) Department of Pharmacy, Uppsala University, Sweden (2) Roche Pharma Research and Early Development, Roche Innovation Center Basel, Basel, Switzerland
Introduction: There is currently no ‘on the fly’ assessment available for the prediction of influential individuals (InfIDs) and outlying individuals (OutIDs) during model development. Currently, one assessment of InfIDs is conducted by removing an individual from a model, re-estimating the model and comparing the difference in OFV (dOFVi) for all other individuals under the two models [1]. OutIDs may be assessed by repeated simulations from the model and design with the final estimates and then re-estimation of individual OFVs (iOFVs), from which a residual is computed relative the iOFV of the ID´s real data [2]. The update to the FDA PopPK guidance stresses the identification and evaluation of outlying individuals, but puts the definition of such individuals on the sponsor [3]. The processes of identifying influential and outlying individuals are both time-consuming, especially if run concurrently during the model building process. It would therefore be useful to be able, on-the-fly, to quickly predict whether there are any InfIDs and OutIDs for a model and how that changes during model development. Artificial neural networks (ANNs), are potentially useful tools when there are numerous predictors or properties such as those associated with being an InfID or OutID.
Objectives: Develop, implement and ANNs as tools to predict InfIDs and OutIDs during the model building process, and test on 8 drugs at different points of a model building process.
Methods: (i) A database for the development of the ANNs was created by extracting multiple potential predictors (n = 14) from 27 published pharmacometric models. The time-consuming case-deletion (cdd in PsN [4]) and simulation-re-estimation (simeval in PsN) methods were executed to obtain the assumed true InfIDs and OutIDs for the ANN outputs. (ii) Two ANNs using the predictors and true outputs were developed for InfIDs and OutIDs, using Tensorflow within Python. The final ANN models were chosen based on a 10-fold cross validation on the data, which were split 90:10 train:test in each fold and then re-trained on the whole database. Sensitivity, specificity and precision were calculated based on a cut-off of 3.84 for dOFVi and 3 for the OutID residuals. To enable predictions of InfIDs and OutIDs to be made ‘on the fly’ within Pharmpy [5] and to reduce computer memory usage compared to Tensorflow, the final ANNs were converted to a Tensorflowlite model. (iii) The ANNs were applied to pharmacometric models of 8 drugs on 4 different milestone models in a predefined model building process [6]: start model, final structural model, final intra-individual variability model, and the final residual variability model.
Results: (i) Using cdd and simeval in PsN, 27/2537 subjects (~1%) were identified as InfIDs and 100/1807 were identified as OutIDs (6%). (ii) The ANN for OutIDs was able to predict with 79% sensitivity, 83% precision and 99.1% specificity. The ANN for InfIDs was able to predict with 58% sensitivity, 63% precision and 99.6% specificity. The differences in the sensitivity and precision between the ANNs may be, in part, due to the sparsity of the InfIDs dataset as well as the complexity in the prediction of InfIDs from a dataset. (iii) When the ANNs were applied to the 8 drugs, the number of OutIDs typically decreased as the models were improved in the development process, whilst for the InfID it did not.
Conclusions: Two ANNs for the rapid assessment of OutIDs and InfIDs were successfully developed and evaluated at various stages of a model development process for 8 different drugs. These newly developed ANNs can potentially be used as a tool to guide model development in a more efficient manner.
Acknowledgments: This work was supported by F. Hoffmann-La Roche Ltd., Basel, Switzerland.
References:
[1] Carter SJ et al. (in draft): Assessment of Influential Individuals and Outliers in Pharmacometric Models Through the Use of Case Deletion Diagnostics (CDD) and Simulation Evaluations (simeval) Tools (2022).
[2] Largajolli et al., The OFVPPC: A simulation objective function based diagnostic. PAGE 23 (2014) Abstract 3208 [www.page-meeting.org/?abstract=3208].
[3] FDA Population Pharmacokinetics – Guidance for Industry. February 2022. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/population-pharmacokinetics (accessed: 15th March 2022).
[4] Karlsson, M. O. & Nordgren, R. PsN: An Open Source Toolkit for Non-linear Mixed Effects Modelling. Available at: https://uupharmacometrics.github.io/PsN/ . (Accessed: 15th March 2022).
[5] Nordgren R et al. Pharmpy and assemblerr - Two novel tools to simplify the model building process in NONMEM. PAGE 29. (2021). Abstract 9656 [https://www.page-meeting.org/default.asp?abstract=9656].
[6] Hamdan A et al. Automatic development of pharmacokinetic structural models. (Abstract Submitted to PAGE (2022))
