Machine Learning Based Model Selection with pyDarwin in Pirana
Mark Sale (1), Keith Nieforth (1), James Craig (1)
(1) Certara
Introduction: Machine learning based pharmacometric (PMX) model development may offer advantages over manual approaches by providing an efficient, objective and more robust method for model selection. pyDarwin[1] is a recently released open-source Python package for automated NONMEM pharmacokinetic/pharmacodynamic (pk/pd) model selection developed under a grant from, and in collaboration with FDA[2]. It offers a general solution for searching a wide range of pk and pd models within a user defined model search space and permits user customization of model fitness criteria to allow prioritization of desired model performance characteristics in the model selection process. Machine learning algorithms for model selection include Genetic algorithm, Bayesian optimization, random forest, gradient boosted random tree, particle swarm optimization and full, exhaustive search. The python command line interface to pyDarwin is logical and well organized but complex, requiring the editing of three text files; one to define the basic NONMEM control file structure, one to define the search space and one to define the search options and parameters as well as some knowledge of the underlying python language. A graphical user interface that greatly simplifies execution of pyDarwin model searches has been recently made available in Pirana[3].
Pirana is a PMX analysis workbench that integrates separate software tools under a common user interface, lending organization and efficiency to the PMX workflow while preserving the flexible nature of the underlying toolsets. Originally developed for use with NONMEM, Perl speaks NONMEM (PsN), and R/xpose package, Pirana has been enhanced to support command line modeling in R with Certara’s NLME engine[4], and as above, now supports machine learning based model selection through a graphical user interface to pyDarwin.
Objectives: To demonstrate the setup, execution and post-processing of a machine learning based model selection using the Darwin feature in Pirana.
Methods: A basic introduction to machine learning model selection will be provided covering general theory and an overview of available search algorithms in pyDarwin. This will be followed by a demonstration of the Pirana software covering the following topics:
- Overview of Pirana user interface demonstrating how to create, execute, post-process and generate report objects for a single model run
- Creation and execution of an automated model search with pyDarwin interface
- Initiating the Pirana Darwin workspace
- General: Selecting the desired search algorithm, setting options, and selecting performance settings
- Data Setup: Selecting dataset and mapping columns with optional centering of covariate values
- Model Template Setup: Enabling ADVANs, setting model parameterization and enabling covariates to be included in the search
- ADVAN Setup: Selecting structural and omega model parameters, identifying what model features are to be searched, included or excluded and setting initial parameter estimates
- ADVAN-Covariate Setup: Setting covariate-parameter relationships to search, model form and setting initial estimates
- Template Extras: Specify additional model block statements for template customization, such as table files or simulations
- Sigma Setup: Specifying residual error model form and setting initial estimates
- Downhill Step Setup: Set options for downhill model search step
- Penalties: Set penalties for model fitness calculations
- Postprocessing: Set path to R and/or python script used for run postprocessing
- Directories: Set custom directories for search inputs/outputs
- GA Setup: Set detailed genetic search algorithm options
- Model cache: Set custom controls for model cache
- Custom options: Set custom options for the options.json file
- Post-processing, and generation of report objects for a final model selected by automated model search
Results: The demonstration provides a comprehensive overview of the theory and execution of a machine learning based PMX model search
Conclusions: Pirana provides a user friendly interface to setup, execute, and postprocess automated machine learning based PMX model searches using pyDarwin
References:
[1] https://certara.github.io/pyDarwin/html/index.html
[2] https://grants.nih.gov/grants/guide/rfa-files/rfa-fd-21-027.html
[3] Keizer RJ, Karlsson MO and Hooker A. Modeling and Simulation Workbench for NONMEM: Tutorial on Pirana, PsN, and Xpose. CPT: Pharmacometrics & Systems Pharmacology (2013) 2, e50; doi:10.1038/psp.2013.24.
[4] https://certara.github.io/R-Certara/
