User-friendly parameter identifiability methods with a workflow to guide model development, using categorical and continuous scales
Martijn van Noort, Martijn Ruppert, Joost deJongh, Eleonora Marostica, Rolien Bosch, Emir Mešic´, Nelleke Snelder
LAP&P Consultants BV, Leiden, The Netherlands
Objectives: Parameter identifiability methods assess whether the parameters of a model are uniquely determined by the observations. While the success of a model fit can provide some information on this, it can be valuable to determine identifiability before attempting a fit, or to separate identifiability from other issues. Traditional identifiability methods provide a single categorical (yes/no) answer [1-12] to the question of identifiability, often assuming idealized data. In many cases this is not very informative, and identifiability depends on study design (e.g., dose levels or sampling times) and parameter values. Indicators on a continuous scale characterizing the level of identifiability and taking design limitations into account can provide more detailed and relevant information to guide model development.
This investigation 1) presents new methods that characterize identifiability on a continuous scale; 2) shows that the Sensitivity Matrix (SM) and Fisher Information Matrix (FIM) can be used for this purpose; 3) demonstrates the application of these methods to an intuitive example and 4) describes a workflow for employing parameter identifiability analyses in model development.
Methods: We present and demonstrate two newly developed methods that characterize identifiability with categorical and continuous indicators, namely the Sensitivity Matrix Method (SMM) and the Fisher Information Matrix Method (FIMM) [13], and assess practical identifiability by restricting the observed quantities to user-selected time points.
SMM examines the SM, consisting of the derivatives of the model output, evaluated at a finite set of time points, with respect to its parameters. Local unidentifiability is formally characterized by a non-trivial null space of the SM. Continuous indicators are the skewing angle, M-norm, and L-norm.
FIMM computes the FIM at a given parameter point and observation times. Local unidentifiability is formally characterized by a zero curvature of the log-likelihood surface, corresponding to a zero eigenvalue of the FIM. Continuous indicators are the curvatures and the relative parameter changes.
The methods are demonstrated on an intuitive example, namely a one-compartmental linear PK model with first order absorption, in three scenarios:
• A: all parameters are identifiable.
• B: a lower absorption rate creates an identifiability problem.
• C: addition of a bioavailability parameter makes the model formally unidentifiable.
A more complex example will be shown and a workflow to apply these methods in model development is presented.
Results:
The categorical yes/no answers of SMM and FIMM to the identifiability question are valuable in case of structural unidentifiability (C): if a model is categorically unidentifiable, then the model needs to be redefined or additional analytes need to be measured to resolve this. Continuous indicators provide more detail and may indicate badly identifiable cases (B). This may be due to study design limitations or the values of the parameters. The remedy is to change the model or adapt the design. Continuous indicators are subject to interpretation by the analyst and therefore require some experience in their application.
In line with expectations, all categorical and continuous indicators indicated identifiability for scenario A and unidentifiability for C. For B, categorical indicators showed identifiability, but continuous ones indicated this scenario was less identifiable than A. The second example showed that the methods can be applied to more complex cases as well.
To provide guidance for using these methods in daily practice, we present a workflow for incorporating identifiability analysis in model development, with separate processes before, during and at the end of model development [14]. These processes incorporate structural and practical identifiability analyses.
Conclusions: Two newly developed methods for parameter identifiability, SMM and FIMM, are implemented [15] in R [16]. FIMM provides the clearest and most useful answers. SMM is an intuitive method that is computationally more efficient than FIMM, but the cut-off between identifiability and unidentifiability is more difficult to establish. The availability of the methods, together with the workflow recommendation, facilitate the addition of parameter identifiability analysis to the toolbox of the modeler to diagnose over-parameterization and assess a model’s suitability in relation to study design.
References:
1. Bellu, G., Saccomani, M. P., Audoly, S. & D’Angiò, L. DAISY: A new software tool to test global identifiability of biological and physiological systems. Comput. Methods Programs Biomed. 88, 52–61 (2007).
2. Breiman, L. & Friedman, J. H. Estimating Optimal Transformations for Multiple Regression and Correlation. J. Am. Stat. Assoc. 80, 580–598 (1985).
3. Sedoglavic, A. A Probabilistic Algorithm to Test Local Algebraic Observability in Polynomial Time. J. Symb. Comput. 33, 735–755 (2002).
4. Ligon, T. S. et al. GenSSI 2.0: Multi-experiment structural identifiability analysis of SBML models. Bioinformatics 34, 1421–1423 (2018).
5. Balsa-Canto, E., Alonso, A. A. & Banga, J. R. An iterative identification procedure for dynamic modeling of biochemical networks. BMC Syst. Biol. 4, (2010).
6. Augustin, F., Braakman, S. T. & Paxson, R. a Workflow To Detect Non-Identifiability in Parameter Estimation Using Simbiology (2019). Accessed Feb 1, 2024. at <https://github.com/mathworks-SimBiology/AliasingScoreApp/blob/master/AliasingScore_Poster.pdf>
7. Cintrón-Arias, A., Banks, H. T., Capaldi, A. & Lloyd, A. L. A sensitivity matrix based methodology for inverse problem formulation. J. Inverse Ill-Posed Probl. 17, 545–564 (2009).
8. Denis-Vidal, L., Joly-Blanchard, G. & Noiret, C. Some effective approaches to check the identifiability of uncontrolled nonlinear systems. Math. Comput. Simul. 57, 35–44 (2001).
9. Raue, A. et al. Structural and practical identifiability analysis of partially observed dynamical models by exploiting the profile likelihood. Bioinformatics 25, 1923–1929 (2009).
10. Chis, O. T., Banga, J. R. & Balsa-Canto, E. Structural identifiability of systems biology models: A critical comparison of methods. PLoS One 6, (2011).
11. Cucurull-Sanchez, L. et al. Best Practices to Maximize the Use and Reuse of Quantitative and Systems Pharmacology Models: Recommendations From the United Kingdom Quantitative and Systems Pharmacology Network. CPT Pharmacometrics Syst. Pharmacol. 8, 259–272 (2019).
12. Raue, A., Karlsson, J., Saccomani, M. P., Jirstrand, M. & Timmer, J. Comparison of approaches for parameter identifiability analysis of biological systems. Bioinformatics 30, 1440–1448 (2014).
13. Noort, M. van & Ruppert, M. Two new user-friendly methods for parameter identifiability on categorical and continuous scales (Submitted for publication).
14. Noort, M. van et al. Comparing parameter identifiability methods on several example models (Submitted for publication).
15. LAP&P Consultants B.V. Identifiability Analysis (2023). Accessed June 30, 2023. at http://www.lapp.nl/lapp-software/parid.html
16. R Foundation for Statistical Computing R: A Language and Environment for Statistical Computing (2020). Accessed May 4, 2023. at <https://www.r-project.org>
