Evaluation of assumptions underpinning pharmacometric models
Qing Xi Ooi
School of Pharmacy, University of Otago
Background: All models are underpinned by assumptions. The validity of any inference drawn from a model depends on the appropriateness and likely impact of the underlying assumptions [1]. Assumption evaluation is therefore an integral part of model building and model use. Current guidelines by the Food and Drug Administration (FDA) [2], the European Medicines Agency (EMA) [3], and the European Federation of Pharmaceutical Industries and Associations (EFPIA) [1] stipulate that all assumptions inherent to model development and model application should be explicitly expressed and evaluated. However, in the literature surrounding pharmacometric models, assumptions are not routinely acknowledged, described, or evaluated. This is also apparent in the analyses submitted for regulatory review, where the EMA outlined the lack of transparent description of influential assumptions and an ineffective evaluation or reporting of the impact of assumptions on model inference to be a major limitation [4]. These form an important barrier for effective model use and regulatory review. Here, while the importance of assumption evaluation is well-recognised, how these assumptions should be systematically approached and be effectively assessed has received limited attention.
Objectives: In this work, we proposed a framework for evaluating assumptions systematically that is generalisable to both top-down and bottom-up pharmacometric model development. The objectives of this work were:
1. to define an assumption within the context of this work,
2. to develop a flowchart for systematic evaluation of assumptions,
3. to propose a standardised table for documentation of assumptions and evaluation results, and
4. to apply the flowchart to a top-down and a bottom-up model.
Methods: Medline (1946 – December 2017), EMBASE (1947 – December 2017), Google Scholar (1947 – December 2017), as well as the websites of medicines regulatory authorities were searched for pharmacokinetic and pharmacodynamic guidelines and good practice papers. Subsequently, these papers were screened for methods related to evaluating model assumptions. Relevant papers were mined for additional articles. Two key articles by EFPIA [1] and Karlsson et al. [5] provided specific frameworks for evaluating model assumptions and were used as a starting point for this work. The existing frameworks were expanded based on an evaluation of the risk management literature, expert opinion, and logical reasoning. The typical workflow for assumption evaluation was generalised into a qualitative flowchart. The flowchart was developed in a stepwise manner. In the first step, a decision tree that mapped all possible outcomes from a sequential evaluation of the impact and the probability of assumption violation was built. In the next step, the multilevel decision tree was streamlined to a simple flowchart for assumption evaluation. Subsequently, a table was designed based for documentation of assumptions and evaluation results. The utility of the flowchart was illustrated for both: (a) a top-down model building process and (b) a bottom-up work based on a quantitative systems model. The top-down approach was based on a kinetic-pharmacodynamic (KPD) model for warfarin and the coagulation proteins [6]. For the bottom up approach, we considered the development of a warfarin dosing method based on a systems coagulation network model [7].
Results: For the purpose of this work, an assumption is defined as a perception of the truth that can be distinguished from a hypothesis (a testable belief), an axiom (a self-evident belief), a theorem (a proven belief) and a limitation (a boundary beyond which the assumption no longer holds). We categorise assumptions into two types: (a) implicit in which a theorem is being relied upon to form a framework of the modelling process (e.g. linearity between two variables if relationships were to be quantified using Pearson correlation); (b) explicit which arises from a gap in knowledge for which an imputation by the investigator will be required (e.g. heuristic application of a Michaelis-Menten model to describe a system response for which we have no prior knowledge). To be exhaustive in identifying assumptions, modellers are encouraged to list the assumptions systematically according to the nature of the assumption: (a) biological or physiological, (b) pathophysiological, (c) pharmacological or pharmaceutical, (d) experimental, (e) study conduct, and (f) statistical or mathematical assumptions.
A flowchart for the systematic evaluation of assumptions was developed. For each assumption, the impact of assumption violation, I (“significant”, “insignificant”, “unknown”), is first assessed to stratify risk. If I is significant or unknown then the probability of assumption violation, P (“likely”, “unlikely”, “unknown”) is evaluated. Here, the ratings for I and P are rated based on prior knowledge or the result of an additional bespoke study (often a simulation study), termed posterior. In this work, both I and P are evaluated for their influence on: (a) an internal component of the model building, or (b) an external use of the model (i.e. a circumstance in which the model is used for inference other than for the data that was used to build the model). The outcomes of the flowchart included go / no-go decision for internal and external use of the model. The decisions may be accompanied by an acknowledgement of the assumption as a limitation, for instance, when I is unknown but P is unlikely in the specific current scenario. For documentation of assumptions and evaluation results, an assumption table with standardised headings (“assumption”, “prior or posterior I or P (subheadings: ‘evaluation methods’, ‘results’, ‘ratings’)”, “action point”, “decision”) are proposed.
Finally, the utility of the flowchart was demonstrated using assumptions from both top-down and bottom-up model. For brevity, only one assumption from each type of model is illustrated here. Using the KPD model example, I was found significant and P unlikely for the assumption “residual errors are normally distributed” thereby giving a final go decision for model building. For the systems model example, the assumption that “the model developed accurately describes factor VII and the anticoagulant response” was associated with significant I and likely P owing to the biased predictions produced beyond the dose range modelled. This resulted in a no-go decision for external model use.
Conclusions: A framework for systematic evaluation of assumptions is proposed and its utility is demonstrated using both top-down and bottom-up examples. The next step of this work is to apply the framework to a series of other settings to fully assess its practicality and its value in identifying and making inference from assumptions.
References:
[1] Marshall SF, Burghaus R, Cosson V, Cheung SY, Chenel M, DellaPasqua O, et al. Good practices in model-informed drug discovery and development: practice, application, and documentation. CPT Pharmacometrics Syst Pharmacol. 2016;5(3):93-122.
[2] U.S. Department of Health and Human Services, Food and Drug Administration, Center for Drug Evaluation and Research (CDER), Center for Biologics Evaluation and Research (CBER). Guidance for industry: population pharmacokinetics. 1999 [Available from: https://www.fda.gov/downloads/drugs/guidances/UCM072137.pdf.
[3] European Medicines Agency (EMA). Guideline on reporting the results of population pharmacokinetic analyses. 2007 [Available from: http://www.ema.europa.eu/docs/en_GB/document_library/Scientific_guideline/2009/09/WC500003067.pdf.
[4] European Medicines Agency (EMA). EFPIA-EMA modelling and simulation workshop (2011) report. London, United Kingdom; 2012.
[5] Karlsson MO, Jonsson EN, Wiltse CG, Wade JR. Assumption testing in population pharmacokinetic models: illustrated with an analysis of moxonidine data from congestive heart failure patients. J Pharmacokinet Biopharm. 1998;26(2):207-46.
[6] Ooi QX, Wright DF, Tait RC, Isbister GK, Duffull SB. A Joint Model for Vitamin K-Dependent Clotting Factors and Anticoagulation Proteins. Clin Pharmacokinet. 2017.
[7] Wajima T, Isbister GK, Duffull SB. A comprehensive model for the humoral coagulation network in humans. Clin Pharmacol Ther. 2009;86(3):290-8.
