Generating uncertainty estimates in empirical forest plots
Andrew C. Hooker (1), Joakim Nyberg (2), E. Niclas Jonsson (2)
(1) Pharmacometrics Research Group, Department of Pharmacy, Uppsala University, Uppsala, Sweden (2) Pharmetheus AB, Uppsala, Sweden
Introduction: Forest plots for the covariates included in a pharmacometrics model provide a graphical representation of the effect sizes of covariates on primary and secondary parameters and are a useful tool for communicating the findings of a modeling process to a wider audience, including researchers, clinicians, policymakers, and patients [1]. Typically, a forest plot may be constructed from a final model that has been fit to data, its parameter estimates and the uncertainty of those estimates, without using the data in the creation of the plot (parametric forest plots). However, if a covariate is not part of the final model there is no direct way to predict the potential effect the covariate has on the endpoint of interest.
Empirical forest plots allows the approximation of covariate effect of interest even if the covariate is not in the model. This approach uses model predicted individual parameter values, which can be summarized based on the covariate of interest for each individual (e.g., the median of the individual clearance values for males and females). In Jonsson and Nyberg [1], the individual parameter values are computed based on typical values of the model and included covariate effects. This avoids the problem of shrinkage, but may underestimate the uncertainty of the estimated effect. Additionally, if informative covariates are missed in the final model, and no covariate is present in the final model with relatively high correlation to the missed informative covariate, the informative covariate will not show up as impactful in an empirical forest plot.
In this work we investigate the use of empirical forest plots where the individual parameter values and the uncertainties are based on samples from the individual conditional distribution of individual parameters [2]. We hypothesize that this will more accurately capture the effect size and uncertainty of covariates not included in the model.
Methods: To generate these improved empirical forest plot we:
(1) establish a final model with covariate effects
(2) predict individual parameter values based on the typical values, covariate effects in the model, and the conditional mean of the individual conditional distribution of the parameter.
(3) Summarize the primary or secondary parameters based on the covariates of interest to generate data for the forest plot.
(4) Compute the uncertainty for the summary statistics computed in (3). One method includes utilizing samples from the individual conditional distribution of the individual parameters, other methods involve using the variance-covariance matrix from the model fit to data or by bootstrapping the model and iterate over steps (2) and (3).
(5) Generate the forest plot.
We perform a simulation study using NONMEM [3] and R [4] where data is simulated from various covariate models and then evaluated with the simulation model or with misspecified covariate models (including a base model without covariates). Simulations include situations with models that have covariates on parameters with low shrinkage as well as on parameters with high shrinkage. The generated forest plots are compared to parametric forest plots.
Results: If the estimation model matches the simulation model then generation of the empirical forest plot match well with the parametric forest plot. We can estimate effects that are not in the model using the samples from the individual conditional parameter distributions. If the estimation model does not match the simulation model, including if the estimation model has no covariates, then an empirical forest plot can typically give a qualitative picture similar to the parametric forest plot. High shrinkage shifts the empirical forest plots towards the center of the plot, but sampling from the conditional distribution of the individual parameters (as opposed to using empirical bayes estimates) clearly reduces this shift.
Conclusions: Empirical forest plots are a viable way to generate expected effect sizes from models without those covariates in the model. One can generate these plots from the final model or an initial model without covariates. In cases of high shrinkage it is important to sample from conditional distributions of individual parameter values when creating empirical forest plots.
References:
[1] E. N. Jonsson and J. Nyberg, “Using forest plots to interpret covariate effects in pharmacometric models,” CPT Pharmacomet. Syst. Pharmacol., p. psp4.13116, Feb. 2024, doi: 10.1002/psp4.13116.
[2] M. Lavielle and B. Ribba, “Enhanced Method for Diagnosing Pharmacometric Models: Random Sampling from Conditional Distributions,” Pharm. Res., vol. 33, no. 12, pp. 2979–2988, 2016, doi: 10.1007/s11095-016-2020-3.
[3] S. L. Beal, L. B. Sheiner, A. J. Boeckmann, and R. Bauer, “NONMEM 7.5.1 User’s Guides.” ICON plc, Gaithersburg, MD, USA, 2024 1989. [Online]. Available: https://nonmem.iconplc.com/#/nonmem750/guides.
[4] R Core Team, “R: A language and environment for statistical computing,” R Foundation for Statistical Computing, Vienna, Austria, manual, 2024. [Online]. Available: https://www.R-project.org/.
