Evaluation of IRT-derived disease severity as a predictor of clinical success in critically ill patients with severe antibiotic resistant infections
Leticia Arrington(1,2), Viktor Rognås (1), Lena E. Friberg(1), Mical Paul (3), Mats O. Karlsson(1), Sebastian Ueckert(1)
(1) Department of Pharmacy, Uppsala University, Uppsala, Sweden (2) Merck& Co., Inc., Kenilworth, NJ, USA, (3) Institute of Infectious Diseases, Rambam Health Care Campus, Haifa, Israel
Introduction: The Sequential Organ Failure Assessment score (SOFA) is used to measure the severity of organ dysfunction and is widely used in the critical care community especially in those patients with severe bacterial infections [1]. The SOFA score contains 6 subscores that quantify the level of functioning of 6 different organ systems: central nervous system, respiratory, cardiac, coagulation, hepatic and renal. Each SOFA subscore contains 5 categories in increasing disease severity. Typically, the SOFA score is reported as a composite score. However, there may be more to learn about disease severity and predictability of clinical success/failure at the subscore level. Item response theory (IRT) is a methodology that describes the relationship between the probability of response at the item level and a patient’s disease severity. This approach provides an opportunity to obtain a disease score that is based on a weighted average of the individual item data. Items with a stronger relationship to the latent variable (i.e., disease severity) implicitly receive more weight and have a stronger influence on the disease status. Herein SOFA subscores are considered items.
Objectives: The objective of this work was to compare total SOFA score, SOFA-subscores and IRT-derived disease score at randomization in their ability to predict clinical success.
Methods: Data: The AIDA trial (N=406) was a randomized control, superiority study that evaluated colistin monotherapy and colistin+meropenem combination therapy in patients with confirmed meropenem resistant infections [2,3]. The primary outcome of the trial was clinical success at 14 days after randomization, defined as a composite of the following: survival, stable or improved respiratory status (for patients with pneumonia), hemodynamic stability, microbiological cure (for patients with bacteremia), and stability in SOFA score. The primary outcome was treated as a binary variable in the analysis.
IRT modeling: A unidimensional graded response IRT model was developed for the randomization day SOFA scores using the R package mirt [4]. Item characteristic curves were leveraged to visualize the item parameters. Item information was calculated to identify and rank which subscores were most informative on the latent variable across the range of disease severity.
Clinical success prediction: A logistic regression was performed testing each of the following as a predictor of clinical success i) total SOFA ii) each SOFA subscore independently, iii) pattern response of all SOFA subscores combined, iv) IRT-derived disease score and v) most informative subscores. To compare the predictive ability of each of these measures, a receiver operating characteristic (ROC) curve was generated in R [5] from the results of the logistic regression to determine the ability of these metrics to predict clinical success. The resulting area under the curve (AUC) was used for comparison.
Results: The respiratory, cardiac and CNS function subscores were the most discriminatory and informative subscores and best described the overall disease status within this patient population. The coagulation and hepatic function subscores did not contribute much information to the assessment as they demonstrated low discrimination values of ~0.3, indicating that the data informing these subscores are not explanatory in this population. However, inclusion of these subscores did not impact total test information in the disease severity range of patients studied.
ROC AUCs were calculated for total SOFA (0.7276), all SOFA items (0.7373), IRT-derived disease score (0.7349), and most informative items (0.7287). At an individual subscore level, the more informative subscores were more predictive of clinical success with a higher true positive probability rate (i.e., AUC closer to 1). However, none of the subscores resulted in an AUC >0.68.
Conclusions: An IRT model was developed for SOFA score. In such a severe acute disease total SOFA score, IRT-derived disease score as well as the most informative individual subscores had a similar predictive value. It appears there was no significant benefit of using the IRT-derived scores for predicting clinical success.
References:
[1] Singer M, Deutschman CS, Seymour CW, ShankarHari M, Annane D, Bauer M, et al. The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis3). JAMA 2016,315:801. https://doi.org/10.1001/jama.2016.0287.
[2] Paul M, et al. Colistin alone versus colistin plus meropenem for treatment of severe infections caused by carbapenem-resistant Gram-negative bacteria: an open-label, randomised controlled trial. Lancet Infect Dis 2018;18:391–400. https://doi.org/10.1016/S1473-3099(18)30099-9
[3] Dickstein Y, Leibovici L, Yahav D, EliakimRaz N, Daikos GL, Skiada A, et al. Multicentre openlabel randomised controlled trial to compare colistin alone with colistin plus meropenem for the treatment of severe infections caused by carbapenem resistant
Gram negative infections (AIDA): A study protocol. BMJ Open 2016;6:e009956. https://doi.org/10.1136/bmjopen2015009956.
[4] Chalmers RP. mirt: A Multidimensional Item Response Theory Package for the R Environment J Stat Soft. 2012;48(6):1-29. doi: 10.18637/jss.v048.i06.
[5] Robin X, Turck N, Hainard A, Tiberti N, Lisacek F, Sanchez J, Müller M (2011). “pROC: an open-source package for R and S+ to analyze and compare ROC curves.” BMC Bioinformatics, 12, 77.