Commentary

What price perfection? Calibration and discrimination of clinical prediction models

Few studies have described the aetiologies of neonatal cholestasis, and the overall neonatal cholestasis-related mortality (NCM) rate is unclear. We investigated the aetiology and outcome of neonatal cholestasis in a tertiary hospital and developed an NCM prediction model for these patients.

Patients aged <100 days with serum direct bilirubin (DB) levels of >1.0 mg/dL were retrospectively screened. Diagnostic and laboratory data during the 8-week follow-up period after enrolment between 2005 and 2020 were extracted digitally, and medical charts were reviewed manually by clinicians. Logistic regression was used to derive a prediction model for the 1-year mortality outcome of neonatal cholestasis, and performance evaluation and external validation were conducted for the NCM prediction model.

We enrolled 4028 neonates with DB of >1.0 mg/dL at least once. Prematurity and birth injury (35.4%), complex heart anomalies (18.6%), liver diseases (11.4%), and gastrointestinal anomalies (9.2%) were the most common aetiologies; 398 (9.9%) patients died before one year of age. The peak value of DB was positively correlated to the 1-year mortality rate. In the multivariate analysis, simple laboratory indices, including platelet, prothrombin time, aspartate aminotransferase, albumin, direct bilirubin, creatinine, and C-reactive protein, were independent predictors of 1-year mortality outcome of complete-case subjects. Using these laboratory indices, a logistic regression-based NCM prediction model was constructed. It showed acceptable performances on discrimination (area under the curve, 0.916), calibration (slope, 1.04) and Brier scoring (0.072). The external validation of the sample (n = 920) from two other centres also revealed similar performance profiles of the NCM model.

Various aetiologies of neonatal cholestasis were identified in a tertiary hospital, resulting in unfavourable outcomes of a large proportion. The NCM prediction model may have the potential to help clinicians to be aware of high-risk neonatal cholestasis.

Ministry of Health & Welfare, Republic of Korea.

Model calibration, critical to the success and safety of clinical prediction models, deteriorates over time in response to the dynamic nature of clinical environments. To support informed, data-driven model updating strategies, we present and evaluate a calibration drift detection system. Methods are developed for maintaining dynamic calibration curves with optimized online stochastic gradient descent and for detecting increasing miscalibration with adaptive sliding windows. These methods are generalizable to support diverse prediction models developed using a variety of learning algorithms and customizable to address the unique needs of clinical use cases. In both simulation and case studies, our system accurately detected calibration drift. When drift is detected, our system further provides actionable alerts by including information on a window of recent data that may be appropriate for model updating. Simulations showed these windows were primarily composed of data accruing after drift onset, supporting the potential utility of the windows for model updating. By promoting model updating as calibration deteriorates rather than on pre-determined schedules, implementations of our drift detection system may minimize interim periods of insufficient model accuracy and focus analytic resources on those models most in need of attention.

Risk prediction models are statistical models that estimate the probability of individuals having a certain disease or clinical outcome based on a range of characteristics, and they can be used in clinical practice to stratify disease severity and characterize the risk of disease or disease prognosis. With technological advancements and the proliferation of clinical and biological data, prediction models are increasingly being developed in many areas of nephrology practice. This article guides the reader through the process of creating a prediction model, including (i) defining the clinical question and type of model, (ii) data collection and data cleaning, (iii) model building and variable selection, (iv) model performance, (v) model validation, (vi) model presentation and reporting, and (vii) impact evaluation. An example of developing a prediction model to predict mortality after intensive care unit admission for patients with end-stage kidney disease is also provided to illustrate the model development process.