The study evaluates the effectiveness of Epic’s Risk of hospital-acquired acute kidney injury (HA-AKI) model, utilizing data from a cohort within the Mass General Brigham network. Despite demonstrating moderate success in predicting HA-AKI, the model exhibits limitations, particularly in accurately identifying high-risk patients. Performance variations across different stages of HA-AKI highlight the need for further refinement and clinical validation. Findings underscore the potential of predictive analytics in guiding clinical decisions but emphasize the importance of cautious interpretation and additional research before widespread implementation.
In the realm of healthcare, predictive analytics tools hold promise for identifying and mitigating risks associated with hospital-acquired conditions. Recently, researchers from Mass General Brigham Digital investigated the effectiveness of Epic’s Risk of hospital-acquired acute kidney injury (HA-AKI) model. Published in NEJM AI, their study revealed insights into the model’s predictive capabilities and its potential impact on patient outcomes.
The Epic Risk of HA-AKI model operates by continuously monitoring inpatient encounters for early indicators of kidney injury, such as elevations in serum creatinine levels. Despite its innovative approach, predicting HA-AKI poses challenges due to the complexity of its etiology. Nonetheless, the integration of artificial intelligence (AI) and machine learning offers avenues for enhancing predictive analytics in this domain, although concerns regarding model performance persist.
To evaluate the efficacy of Epic’s model, the research team analyzed data from a cohort comprising 39,891 adult patients within the Mass General Brigham network. Patient information, including demographics, comorbidities, laboratory results, and predictive model scores, was extracted from electronic health records (EHRs). Performance assessment metrics such as the area under the receiver operating curve (AUROC) and the area under the precision-recall curve (AUPRC) were utilized.
Results indicated that while the model demonstrated moderate success, certain limitations were evident. The incidence of stage 1 HA-AKI within the cohort was notable, standing at 24.5 percent. At the encounter level, the model exhibited an AUROC of 0.77 and an AUPRC of 0.49, indicating its ability to discriminate between positive and negative cases. However, when considering a prediction horizon of 48 hours, the AUROC dropped slightly to 0.76, with an AUPRC of 0.19.
Despite achieving a positive predictive value of 88 percent, the model’s performance varied depending on the risk level of patients. It excelled in identifying low-risk individuals but struggled with accurately predicting HA-AKI onset in higher-risk patients. Furthermore, performance discrepancies were observed across different stages of HA-AKI, with predictions for stage 1 being more reliable compared to later stages.
Comparisons with internal validation data from Epic revealed a disparity in performance metrics, suggesting a potential overestimation of the model’s effectiveness. The researchers cautioned against high false-positive rates associated with model deployment, emphasizing the need for further clinical validation.
Dr. Sayon Dutta, lead study author and member of Mass General Brigham Digital’s Clinical Informatics team, emphasized the model’s potential to support clinical decisions but highlighted the necessity of additional studies before widespread implementation. Identifying HA-AKI risk could aid in guiding treatment strategies, such as avoiding nephrotoxic medications, thereby improving patient outcomes.
This study contributes to a growing body of research scrutinizing predictive analytics tools in healthcare. Recently, concerns were raised regarding the Epic Sepsis Model (ESM), with findings suggesting potential limitations in accuracy and timeliness. A team from the University of Michigan underscored the importance of clinical validation, particularly in scenarios where clinicians may preemptively diagnose and treat conditions like sepsis before meeting formal criteria.
The evaluation of Epic’s Risk of HA-AKI model reveals valuable insights into its predictive capabilities and clinical utility. While the model demonstrates promise in identifying patients at risk of HA-AKI, its performance varies across different risk levels and stages of the condition. Addressing these limitations is crucial for enhancing the model’s reliability and ensuring its effective integration into clinical practice. Further research and validation are necessary to maximize the model’s potential for improving patient outcomes and guiding evidence-based decision-making in healthcare settings.
