Table of Contents
More than 90% of health systems have deployed third-party AI solutions, but only 44% have a dedicated platform for testing those tools before rollout, according to a new report from the Center for Connected Medicine at UPMC and KLAS Research, revealing a significant gap in health systems AI testing platforms infrastructure relative to adoption speed.
What the Report Behind This Health Systems AI Testing Platforms Gap Examined
The report, “Validation and Trust: How Systems Are Testing and Governing AI Solutions,” draws on responses from more than two dozen health system leaders. It examines how organizations are adopting and governing AI amid rapid deployment across administrative and clinical workflows.
The Most Common AI Use Cases Driving This Testing Gap
Clinical documentation is the most common AI use case, cited by 52% of respondents, followed by revenue cycle, coding and billing applications at 36%. This concentration in documentation and revenue cycle functions suggests health systems are prioritizing AI tools with relatively well-defined, measurable outputs before expanding into more complex clinical decision-support applications.
Testing Practices Within This Health Systems AI Testing Platforms Landscape
Testing is nearly universal, with 92% of health systems evaluating third-party AI tools before deployment, though validation methods range from formal vendor protocols to informal pilot programs. This wide variance between formal and informal testing approaches, despite near-universal testing overall, highlights the core finding: evaluation happens broadly, but the dedicated infrastructure to standardize and scale that evaluation remains uncommon.
Why Ad Hoc AI Strategies Remain the Norm
Nearly two-thirds of organizations, 63%, describe their AI strategy as developing or ad hoc rather than fully established. Limited resources, insufficient time and shortages of specialized talent rank as the leading barriers to adoption, and respondents report no consensus on the metrics used to measure AI success.
Why the Lack of Success Metrics Consensus Matters for Health Systems AI Testing Platforms
Without agreed-upon metrics for measuring AI success, systems may struggle to justify continued investment in dedicated testing infrastructure, even as they continue deploying third-party AI tools at a rapid pace. This gap between deployment speed and measurement standardization could leave organizations unable to clearly demonstrate return on investment to leadership or board stakeholders.
UPMC’s Perspective on What Comes Next
Rob Bart, MD, chief medical information officer at UPMC, said in the report that implementation is “only the first step,” with health systems now focused on building the governance structures and testing capabilities needed to ensure AI delivers measurable value.
What This Health Systems AI Testing Platforms Report Means Going Forward
Given that 92% of systems already test AI tools in some form while only 44% have a dedicated testing platform, this gap suggests most organizations are relying on informal or vendor-driven validation processes rather than building internal, standardized infrastructure. As health systems continue deploying AI across clinical documentation and revenue cycle functions, the persistent shortage of specialized talent and limited resources identified in this report may continue slowing the transition from ad hoc testing toward the more formalized governance structures Bart describes as the necessary next step.
What to Watch Going Forward
As health systems work to close this testing infrastructure gap, industry observers will likely watch whether organizations begin converging on shared metrics for measuring AI success, an area the report found lacks consensus industrywide. Given that clinical documentation and revenue cycle applications currently dominate AI use cases, how health systems extend testing rigor to more clinically consequential AI applications may become an increasingly important benchmark for evaluating the maturity of health systems AI testing platforms across the industry.
For more healthcare industry updates, insights and news, visit DistilINFO. Click here to subscribe to stay informed.
