The question healthcare CIOs are asking about artificial intelligence has changed. A year ago, the debate centered on whether to adopt. Now, it centers on whether organizations can actually see what their AI is doing once it’s in the building, a shift eight health system leaders describe as the rise of invisible AI healthcare governance risk.
Why Invisible AI Healthcare Governance Risk Is Spreading Undetected
“One of the most dangerous trends in health IT today is the invisible integration of AI into clinical workflows,” said Charity Dorazio, CIO of Adventist HealthCare in Gaithersburg, Md. “This can occur when vendors enable new AI features without informing customers or when staff adopt AI tools independently. AI that is not properly evaluated, monitored, and governed can create significant risks, including data exposure, gaps in oversight, and the inability to implement effective monitoring and control processes.”
Why Organizations Only Govern What They Know About
Dorazio’s concern reflects a dynamic that has become increasingly common as vendor-embedded AI proliferates: organizations that believe they have governed AI adoption discover they have governed only the tools they know about. Shadow adoption, staff using consumer AI products, vendors silently enabling features, is expanding the perimeter in ways most current governance frameworks were not designed to address.
Model Drift and Automation Bias Within This Invisible AI Healthcare Governance Risk
Carter Smith, managing director of Cone Health Ventures in Greensboro, N.C., has seen the downstream consequence of that visibility gap. “Most health systems can tell you how many AI tools they’ve launched. Far fewer can tell you which ones are still performing the way they did on day one,” he said. “Models drift, patient populations change, documentation patterns shift, but the monitoring to catch that rarely ships with the pilot.”
How Automation Bias Compounds This Problem
The compounding effect is automation bias, the tendency for clinicians and staff to stop questioning a tool once it is embedded in a workflow. “That’s how quiet degradation in model performance becomes a patient safety or compliance problem nobody sees coming,” Smith said. His prescription is structural: every AI deployment should have a named owner, a performance baseline, and a kill switch. “If a tool can’t clear that bar, it isn’t ready for production.”
Building AI on Unready Foundations Within This Invisible AI Healthcare Governance Risk
Bob Berbeco, CIO of Mahaska Health in Oskaloosa, Iowa, said there is also risk in building AI on top of existing structures if they haven’t been updated. “The most dangerous trend in healthcare IT is the rush to layer more technology, especially AI, into workflows and data foundations that are not ready for it,” he said. “Organizations that chase innovation without first addressing data quality, interoperability, cybersecurity, governance, and workflow design don’t create value faster. They create complexity and additional risk at an accelerated rate.”
Why Misplaced Credibility Concerns Berbeco Most
“AI makes this risk especially acute,” he said. “Once it’s embedded in a trusted clinical workflow, people may give its recommendations more credibility than earned. That’s what concerns me, not whether healthcare adopts AI fast enough, but whether we hold onto human judgment, accountability, and trust as we do it.”
Scaling Inaccurate Decisions Within This Invisible AI Healthcare Governance Risk
Nonku Kunene Adumetey, director of population health data analytics at ChristianaCare Health System in Newark, Del., put the same concern in operational terms. “When organizations implement tools without validating the data, monitoring for bias or integrating insights into clinical and operational workflows, they risk scaling inaccurate decisions rather than improving care,” she said. “The real opportunity is not simply to adopt more technology, but to ensure it produces trusted, actionable insights that improve outcomes, reduce disparities and support the people delivering care.”
Data Stewardship and Expanding Cybersecurity Threats
Ray Lowe, senior vice president and CIO of AltaMed in Los Angeles, connected the data governance argument to the cybersecurity surface that AI simultaneously expands. “AI is only as good as the data behind it, and without strong data stewardship, organizations risk automating bad decisions at scale,” he said. Lowe also flagged that cybercriminals are now leveraging AI to accelerate phishing, identify vulnerabilities, and automate attacks faster than organizations can respond, warning that a successful attack can now disrupt patient care and erode trust, not just cause a data breach.
Losing Focus on Fundamentals Within This Invisible AI Healthcare Governance Risk
Susan Goodson, senior vice president and chief digital information officer at Lurie Children’s in Chicago, argued that the proliferation of AI tools has introduced a different kind of organizational risk: the distraction of chasing capability at the expense of operational foundation. “The most dangerous trend in health IT is taking our eye off the ball,” Goodson said. “The most successful organizations are focused on the fundamentals: resilient operations, secure and trustworthy data, strong governance, and deploying technology that has impact for our patients and our employees.”
The AI Arms Race Between Health Systems and Payers
Judd Hollander, MD, senior vice president for health delivery innovation and chief virtual care officer at Jefferson Health in Philadelphia, pointed to a systemic dynamic beyond any single organization’s walls. “The AI arms race is the most dangerous trend,” Hollander said. “I fear that health systems will race to utilize AI to decrease denials and improve pre-authorization while payers will race to utilize AI to maintain denials and limit pre-authorization. Thus, we will all drive up healthcare costs reaching the exact same status quo we currently have.” His conclusion: “We cannot simply make poorly designed systems more efficient. We must re-design the system.”
What This Invisible AI Healthcare Governance Risk Discussion Means Going Forward
Across these eight perspectives, the through-line is less about caution toward AI than about the institutional discipline required to use it responsibly. None of the leaders quoted are arguing for slower adoption; instead, they’re arguing that accountability structures, governance frameworks, performance monitoring, data foundations, named owners, and kill switches, need to ship with the technology rather than follow it. Given the consistency of this message across health systems of very different sizes and specialties, other CIOs evaluating their own AI governance maturity may find these shared principles, particularly Smith’s named-owner-and-kill-switch framework, a practical starting point for auditing existing deployments.
What to Watch Going Forward
As more health systems confront the gap between what they believe they’ve governed and what AI tools are actually operating within their environments, industry observers will likely watch whether formal shadow AI detection and vendor feature-disclosure requirements become standard practice across the industry. Given Hollander’s warning about an AI arms race between health systems and payers potentially locking in the current denial and pre-authorization status quo at higher cost, this invisible AI healthcare governance risk discussion may extend beyond individual organizational governance into broader questions about how AI reshapes, or fails to reshape, the structural incentives underlying healthcare’s most persistent administrative frictions.
For more healthcare industry updates, insights and news, visit DistilINFO. Click here to subscribe to stay informed.
