Table of Contents
Healthcare is at the forefront of other industries when it comes to its adoption of artificial intelligence, with organizations of all shapes and sizes using AI to transform clinical care, improve administrative workflows and advance research. But there are still growing pains, and some significant imperatives around healthcare AI governance security need addressing, according to leaders from the National Institute of Standards and Technology and the Coalition for Healthcare AI.
The Core Challenge Behind Healthcare AI Governance Security
As the front end of the AI adoption curve relies heavily on vendor assessments to screen for clinical safety, data integrity and model validation before AI is integrated into clinical workflows, ongoing AI monitoring and continued navigation of emerging legal and regulatory uncertainties are needed to safeguard patients and data.
A Central Question for the Industry
“As healthcare moves from experimenting with AI to deploying it at scale, who will define what trustworthy actually means? And how do organizations put that into practice?” Baxter Lee, president of cybersecurity firm Clearwater, asked during a recent webinar featuring Brendan Hill, CHAI’s head of operations and general counsel, and Julie Chu, NIST’s applied cybersecurity vision director.
A Year of Optimism Around Healthcare AI Governance Security
“I find it interesting that healthcare, at least from my perspective, is one of the industries that’s kind of leading AI adoption right now,” said Hill, who was previously the regulatory strategy and compliance manager at Mayo Clinic Platform and a member of its Software as a Medical Device Review Board. Healthcare is also cognizant and applying lessons learned, according to Chu, who previously directed the Governance, Risk Management, and Compliance Division within HHS’s Office for Information Security. “There is a lot of leaning in, in terms of being on top of the governance piece,” she said.
Defining Organizational Trustworthiness
Chu said organizations are asking, “How do we define our trustworthiness within our organizations? How do we look at that metric or that measure for what we think as an organization should be trustworthy or should be ethical or should be all the other AI risk management framework characteristics for what trustworthy AI is?”
Starting With AI Amid Growing Tool Portfolios
Anecdotal evidence indicates that some health systems are considering between 200 and 500 AI tools in their queues, Hill said. Both the Joint Commission and CHAI have put together playbooks to help healthcare organizations manage the challenges those investments bring. The Joint Commission recently released governance playbooks as part of its voluntary AI responsibility certification program, and CHAI launched its national PULSE initiative this month to engage public health practitioners in co-developing and piloting five key generative AI use cases.
The Hardest Part: AI Monitoring
The goals of CHAI’s playbooks are to “position organizations to think critically,” Hill said, focusing on control methodologies to help organizations understand the entire AI lifecycle. But one of the hardest parts for organizations to implement is AI monitoring, largely due to how fast the technology is moving. “There are no clearly defined metrics for what we should monitor,” Hill said. “Science hasn’t quite caught up with where people are adopting and what people are doing. There’s still very much a ‘black box’ sense to AI, even though we have more in-depth understanding about it.”
Training and ROI Challenges Within Healthcare AI Governance Security
Training, which also affects AI uptake within organizations, is a second key challenge, Hill said. “I think one of the more emerging issues that organizations are starting to struggle with is the training and education component,” he said. “How do you train a staff member to use AI responsibly?” A third key hurdle is scaling the use of AI so that it brings a return on investment. “Not many organizations, if any, have been able to tackle that yet,” Hill said. “But I think there’s going to be a lot of thought that’s put into that in the coming years.”
Establishing Risk-Based Tiers
Healthcare leaders must treat AI vendors as a supply chain issue, contractually requiring third-party vendors to notify the organization before baking AI into existing tools as organizations build auditable trails to justify their AI governance choices in the event of an incident, Hill said. Smart governance accelerates adoption by categorizing AI into low, medium and high-risk tiers. “Start with the low-hanging fruit,” he said. When low-risk applications move quickly through streamlined checks, high-risk tools can undergo intensive reviews.
Why Governance Speeds Rather Than Slows Adoption
“The reason why governance does the opposite of slowing [AI] down is you’re getting the right people with the right expertise solving the right problem along the governance chain,” Hill said. “An enterprise governance mechanism should own AI as a leadership team,” Chu added. Organizations succeeding at this take a risk-based approach where each level of risk requires a specific level of governance to be applied, rather than getting bogged down in details for everything.
Vendor Vetting for Clinical Safety
After studying more than 800 intake questions from 15 different institutions, Hill said clinical safety and efficacy are the chief concerns, so vendor assessments must evaluate tools for fairness and nondiscrimination. Considering how an ambient scribe might work across clinical practices, for example, requires examining legal and regulatory compliance, security and access controls, workflow integration, and data integrity with demonstrable data lineage. CHAI has worked to bring its intake questionnaire down to about 60 questions it plans to release in the coming months.
How Changing Cybersecurity Frameworks Support Healthcare AI Governance Security
Chu, who co-led the development of the Cybersecurity Act of 2015 Section 405, said NIST is looking at AI from a cybersecurity perspective: “How is AI changing the guidance, the guidelines, frameworks, standards that we have today in terms of cybersecurity and privacy?” Governance and enterprise risk management go “hand in hand,” she said. The NIST AI Risk Management Framework, which is undergoing updates with a public comment period expected to open later this year, includes a set of trustworthy characteristics that are AI-specific and can be integrated with any organization’s existing governance structure.
Additional NIST Resources
Chu recommended other NIST-based resources, including NIST’s Small Business Program, Quick Start Guides on cybersecurity, enterprise risk management and workforce management, and the Cyber AI Profile. “We use the NIST AI RMF to inform a lot of our frameworks, as well,” Hill added.
Shifting to Product-Centric AI Within Healthcare AI Governance Security
Organizations are moving away from top-down, centralized AI committees toward product managers who directly own specific use cases and act as points of contact between clinical staff and vendors, enabling operational and contractual guardrails and empowering health systems to negotiate critical protections into contracts, such as the right to pre-approve model updates, tie payments to performance benchmarks, or disable unverified features.
Why Cross-Collaboration Strengthens Contracts
Cross-collaboration with business users will ultimately lead to stronger contracts, Hill said. “The lawyer is not going to have the subject matter expertise to say, ‘We should have a number of different things in a contract to protect our organization,’ because they just don’t understand the nuances of how it’s going to be used in the workflow,” he said.
Why Elevating Risk Is a Sign of Good Governance, Not Failure
“It is very important that people realize, when you elevate a risk, it’s not failure,” Chu said. “It means you’re managing and you have a good governance process, method, approach. That’s why a risk is elevated, or that’s why it’s become much more important or the priority has changed.” Both leaders acknowledged the difficulty of the work. “It is not easy to do governance well,” Chu said. “It is hard work,” Hill agreed. “It is a process, and dedication is needed when pulling these together,” Chu added.
What This Means for Health Systems Building Their Own AI Governance
As health systems continue expanding their AI tool portfolios into the hundreds, this discussion around healthcare AI governance security underscores that successful governance isn’t about slowing adoption, but about routing decisions to the right people with the right expertise at each risk tier. Organizations still struggling with monitoring metrics, staff training, and ROI measurement may find CHAI’s forthcoming 60-question vendor intake framework and NIST’s updated AI Risk Management Framework useful starting points as they build out their own structures.
What to Watch Going Forward
With NIST’s AI RMF update expected to open for public comment later this year, and CHAI planning to release its refined vendor intake questionnaire in the coming months, health systems building or refining their governance frameworks should watch for how these evolving standards shape best practices industrywide. Given how quickly the underlying technology continues to change, both Hill and Chu’s emphasis on treating governance as an ongoing process, rather than a one-time policy exercise, suggests organizations should expect their healthcare AI governance security frameworks to require continuous revision rather than a single definitive setup.
For more healthcare industry updates, insights and news, visit DistilINFO. Click here to subscribe to stay informed.
