Understanding the AI Chatbot Threat
The misuse of artificial intelligence chatbots such as ChatGPT, Gemini, and Copilot in healthcare settings has emerged as the most significant health technology hazard for 2026, according to ECRI, a leading nonprofit patient safety organization. This alarming designation comes from the organization’s comprehensive annual assessment, which evaluates potential dangers through member surveys, extensive literature reviews, rigorous medical device testing in their laboratories, and thorough investigations of patient safety incidents across healthcare facilities.
The designation of AI chatbots as the top hazard represents a paradigm shift in healthcare technology risks. Unlike traditional medical device failures or equipment malfunctions, this threat stems from the widespread adoption of tools never intended for clinical use. ECRI’s executive brief provides journalists and healthcare professionals with detailed insights into these emerging dangers and practical recommendations to reduce the risks of patient harm.
The ECRI Assessment Process
ECRI’s annual top 10 hazards list serves as a critical early warning system for the healthcare industry. The organization’s multifaceted evaluation process combines real-world incident data with predictive analysis, enabling healthcare systems to proactively address emerging threats before they result in widespread patient harm. This year’s focus on AI chatbots reflects the rapid integration of consumer-grade artificial intelligence into clinical workflows without adequate safeguards or regulatory oversight.
Why AI Chatbots Pose Healthcare Risks
The fundamental problem with AI chatbots in healthcare isn’t the technology itself becoming dangerous, explains ECRI’s president and CEO, Marcus Schabacker, M.D., Ph.D. Rather, the risk emerges when chatbot outputs “feel helpful and definitive,” leading people to rely on them without critical evaluation. This false confidence creates a dangerous gap between perceived reliability and actual accuracy.
Regulatory Gaps and Unintended Use
ECRI experts emphasized during a January webcast that popular chatbots like Gemini and Copilot are not specifically designed for healthcare applications. “They’re not medical devices. They’re not FDA-approved and regulated for that purpose,” said Rob Schluth, a principal project officer of device safety at ECRI. However, as these tools become deeply integrated into daily life, healthcare professionals and patients increasingly turn to them for medical advice, treatment options, and health-related guidance—uses for which they were never intended or validated.
How Healthcare Professionals Misuse AI Tools
The integration of AI chatbots into healthcare workflows has occurred organically and without formal oversight. Clinicians use these tools to research health conditions, identify potential treatment options for patients, and even generate clinical documentation. Hospital administrators and staff leverage them for purchasing decisions, report writing, and operational planning. Each of these applications introduces potential risks when unverified AI-generated information influences medical decisions.
The Illusion of Expertise
Large language models like ChatGPT are designed to maintain user engagement rather than provide clinically validated medical guidance. These systems don’t challenge or correct flawed assumptions embedded in user queries. Instead, they generate responses based on patterns in their training data, potentially reinforcing dangerous misconceptions or providing inaccurate medical information presented with unwarranted confidence.
The Technical Limitations of Large Language Models
A widespread misconception suggests that LLMs understand the content they generate, explains Christie Bergerson, Ph.D., a device safety analyst with ECRI. In reality, these systems predict the next word based on statistical patterns and probabilities derived from their training data. They identify words that commonly appear together in discussions about specific topics and arrange them into coherent-sounding sentences—without any genuine comprehension of medical concepts, contraindications, or individual patient contexts.
Hallucinations and Fabrications
AI chatbots can fabricate information—a phenomenon known as “hallucination”—while presenting it with complete confidence. These systems are programmed to sound definitive rather than express uncertainty. They won’t say “I’m not sure” or “I can’t help you with this,” even when operating beyond their knowledge base or generating unreliable information. This creates particular danger in healthcare, where confident misinformation can directly impact patient safety.
Appropriate Use Cases
Despite these limitations, chatbots can serve valuable purposes when used appropriately. They excel at brainstorming, providing background information, and explaining complex topics in accessible language. However, users must verify all information through reliable sources and “check in with a human expert before taking actions or making decisions based off an LLM’s response,” Bergerson emphasized.
Additional Critical Health Technology Hazards
Beyond AI chatbots, ECRI’s 2026 report identifies nine additional technology-related threats to patient safety:
Digital Darkness Events
Unpreparedness for sudden loss of access to electronic systems poses the second-greatest risk. Cyberattacks, natural disasters, vendor outages, and internal system failures can paralyze healthcare facilities, delaying treatment and jeopardizing patient safety. Health systems must strengthen disaster recovery planning, establish downtime procedures, implement reliable data backup processes, and conduct regular training and safety drills.
Counterfeit Medical Products
Substandard and falsified medical products are reaching U.S. markets “with alarming frequency.” These counterfeit devices and supplies pose serious risks even when they function as designed. Healthcare providers must strengthen supply chains, demand high-quality products, and implement protective measures against flawed products.
Home Diabetes Device Recall Communications
Communication failures regarding recalls and updates for continuous glucose monitors and other home diabetes management technologies can leave patients using dangerous devices. Patients should proactively monitor safety notices, while providers and manufacturers must ensure clear, timely product safety information reaches end users.
Medical Connection Errors
Inappropriate connections of syringes or tubing to patient lines intended for different purposes can introduce medications, solutions, IV nutrition, or gases into wrong pathways with severe consequences. Hospitals should adopt safety connector devices designed to prevent such misconnections.
Perioperative Medication Safety
Underutilization of medication safety technologies in surgical settings allows errors with high-alert medications like opioids. Healthcare organizations should implement barcode medication administration systems where workers scan patient wristbands and medication labels to ensure proper matching.
Device Cleaning Instructions
Inadequate cleaning instructions for reusable medical devices can spread infections or cause device damage. The wide variation in manufacturer reprocessing instructions complicates proper sterilization. Health organizations should evaluate reprocessing requirements before purchasing decisions.
Legacy Device Cybersecurity
Older software-based devices lacking current cybersecurity updates provide entry points for hackers. Health systems should consider disconnecting vulnerable devices from networks, deploying security management tools, or planning device replacements.
Unsafe Clinical Workflows
Implementing healthcare technologies without comprehensive user training can prompt unsafe workarounds and various patient harms. Health systems must conduct thorough workflow analyses before deploying new technology and establish comprehensive training programs.
Water Quality in Sterilization
Poor water quality during instrument sterilization exposes patients to infectious pathogens and can corrode or contaminate instruments. Health systems should routinely assess processed device cleanliness and monitor water quality standards.
Protecting Patients from Technology Risks
ECRI’s comprehensive assessment provides healthcare organizations with actionable strategies to mitigate these technology-related hazards. By understanding these risks and implementing recommended safeguards, healthcare systems can harness the benefits of modern technology while protecting patients from emerging threats. The key lies in maintaining critical evaluation of AI-generated information, strengthening infrastructure resilience, ensuring supply chain integrity, and prioritizing comprehensive staff training on all new technologies.
