Google and the Department of Defense have collaborated to create an Augmented Reality Microscope (ARM) powered by artificial intelligence. This advanced microscope aids pathologists in diagnosing cancer with precision, highlighting tumor locations, and assessing severity. While still in its early stages, the ARM aims to be a valuable tool for pathologists, particularly in remote areas. It complements traditional microscopes and offers cost-effective solutions for storing diagnostic data. The ARM’s development is ongoing, with potential widespread availability for government users in the future, promising improved healthcare outcomes.
In his office at the VA hospital in Seattle, Dr. Nadeem Zafar found himself amid a professional disagreement. Dr. Zafar, a pathologist specializing in clinical lab tests for conditions like cancer, was consulted on a prostate cancer case. While it was evident that the patient had cancer, there was a disparity in the severity assessment between Zafar and his colleague. Zafar believed the cancer was more aggressive.
To settle this dispute, Zafar turned to an extraordinary tool in the field of pathology—an AI-powered microscope jointly developed by Google and the U.S. Department of Defense. This innovative microscope, known as an Augmented Reality Microscope (ARM), swiftly examined the case and confirmed Zafar’s assessment. In mere seconds, the AI identified the exact region of the tumor that Zafar had pinpointed as more aggressive, convincing his colleague.
This ARM technology represents a promising advancement in the world of pathology, although it is still in its early stages and not yet in active clinical use. It is envisioned as a valuable tool for pathologists who lack immediate access to a second opinion.
Currently, there are 13 ARMs in existence, with one located at a Mitre facility near Washington, D.C. Researchers at Mitre are collaborating with the ARM to assess its practicality and potential vulnerabilities in clinical settings.
At first glance, the ARM resembles a typical microscope found in a high school biology classroom. However, it is intricately connected to a computer tower housing AI models. When a glass slide is placed beneath the microscope, the AI can outline the location and severity of cancer, presenting this information as a bright green line visible through the eyepiece and on a separate monitor. It also generates a black-and-white heat map on the monitor, illustrating the cancer boundary in a pixelated format.
The ARM was designed with ease of use in mind, seamlessly integrating with the established workflow of pathologists. This utility-driven approach addresses the challenges faced by pathologists due to workforce shortages and increasing caseloads, particularly with an aging population. Missing a critical diagnosis due to the workload can have serious consequences for patients.
While various organizations have explored digital solutions for pathology, these often come with significant infrastructure and storage costs. The ARM does not aim to replace digital pathology systems but rather complements them. Pathologists can capture screen grabs of slides using the ARM’s software, significantly reducing storage expenses.
The cost of implementing the ARM in healthcare systems typically ranges from $90,000 to $100,000. Importantly, it ensures that the physical microscope remains an integral part of the pathologists’ diagnostic process.
Dr. Niels Olson, the Chief Medical Officer of the Defense Innovation Unit (DIU) at the Department of Defense, understands the challenges faced by pathologists intimately. The DIU, established in 2015, serves as a bridge between military and commercial technology. Olson, a pathologist himself, previously worked in Guam, where he often made critical diagnoses in isolation due to a lack of pathologists on the island.
The ARM is envisioned as a second line of defense for pathologists, not a replacement. It is particularly valuable in smaller, remote labs and as a resource for pathology residents in training.
Olson had conceived the idea of a tool like the ARM long before his time in Guam. In 2016, he reached out to Google with his concept, and months later, he witnessed an early AI-powered microscope successfully identifying cancer in slides he provided during a visit to Google’s offices in California.
With the hardware provided by Jenoptik and Google responsible for the software, the DIU has conducted initial research to assess the ARM’s effectiveness. While promising, rigorous testing is required before it can be used for patient care, as changes in cancer diagnosis methods are significant and come with inherent risks.
Ongoing research and feedback from organizations like Mitre and Veterans Affairs will inform the ARM’s future development. The DIU is actively exploring how to scale the technology and collaborate with regulators. Agreements with Google and Jenoptik aim to make the ARM available to government users through the General Services Administration website shortly.
Ultimately, the ARM promises to benefit not only pathologists but also the general public. Its accuracy, speed, and cost-effectiveness are poised to enhance healthcare and diagnostic capabilities. Embracing these technological advancements in medicine is essential for improving patient care.
