Introduction
Can artificial intelligence reliably manage the complexity and nuance of cancer care? A new systematic evaluation from Shanghai places that critical question under rigorous clinical scrutiny, comparing two prominent Chinese large language models (LLMs) in the context of ovarian cancer diagnosis and treatment.
Published in Diagnostics, the study titled “Decoding AI Competence: Benchmarking Large Language Models in Ovarian Cancer Diagnosis and Treatment” presents one of the most structured head-to-head assessments of generative AI in gynecologic oncology. The findings reveal sharp performance differences between DeepSeek-R1 and Doubao-1.5-Pro, while also exposing the broader limitations of AI in complex medical decision-making.
Study Design and Methodology
Researchers at the Shanghai First Maternity and Infant Hospital designed a methodologically controlled benchmark to evaluate whether these AI models could align with established international clinical guidelines. Ovarian cancer remains one of the most aggressive gynecologic malignancies worldwide, characterized by high mortality rates and intricate treatment pathways requiring precision across surgery, chemotherapy, genetic testing, maintenance therapy, and long-term follow-up.
The evaluation framework consisted of 20 standardized questions grounded in NCCN, FIGO, and ESMO guidelines, divided equally across four clinical domains: Risk Factors and Prevention, Surgical Management, Medical Treatment, and Surveillance. Each model independently answered all 20 questions in isolated sessions to minimize interaction bias.
Five senior gynecologic oncology chief physicians then rated each response on a 10-point scale, assessing both accuracy and completeness. Scores above seven were classified as clinically “Excellent.” A total of 200 expert ratings were collected — 100 per model.
DeepSeek-R1 Outperforms Doubao-1.5-Pro
The results show a decisive advantage for DeepSeek-R1. Of its 100 individual expert ratings, 98 were classified as Excellent, and all 20 responses achieved average scores above the seven-point threshold. In contrast, Doubao-1.5-Pro received only 41 Excellent ratings out of 100, with just nine of its 20 answers surpassing the minimum excellence benchmark.
A radar-based performance comparison across all 20 questions confirmed that DeepSeek-R1 outperformed Doubao-1.5-Pro in 19 out of 20 cases. Only one surgical protocol question saw Doubao achieve a marginally higher average score. Statistical testing further reinforced these findings: DeepSeek-R1 showed no significant variation across domains, reflecting consistent knowledge depth, while Doubao-1.5-Pro demonstrated statistically significant differences between domains, indicating uneven clinical understanding.
Domain-by-Domain Performance Breakdown
DeepSeek-R1 achieved near-universal excellence in Medical Treatment and Surveillance — two categories demanding detailed knowledge of chemotherapy protocols, PARP inhibitors, immunotherapy considerations, recurrence definitions, and adverse effect management.
Doubao-1.5-Pro performed comparatively better in Risk Factors and Prevention, where questions centered on BRCA mutation testing, family history assessment, and prevention strategies. However, its performance declined sharply in Medical Treatment, with only 12% of ratings reaching the Excellent threshold — a clear deficiency in complex therapeutic decision-making. The largest performance gap between both models was observed in the Medical Treatment domain.
Structural Weaknesses and Error Analysis
Despite strong overall performance, even DeepSeek-R1 showed specific inaccuracies. In certain surgical eligibility scenarios, it simplified clinical indications without differentiating based on histological subtype. In secondary cytoreductive surgery discussions, it omitted a key eligibility condition, and it applied hyperthermic intraperitoneal chemotherapy indications more restrictively than current guidelines specify. These errors were categorized as minor inaccuracies rather than dangerous misrecommendations, but they highlight the risk of static training data producing outdated interpretations.
Doubao-1.5-Pro showed broader weaknesses. Many responses remained at the level of general medical explanation rather than professional clinical guidance. In high-stakes areas such as maintenance therapy eligibility, platinum resistance definitions, and immunotherapy indications, its answers lacked essential decision-making criteria. While the model did not generate explicitly harmful recommendations, omission of critical clinical details in surgical and pharmacologic contexts was deemed a serious limitation.
Stylistically, DeepSeek-R1 produced longer, structured, evidence-based responses — enhancing completeness but sometimes introducing overly technical language for non-specialist users. Doubao’s responses were shorter and simpler but frequently lacked necessary clinical nuance.
Implications for Clinical Practice
DeepSeek-R1 demonstrates meaningful potential as a supplementary educational tool and assistive clinical support system. Its strengths in risk assessment, surgical principles, treatment planning, and follow-up management suggest that high-performing LLMs can enhance information synthesis and guideline referencing for clinicians.
However, the study firmly rejects independent clinical deployment of LLMs. Human clinicians must retain ultimate responsibility for all diagnostic and therapeutic decisions. Even minor inaccuracies in staging interpretation or treatment eligibility can carry significant consequences in oncology, where precision is non-negotiable.
The researchers call for continuous model updating, integration of guideline-grounded retrieval systems, and multidimensional safety assessments. Models must address hallucination risks, outdated references, and excessive verbosity, while also being optimized for clarity and human-centered communication.
Future Research Directions
The authors outline several priorities for future investigation, including repeated-response testing to assess output stability, variation in question phrasing to evaluate model adaptability, and inclusion of leading international models such as GPT-4o and Claude for broader benchmarking. Comparative analysis with practicing oncologists would further illuminate real-world performance gaps.
The study acknowledges its own limitations: only 20 questions were evaluated, expert raters came from a single institution, and the seven-point excellence threshold, while reflecting expert consensus, remains inherently subjective. The analysis also captured a fixed point in time and did not assess integration into live clinical workflows — an important next step as AI tools move closer to real-world deployment.
