DeepSeek AI in Chinese Hospitals: Rising Concerns

A team of Chinese researchers has voiced concerns regarding the swift integration of DeepSeek, an artificial intelligence model, into hospital settings. Their analysis highlights potential dangers to clinical safety and data privacy, particularly due to the widespread use of the start-up’s economical open-source models.

By early March, DeepSeek’s large language models (LLMs) were already being employed in at least 300 Chinese hospitals for clinical diagnostics and medical decision support.

Published in the Journal of the American Medical Association (JAMA), the research paper points to DeepSeek’s inclination to generate outputs that seem convincing but are factually inaccurate. Despite the AI’s robust reasoning capabilities, this could create significant clinical risks. Wong Tien Yin, the founding head of Tsinghua Medicine, a research division at Tsinghua University in Beijing, is a member of the research team.

This cautionary note contrasts with the prevailing enthusiasm for DeepSeek in China. The start-up, celebrated for its affordable and high-performing V3 and R1 models, has become a symbol of China’s AI advancements.

Wong and his co-authors emphasized the risk that healthcare professionals might become overly dependent on or accept DeepSeek’s outputs without critical evaluation. This could lead to errors in diagnosis or biased treatment plans. Conversely, clinicians who remain cautious would face the added burden of verifying AI outputs under time constraints.

Security Risks in On-Site Deployment

While hospitals often opt for private, on-site deployments of DeepSeek models to reduce the risks associated with security and privacy, this approach introduces its own set of complications. According to the researchers, it "shifts security responsibilities to individual healthcare facilities," many of which may lack the necessary cybersecurity defenses.

The researchers also noted that the combination of inadequate primary care infrastructure and widespread smartphone use in China creates a "perfect storm" that exacerbates concerns about clinical safety.

The researchers state, "Underserved populations with complex medical needs now have unprecedented access to AI-driven health recommendations, but often lack the clinical oversight needed for safe implementation."

Scrutiny of LLMs in Healthcare Settings

This paper contributes to growing conversations about the utilization of LLMs in clinical and medical settings. Other organizations in China are also beginning to scrutinize LLMs as adoption accelerates. Another paper published last month by researchers at the Chinese University of Hong Kong examined the cybersecurity vulnerabilities of AI agents, and discovered that those powered by commonly used LLMs were vulnerable to various attacks, with DeepSeek-R1 being the most susceptible.

China has expedited the adoption of LLMs in healthcare amid a surge in generative AI technologies. Last month, Ant Group, a Chinese financial technology company, introduced nearly 100 AI medical agents on its Alipay payments app. These agents are supported by medical experts from prominent Chinese hospitals.

Tairex, a start-up incubated at Tsinghua University, initiated internal testing of a virtual hospital platform in November. The platform features 42 AI doctors covering 21 departments, including emergency, respiratory, paediatrics, and cardiology. The company revealed plans to launch the platform to the public later in the year.

Diving Deeper into the Concerns Surrounding AI in Healthcare

The rapid integration of AI, particularly large language models (LLMs) like DeepSeek, into healthcare settings in China has sparked a debate between those who champion its potential benefits and those who urge caution. While AI offers exciting possibilities for improving diagnostics, treatment, and access to care, several factors warrant a more measured approach. The concerns raised by researchers highlight the complexities and potential pitfalls of deploying AI in such a critical domain.

One of the primary concerns is the reliability of AI-generated information. LLMs are trained on vast datasets, but these datasets may contain biases, inaccuracies, or outdated information. As a result, AI models can sometimes generate outputs that appear plausible but are, in fact, incorrect. This raises a significant risk in medical settings, where diagnostic errors or incorrect treatment recommendations can have severe consequences for patients. The nature of LLMs, which are designed to predict the next word in a sequence rather than to be authoritative sources of factual information, further exacerbates this potential for error. For instance, an LLM might generate a seemingly convincing explanation of a rare disease, complete with treatment options, that is entirely fabricated or based on outdated research. This could mislead healthcare professionals and lead to inappropriate patient care.

Furthermore, the "hallucination" problem, where LLMs generate entirely fictional information, is a well-documented phenomenon. While ongoing efforts are being made to mitigate this issue, it remains a significant concern in high-stakes environments like healthcare. The use of retrieval-augmented generation (RAG) techniques, where LLMs are paired with external knowledge bases, can help to reduce the likelihood of hallucinations, but RAG systems are still vulnerable to errors if the underlying knowledge base contains inaccurate or incomplete information.

The Risk of Over-Reliance on AI

Another concern is the potential for healthcare professionals to become overly reliant on AI and to lose their critical thinking skills. If doctors and nurses begin to treat AI outputs as infallible, they may fail to adequately assess patients’ conditions, overlook important details, or question the AI’s recommendations. This can lead to diagnostic errors, inappropriate treatments, and a decline in the quality of care. The very efficiency and convenience offered by AI tools can subtly erode clinical expertise over time. As clinicians become accustomed to relying on AI for initial assessments and treatment plans, there is a risk that they will become less adept at performing these tasks independently. This skill decay can be particularly problematic in situations where AI assistance is unavailable or unreliable, such as during power outages or cybersecurity incidents.

To combat this potential for over-reliance, it is crucial to emphasize the importance of human expertise and critical thinking. Healthcare professionals should be trained to use AI as a tool to enhance their clinical judgment, not to replace it. They should be encouraged to question AI outputs, to seek alternative opinions, and to always prioritize the well-being of their patients. Regular training and simulation exercises can help to maintain clinical skills and to reinforce the importance of human oversight.

Furthermore, the widespread adoption of AI raises ethical and social questions about data privacy, algorithmic bias, and the potential for job displacement. Patients may be concerned about the security and confidentiality of their health data, particularly if it is being used to train AI models. Algorithmic bias can also perpetuate and exacerbate existing health disparities if AI models are trained on data that does not accurately reflect the diversity of the population. The use of biased training data can lead to AI models that systematically underperform for certain demographic groups, resulting in unequal access to quality healthcare.

The issue of data privacy is particularly sensitive in the healthcare context. Patients entrust their healthcare providers with highly personal and confidential information, and they have a right to expect that this information will be protected. The use of AI in healthcare raises new challenges to data privacy, as AI models often require access to large datasets of patient information in order to be trained effectively. It is essential to implement robust security measures and to adhere to strict data protection regulations in order to safeguard patient privacy.

Striking a Balance Between Innovation and Caution

To mitigate these risks, it is crucial to adopt a more cautious and responsible approach to the integration of AI in healthcare. This includes:

• Rigorous Testing and Validation: Before deploying AI models in clinical settings, they should be thoroughly tested and validated on diverse populations to ensure their accuracy, reliability, and fairness. This testing should involve both quantitative and qualitative assessments, and it should be conducted by independent experts who are not affiliated with the developers of the AI models.
• Human Oversight: AI should be used as a tool to augment, not replace, human judgment. Healthcare professionals should always review and verify AI outputs before making clinical decisions. This oversight should not be a mere formality but should involve a genuine critical assessment of the AI’s recommendations in light of the patient’s individual circumstances.
• Transparency and Explainability: AI models should be transparent and explainable, so that healthcare professionals can understand how they arrive at their recommendations. This can help to build trust in AI and to identify potential errors or biases. The "black box" nature of many AI models makes it difficult to understand how they reach their conclusions, which can make it challenging for healthcare professionals to identify and correct errors.
• Data Privacy and Security: Robust safeguards should be put in place to protect the privacy and security of patient data. This includes obtaining informed consent, implementing strong security measures, and adhering to data protection regulations. The use of anonymization techniques can help to protect patient privacy while still allowing AI models to be trained on large datasets.
• Education and Training: Healthcare professionals should receive comprehensive training on how to use AI effectively and responsibly. This includes understanding the limitations of AI, recognizing potential biases, and critically evaluating AI outputs. This training should be ongoing and should be integrated into the curriculum of medical schools and nursing programs.
• Continuous monitoring and evaluation: AI systems must be continuously monitored in a live clinical setting, beyond the pre-deployment testing. Regular audits should be conducted to evaluate the performance of the AI models and to identify any emerging issues or biases. User feedback mechanisms are also essential for identifying unexpected behaviours or areas for improvement.

Addressing Cybersecurity Vulnerabilities

The cybersecurity vulnerabilities of AI agents, as highlighted by the researchers at the Chinese University of Hong Kong, pose a significant threat to the integrity and security of healthcare systems. If AI models are susceptible to attacks, malicious actors could potentially manipulate the AI’s outputs, access sensitive patient data, or disrupt healthcare operations. Potential attack vectors include adversarial attacks, where malicious inputs are designed to fool the AI model into making incorrect predictions, and data poisoning attacks, where malicious data is injected into the training dataset to corrupt the AI model’s learning process.

To address these vulnerabilities, it is essential to implement robust cybersecurity measures, such as:

• Secure Coding Practices: AI models should be developed using secure coding practices to prevent vulnerabilities such as SQL injection, cross-site scripting, and buffer overflows.
• Regular Security Audits: AI systems should undergo regular security audits to identify and address potential vulnerabilities. These audits should be conducted by independent cybersecurity experts who are familiar with the unique security challenges of AI systems.
• Intrusion Detection and Prevention Systems: Intrusion detection and prevention systems should be implemented to monitor AI systems for malicious activity and to prevent unauthorized access. These systems should be specifically designed to detect and respond to attacks against AI systems.
• Data Encryption: Sensitive patient data should be encrypted both in transit and at rest to protect it from unauthorized access.
• Access Controls: Strict access controls should be implemented to limit access to AI systems and data to authorized personnel. Multi-factor authentication should be used to enhance the security of access controls.
• Robust Input Validation: All inputs to AI models should undergo rigorous validation to prevent malicious inputs from being processed. This validation should include checks for data type, format, and range.
• Model Hardening: Techniques like adversarial training can harden the AI model against adversarial attacks. Adversarial training involves exposing the model to adversarial examples during training, which helps it to become more robust to these attacks.

Ethical Considerations

Beyond the technical challenges, the integration of AI in healthcare raises a number of important ethical considerations. These include:

• Algorithmic Bias: AI models can perpetuate and exacerbate existing health disparities if they are trained on data that does not accurately reflect the diversity of the population. It is crucial to ensure that AI models are fair and unbiased.
• Data Privacy: Patients may be concerned about the privacy of their health data, particularly if it is being used to train AI models. It is essential to obtain informed consent and to protect patient data.
• Transparency and Explainability: AI models should be transparent and explainable, so that patients can understand how they arrive at their recommendations. This can help to build trust in AI.
• Accountability: It is important to establish clear lines of accountability for the decisions made by AI systems. Who is responsible if an AI model makes a wrong diagnosis or recommends an inappropriate treatment? The assignment of liability in cases where AI systems cause harm is a complex legal and ethical issue that requires careful consideration.
• Autonomy and Human Dignity: As AI takes on greater responsibility in healthcare, it is key to safeguard the autonomy and dignity of both patients and healthcare providers. Decision-making should augment, but not replace, human judgment, ensuring patients remain at the heart of care

The Path Forward

The integration of AI in healthcare holds enormous potential to improve patient care, reduce costs, and enhance the efficiency of healthcare systems. However, it is crucial to approach this integration with caution and to address the potential risks and challenges. By adopting a responsible and ethical approach, we can harness the power of AI to transform healthcare for the better. A multi-stakeholder approach, involving clinicians, data scientists, ethicists, policymakers, and patients, is essential to guide the development and deployment of AI in healthcare in a responsible and equitable manner. Research into methods for mitigating biases, enhancing transparency, and ensuring data privacy is crucial to realizing the full potential of AI while minimizing its risks. The ongoing evolution of technology demands continuous evaluation and adaptation of best practices, fostering innovation while safeguarding patient safety and upholding ethical principles. Ultimately, the successful integration of AI depends on building trust among all stakeholders. Further investment and resources should be aimed towards ethical AI development and training, ultimately serving to improve and empower the medical industry.