Data Wars: Securing LLMs in the Open-Source Era

A Wave of Breaches Exposes Vulnerabilities

The rapid adoption of open-source Large Language Models (LLMs) like DeepSeek and Ollama has presented a paradox. While businesses are harnessing these powerful tools to enhance efficiency, the very openness that drives their growth is simultaneously creating a surge in data security risks. A report compiled by NSFOCUS Xingyun Lab highlights this stark reality: in the first two months of 2025 alone, five significant data breaches directly linked to LLMs occurred. These incidents resulted in the exposure of vast amounts of sensitive information, ranging from confidential chat histories and API keys to critical user credentials. These events serve as a wake-up call, underscoring the often-overlooked security vulnerabilities that lie beneath the surface of cutting-edge AI technology. This exploration will delve into these five incidents, analyzing the attack methods, mapping them to the established MITRE ATT&CK framework, and revealing the security blind spots that organizations must urgently address.

Incident 1: DeepSeek’s Misconfigured Database – A Window into Private Conversations

Timeline: January 29, 2025

Scale of Leakage: Millions of lines of log data, including sensitive chat histories and access keys.

Unfolding the Events:

The security research team at Wiz initiated this discovery. They identified an exposed ClickHouse service accessible on the public internet. Further investigation confirmed that this service belonged to the Chinese AI startup, DeepSeek. ClickHouse, designed for efficient handling of large datasets in analytical processing, unfortunately, became a gateway to DeepSeek’s internal data. The researchers accessed approximately a million lines of DeepSeek’s log stream, revealing a wealth of sensitive information, including historical chat logs and crucial access keys.

Wiz promptly alerted DeepSeek to the vulnerability, leading to immediate action and the secure disposal of the exposed ClickHouse service.

Dissecting the Attack:

The core issue stemmed from ClickHouse’s vulnerability to unauthorized access. ClickHouse, an open-source column-oriented database management system, excels at real-time query and analysis of massive datasets, often used for log and user behavior analysis. However, when deployed without proper access controls, its exposed API interface allows anyone to execute SQL-like commands.

The Wiz security team’s approach involved a methodical scan of DeepSeek’s internet-facing subdomains. Initially focusing on standard ports 80 and 443, they found typical web resources like chatbot interfaces and API documentation. To broaden their search, they expanded to less common ports like 8123 and 9000, ultimately uncovering exposed services on multiple subdomains.

The compromised log data, dating back to January 6, 2025, contained a wealth of sensitive information: call logs, text logs for internal DeepSeek API endpoints, detailed chat histories, API keys, backend system details, and operational metadata.

VERIZON Event Classification: Miscellaneous Errors

MITRE ATT&CK Framework Mapping:

• T1590.002 (Collect Victim Network Information - Domain Name Resolution): Attackers likely used the primary domain name to perform subdomain enumeration.
• T1046 (Web Service Discovery): The attackers identified open ports and services associated with the target domain.
• T1106 (Native Interface): The attackers leveraged the ClickHouse API to interact with the database.
• T1567 (Data Exfiltration via Web Service): The attackers used the ClickHouse API to steal data.

Incident 2: DeepSeek’s Supply Chain Attack – A Trojan Horse in the Code

Timeline: February 3, 2025

Scale of Leakage: User credentials and environment variables.

Unfolding the Events:

The attack began on January 19, 2025, when a malicious user, identified as “bvk,” uploaded two malicious Python packages named “deepseek” and “deepseekai” to the popular PyPI (Python Package Index) repository.

The threat intelligence team at Positive Technologies Expert Security Center (PT ESC) detected this suspicious activity on the same day. Their analysis confirmed the malicious nature of the packages, and they promptly notified the PyPI administrators.

PyPI administrators swiftly removed the malicious packages and informed PT ESC. Despite the quick response, statistics revealed that the malware had been downloaded over 200 times across 17 countries through various channels. The malicious packages were subsequently isolated.

Dissecting the Attack:

The malicious packages uploaded by “bvk” focused on two primary objectives: information gathering and stealing environment variables. The stolen data included sensitive information such as database credentials, API keys, and access credentials for S3 object storage. The malicious payload was triggered whenever a user executed DeepSeek or Deepseekai from the command line.

The attacker utilized PipeDream as a command-and-control server to receive the stolen data. The incident highlights several contributing factors:

• Dependency Confusion Attack: The attackers exploited the priority difference between an organization’s private packages and public packages with the same name.
• Package Name Impersonation: The malicious packages mimicked the brand name of DeepSeek, a well-known AI company, to deceive users.
• PyPI Registration Weakness: The PyPI registration process lacked effective verification of developer identity and package name legitimacy.
• Developer Security Awareness: Developers may have mistakenly installed the similarly named malicious packages.

VERIZON Event Classification: Social Engineering

MITRE ATT&CK Framework Mapping:

• T1593.003 (Search Open Websites/Domains - Search Publicly Available Dependency Repository): The attackers searched for information on PyPI.
• T1195.002 (Supply Chain Compromise - Compromise Software Supply Chain): The attackers used malware disguised as Python dependencies and uploaded it to PyPI.
• T1059.006 (Command and Scripting Interpreter - Python): The attackers implanted malicious code in the package, which, upon execution, leaked sensitive data.
• T1041 (Exfiltration Over C2 Channel): The attackers exfiltrated sensitive information via the PipeDream C2 channel.

Incident 3: LLM Hijacking – DeepSeek Targeted for Resource Theft

Timeline: February 7, 2025

Scale of Leakage: Approximately 2 billion model tokens illegally used.

Unfolding the Events:

The Sysdig threat research team initially discovered a novel attack targeting LLMs, dubbed “LLM jacking” or “LLM hijacking,” in May 2024.

By September 2024, Sysdig reported a growing frequency and prevalence of these attacks, with DeepSeek increasingly becoming a target.

On December 26, 2024, DeepSeek released an advanced model, DeepSeek-V3. Shortly after, the Sysdig team found that DeepSeek-V3 had been implemented in an OpenAI reverse proxy (ORP) project hosted on Hugging Face.

On January 20, 2025, DeepSeek released an inference model called DeepSeek-R1. The very next day, an ORP project supporting DeepSeek-R1 appeared, and attackers began exploiting it, populating multiple ORPs with DeepSeek API keys.

Sysdig’s research indicated that the total number of large model tokens illegally used through ORPs had surpassed 2 billion.

Dissecting the Attack:

LLM hijacking involves attackers exploiting stolen cloud credentials to target cloud-hosted LLM services. The attackers leverage an OpenAI (OAI) reverse proxy and stolen credentials to essentially sell access to the victim’s subscribed LLM services. This results in significant cloud service costs for the victim.

The OAI reverse proxy acts as a central management point for access to multiple LLM accounts, masking the underlying credentials and resource pools. Attackers can use expensive LLMs like DeepSeek without paying for them, directing requests through the reverse proxy, consuming resources, and bypassing legitimate service charges. The proxy mechanism hides the attacker’s identity, allowing them to misuse cloud resources undetected.

While the OAI reverse proxy is a necessary component for LLM hijacking, the crucial element is the theft of credentials and keys for various LLM services. Attackers often exploit traditional web service vulnerabilities and configuration errors (like the CVE-2021-3129 vulnerability in the Laravel framework) to steal these credentials. Once obtained, these credentials grant access to cloud-based LLM services like Amazon Bedrock, Google Cloud Vertex AI, and others.

Sysdig’s research revealed that attackers could rapidly inflate victims’ consumption costs to tens of thousands of dollars within hours, and in some cases, up to $100,000 per day. The attackers’ motivation extends beyond data acquisition; they also profit by selling access rights.

VERIZON Event Classification: Basic Web Application Attacks

MITRE ATT&CK Framework Mapping:

• T1593 (Search Open Websites/Domains): Attackers used OSINT (Open-Source Intelligence) methods to gather information on exposed services.
• T1133 (External Remote Services): The attackers identified vulnerabilities in exposed services.
• T1586.003 (Compromise Accounts - Cloud Accounts): Attackers exploited vulnerabilities to steal LLM service or cloud service credentials.
• T1588.002 (Obtain Capabilities - Tool): The attackers deployed an open-source OAI reverse proxy tool.
• T1090.002 (Proxy - External Proxy): Attackers used OAI reverse proxy software to manage access to multiple LLM accounts.
• T1496 (Resource Hijacking): Attackers launched an LLM injection attack to hijack LLM resources.

Incident 4: OmniGPT Data Breach – User Data Sold on the Dark Web

Timeline: February 12, 2025

Scale of Leakage: Personal information of over 30,000 users, including emails, phone numbers, API keys, encryption keys, credentials, and billing information.

Unfolding the Events:

On February 12, 2025, a user named “SyntheticEmotions” posted on BreachForums, claiming to have stolen sensitive data from the OmniGPT platform and offering it for sale. The leaked data reportedly included emails, phone numbers, API keys, encryption keys, credentials, and billing information for over 30,000 OmniGPT users, along with over 34 million lines of their conversations with chatbots. Additionally, links to files uploaded to the platform were compromised, some containing sensitive information like vouchers and billing data.

Dissecting the Attack:

While the precise attack vector remains undisclosed, the type and scope of the leaked data suggest several possibilities: SQL injection, API abuse, or social engineering attacks might have granted the attacker access to the backend database. It’s also possible that the OmniGPT platform had misconfigurations or vulnerabilities that allowed the attacker to bypass authentication and directly access the database containing user information.

The ‘Messages.txt’ file involved in a secondary leak contained API keys, database credentials, and payment card information, potentially enabling further intrusion into other systems or data tampering. Some documents uploaded by platform users contained sensitive business secrets and project data, posing a risk to business operations if misused. This incident serves as a stark reminder of the need for enhanced data security and privacy protection within the AI and big data sectors. Users should exercise extreme caution when utilizing these platforms, and organizations must establish strict data usage policies, implementing measures like encryption, data minimization, and anonymization for sensitive data. Failure to do so can lead to significant legal, reputational, and economic consequences.

VERIZON Event Classification: Miscellaneous Errors

MITRE ATT&CK Framework Mapping:

• T1071.001 (Application Layer Protocol - Web Protocols): Attackers might have accessed leaked user information and sensitive data through OmniGPT’s web interface.
• T1071.002 (Application Layer Protocol - Application Programming Interfaces): Leaked API keys and database credentials could allow attackers to access the system through the platform’s API and perform unauthorized actions.
• T1071.002 (Application Layer Protocol - Service Execution): Attackers might abuse system services or daemons to execute commands or programs.
• T1020.003 (Automated Exfiltration - File Transfer): Leaked file links and user-uploaded sensitive files could be targets for attackers to download, obtaining more sensitive data for subsequent attacks.
• T1083 (File and Directory Discovery): Attackers could use the leaked information to further obtain key business information.

Incident 5: DeepSeek Credentials Leaked in Common Crawl – The Perils of Hard-Coding

Timeline: February 28, 2025

Scale of Leakage: Approximately 11,908 valid DeepSeek API keys, credentials, and authentication tokens.

Unfolding the Events:

The Truffle security team utilized the open-source tool TruffleHog to scan 400 TB of data from December 2024 in Common Crawl, a crawler database encompassing 2.67 billion web pages from 47.5 million hosts. The scan revealed a startling finding: approximately 11,908 valid DeepSeek API keys, credentials, and authentication tokens were hard-coded directly into numerous web pages.

The study also highlighted the leakage of Mailchimp API keys, with around 1,500 keys found hard-coded in JavaScript code. Mailchimp API keys are often exploited for phishing and data theft attacks.

Dissecting the Attack:

Common Crawl, a non-profit web crawler database, regularly captures and publishes data from internet pages. It stores this data in WARC (Web ARChive) files, preserving the original HTML, JavaScript code, and server responses. These datasets are frequently used to train AI models. Truffle’s research exposes a critical issue: training models on corpora containing security vulnerabilities can lead to models inheriting those vulnerabilities. Even if LLMs like DeepSeek employ additional security measures during training and deployment, the widespread presence of hard-coded vulnerabilities in the training data can normalize such ‘unsafe’ practices for the models.

Hard-coding, a common but insecure coding practice, is a pervasive problem. While the root cause is simple, the risks are severe: data breaches, service disruptions, supply chain attacks, and, with the rise of LLMs, a new threat – LLM hijacking. As previously discussed, LLM hijacking involves attackers using stolen credentials to exploit cloud-hosted LLM services, resulting in substantial financial losses for victims.

VERIZON Event Classification: Miscellaneous Errors

MITRE ATT&CK Framework Mapping:

• T1596.005 (Search Open Technical Database - Scan Databases): The attackers gathered information from the public crawler database.
• T1588.002 (Obtain Capabilities - Tool): The attackers deployed a sensitive information discovery tool.
• T1586.003 (Compromise Accounts - Cloud Accounts): Attackers used sensitive information discovery tools to find sensitive credentials in public databases.
• T1090.002 (Proxy - External Proxy): Attackers used OAI reverse proxy software to manage access to multiple LLM accounts.
• T1496 (Resource Hijacking): Attackers launched an LLM injection attack to hijack LLM resources.

Preventing LLM Data Leakage: A Multi-Faceted Approach

The incidents analyzed highlight the urgent need for robust security measures to protect against LLM-related data breaches. Here’s a breakdown of preventative strategies, categorized by the relevant incidents:

Strengthening the Supply Chain:

Applicable to Incident II (malicious dependency package attack) and Incident V (public data breach):

1. Trusted Verification of Dependency Packages:

   • Employ tools like PyPI/Sonatype Nexus Firewall to intercept unsigned or suspiciously sourced dependency packages.
   • Prohibit direct fetching of dependencies from public repositories in development environments. Mandate the use of corporate private repository proxies (e.g., Artifactory).

2. Supply Chain Threat Monitoring:

   • Integrate tools like Dependabot/Snyk to automatically scan for dependency vulnerabilities and block the introduction of high-risk components.
   • Verify the code signature of open-source packages to ensure the hash value matches the official one.

3. Data Source Cleaning:

   • During training data collection, filter sensitive information from public datasets (like Common Crawl) using regular expressions and AI-based redaction tools for double verification. Employ techniques like differential privacy to add noise to the data, making it harder to identify individuals while preserving the overall utility of the dataset. Implement data provenance tracking to maintain a clear audit trail of data sources and transformations, facilitating identification and remediation of potential vulnerabilities.

Implementing Least Privilege and Access Control:

Applicable to Incident I (database configuration error) and Incident IV (third-party tool data breach):

• Enable bidirectional TLS authentication by default for databases (like ClickHouse) and prevent exposure of management ports on public networks. Implement network segmentation to isolate sensitive databases and LLM services from other parts of the network, limiting the potential impact of a breach.
• Utilize solutions like Vault/Boundary to dynamically distribute temporary credentials, avoiding long-term static key retention. Implement a robust credential rotation policy, regularly changing passwords and API keys to minimize the window of opportunity for attackers.
• Adhere to the principle of least privilege, restricting user access to only necessary resources through RBAC (Role-Based Access Control). Regularly review and update access permissions to ensure they remain aligned with current roles and responsibilities.
• Implement IP whitelisting and rate limiting for API calls to third-party tools (like OmniGPT). Monitor API usage patterns for anomalies, such as unusually high request volumes or access from unexpected locations. Implement multi-factor authentication (MFA) for all user accounts, especially those with administrative privileges.

Ensuring Full-Lifecycle Protection of Sensitive Data:

Applicable to Incident III (LLM hijacking):

• Data Redaction and Encryption: Enforce field-level encryption (e.g., AES-GCM) for user input and output data. Mask sensitive fields in logs. Implement format-preserving encryption (FPE) where applicable, allowing data to be processed without decryption while maintaining its original format.
• Enable real-time redaction for the interactive content of LLMs (e.g., replacing credit card numbers and phone numbers with placeholders). Utilize natural language processing (NLP) techniques to identify and redact sensitive information based on context and semantic understanding.
• Data Loss Prevention (DLP): Implement DLP solutions to monitor and prevent the exfiltration of sensitive data. Configure DLP rules to detect and block the transmission of specific data patterns, such as credit card numbers, social security numbers, and API keys.
• Regular Security Audits and Penetration Testing: Conduct regular security audits and penetration testing to identify and address vulnerabilities in LLM systems and infrastructure. Engage external security experts to provide an independent assessment of security posture.
• Incident Response Plan: Develop and maintain a comprehensive incident response plan specifically tailored to LLM-related security incidents. Regularly test the incident response plan through tabletop exercises and simulations.
• Employee Training and Awareness: Provide regular security training to employees on best practices for handling sensitive data and identifying potential threats. Conduct phishing simulations to educate employees about social engineering attacks.
• Monitoring and Anomaly Detection: Implement robust monitoring and anomaly detection systems to identify suspicious activity and potential breaches. Utilize machine learning techniques to detect unusual patterns in LLM usage and data access.
• Secure Development Practices: Integrate security into the software development lifecycle (SDLC) for LLM applications. Conduct code reviews and static analysis to identify and remediate vulnerabilities before deployment.
• Compliance with Regulations: Ensure compliance with relevant data privacy regulations, such as GDPR, CCPA, and HIPAA. Implement appropriate data governance policies and procedures.
• Collaboration and Information Sharing: Collaborate with other organizations and security researchers to share threat intelligence and best practices. Participate in industry forums and working groups focused on LLM security.

These preventative measures, combined with continuous security monitoring and incident response planning, are essential for mitigating the risks associated with the growing use of LLMs. The ‘invisible battlefield’ of LLM security demands constant vigilance and a proactive approach to safeguard sensitive data in this rapidly evolving technological landscape. The future of LLM adoption hinges on building trust and security into the very foundation of these powerful technologies.