LiteLLM Supply Chain Attack: Poisoned Trivy Scanner Explained
On March 24, 2026, a threat actor known as TeamPCP published two malicious versions of LiteLLM to PyPI. LiteLLM is the Python library and proxy that thousands of teams use to route requests to OpenAI, Anthropic, Google, Amazon and dozens of other model providers, and it was reportedly being downloaded about 3.4 million times a day. The attackers never found a bug in LiteLLM. Instead, they compromised Trivy, a popular vulnerability scanner that ran inside LiteLLM’s build pipeline, and used it to steal the credentials needed to publish packages as the real maintainer. A security tool became the way in.
The incident was ranked among the biggest security stories of the year because of what it combined: a package that sits next to the most valuable keys in an AI stack, a trusted delivery channel, and an attack that could run without the victim ever importing the library. This post explains what was disclosed, how the chain of compromise worked, what the malicious code did, who was exposed, how to check your own systems, and what to change so the next incident of this kind costs you less.
Contents
- The incident at a glance
- What LiteLLM is and why it was a target
- Timeline of events
- How the attack worked: from Trivy to PyPI
- What the malicious code did
- Why the .pth trick matters
- Why this attack is different
- Affected and safe versions
- Who was exposed in practice
- How to check if you were affected
- Incident response checklist
- Long-term hardening for AI and Python supply chains
- Lessons for maintainers and security teams
- Frequently asked questions
- Sources and further reading
The incident at a glance
| Item | Details |
|---|---|
| Affected package | litellm on PyPI |
| Malicious versions | 1.82.7 and 1.82.8 |
| Date published | March 24, 2026 |
| Threat actor | TeamPCP |
| Root cause | A compromised Trivy scanner in the CI/CD pipeline leaked the PyPI publishing token |
| Related CVE | CVE-2026-33634 (the Trivy compromise), added to CISA’s Known Exploited Vulnerabilities catalog on March 26, 2026 |
| Impact | Theft of SSH keys, cloud credentials, API keys, Kubernetes secrets, shell history and crypto wallet files |
| Exfiltration domain | models.litellm.cloud (not the official litellm.ai), registered on March 23, 2026 |
| Fix | Remove the bad versions, rotate every reachable secret, move to a rebuilt release (v1.83.0 or later) |
What LiteLLM is and why it was a target
LiteLLM gives developers one consistent interface to many large language model providers. It can be used as a Python library inside an application, or run as a proxy server that sits between your applications and the model vendors. Teams like it because they can switch providers, track spending and apply rate limits without rewriting code.
That convenience creates concentration. A machine running a LiteLLM proxy typically holds API keys for several providers at once, often alongside cloud credentials, database passwords and deployment tokens. An attacker who lands on that machine does not get one secret. They get a map of an organization’s AI infrastructure and the means to spend its money.
The package also spreads quietly. Many agent frameworks, guardrail libraries and LLM toolkits list LiteLLM as a dependency, so a large number of people had it installed without ever choosing it. That is exactly the kind of low-visibility, high-privilege dependency that supply chain attackers look for.
Timeline of events
- Late February 2026: Attackers exploit a misconfiguration in Trivy’s GitHub Actions environment.
- March 1, 2026: The Trivy team discovers the intrusion and tries to contain it by rotating credentials. Later analysis indicates the rotation was incomplete, leaving some access in place.
- March 19, 2026 (about 17:43 UTC): TeamPCP uses retained credentials to force-push malicious commits to 76 of the 77 version tags of
trivy-actionand all 7 tags ofsetup-trivy, and publishes a malicious Trivy v0.69.4 release. - March 19 to 23, 2026: Pipelines that run the poisoned Trivy components may leak their CI/CD secrets. LiteLLM’s build pipeline is among them.
- March 23, 2026: The lookalike domain
litellm.cloudis registered, and an exfiltration endpoint is set up on it. - March 24, 2026: LiteLLM versions 1.82.7 and 1.82.8 are uploaded to PyPI using the stolen token. Reports on how long they stayed live vary, from roughly 40 minutes to about three hours, before PyPI quarantined the project and the versions were removed.
- March 24 to 25, 2026: Maintainers and the community document the compromise in a public GitHub issue. LiteLLM pauses new releases and engages external incident responders.
- March 26, 2026: The Trivy flaw (CVE-2026-33634) is added to CISA’s Known Exploited Vulnerabilities catalog.
- March 27, 2026: Singapore’s Cyber Security Agency publishes an advisory about the wider TeamPCP campaign, which by then included other open-source components beyond Trivy and LiteLLM.
- Afterwards: LiteLLM audits its supply chain and ships v1.83.0 from a rebuilt pipeline with isolated environments.
The practical lesson of this timeline is speed. The malicious versions existed for a short time, but automated installers, dependency bots and unpinned CI jobs do not wait for a human to read the news.
How the attack worked: from Trivy to PyPI
This was a chain of trust, and the attackers walked it one link at a time.
- Compromise the scanner. TeamPCP poisoned Trivy and its GitHub Actions. Thousands of pipelines run these automatically, usually with broad access to secrets, because a scanner needs to see the code and the build.
- Harvest CI/CD secrets. Any job that ran the bad Trivy build could expose the environment variables and tokens available to that job.
- Take the publishing token. LiteLLM’s pipeline installed Trivy without pinning a version, so it pulled in the poisoned build. The PyPI publishing token was among the secrets taken.
- Publish as the real maintainer. With a valid token, the attackers uploaded packages that PyPI treated as genuine. No typosquatting was needed, because the package name was exactly right and users had every reason to trust it.
Why an incomplete rotation made it worse
The first Trivy intrusion was found on March 1. The team responded by rotating credentials, which is the right instinct, but some access survived. Two and a half weeks later the attackers came back with credentials they had kept. Rotation only works if it is complete, verified and covers every token that could be reused, including those held by automation accounts.
What the malicious code did
The two versions used different triggers, which widened the number of ways a victim could be hit:
- 1.82.7 hid a payload inside
litellm/proxy/proxy_server.py. It ran when the proxy module was imported. - 1.82.8 added a file named
litellm_init.pth, about 34 KB in size, plus the payload in the proxy module. Because of how Python handles.pthfiles, this version could execute whenever Python started in that environment.
According to published analyses, the payload searched for and sent out:
- SSH keys,
.envfiles and environment variables, including AI provider API keys - AWS, GCP and Azure credentials
- Kubernetes configuration files and secrets
- Git credentials, shell history files and database passwords
- SSL private keys and CI/CD secrets
- Cryptocurrency wallet files
Collected data was sent with an HTTP POST to models.litellm.cloud. The data was reportedly encrypted before sending, which means defenders cannot recover what was taken and must assume that anything the process could read is exposed. Some write-ups also describe attempts at lateral movement inside Kubernetes clusters and self-replication. Treat those as reported behavior and check your own clusters accordingly.
Why the .pth trick matters
Python scans its site-packages folder at startup for files ending in .pth. These files are meant to add directories to the import path, but lines that begin with import are executed as code. Attackers have abused this behavior before because it requires nothing from the victim except starting Python.
In practice this means that on a machine with 1.82.8 installed, simply running python, pip, a test runner or a notebook kernel in that environment could trigger the payload. You did not need to write import litellm. Dependency scanners that only look at imported code, and developers who reason that they never use the proxy feature, would both have missed it.
Why this attack is different
- It targeted the keys to everything. An AI gateway concentrates credentials for several providers in one place.
- It abused a security tool. The entry point was a scanner that teams installed to be safer, running with high privilege inside the build.
- It used a legitimate identity. Packages published with the real maintainer’s token look authentic to PyPI and to anyone checking the publisher name.
- It ran without being imported. The startup-file technique removed the usual assumption that unused code is harmless.
- It reached people who never chose it. Transitive dependencies carried the risk into projects where LiteLLM was invisible.
- It was part of a campaign. The same group was linked to compromises of other open-source components, so the cleanup extends beyond one package.
Affected and safe versions
| Item | Status |
|---|---|
| litellm 1.82.6 and earlier | Not part of this incident. Still audit anything built during the affected window. |
| litellm 1.82.7 | Malicious. Treat any environment that imported it as compromised. |
| litellm 1.82.8 | Malicious. Treat any environment that started Python with it installed as compromised. |
| litellm 1.83.0 and later | Released from the rebuilt pipeline. Verify against the checksums LiteLLM published. |
| Official LiteLLM Proxy Docker image | LiteLLM states that deployments using the official image were not impacted, because that path pins dependencies and does not rely on the compromised PyPI packages. |
Trivy v0.69.4; trivy-action before v0.35.0; setup-trivy before v0.2.6; Trivy images v0.69.5 and v0.69.6 | Listed in Singapore CSA’s advisory as affected components of the wider campaign. |
Version numbers can mislead. A package removed from PyPI can still live in a pip cache, a mirror, a Docker layer or a lock file. Check what actually ran in your environment, not just what the registry shows today.
Who was exposed in practice
Developers and data scientists
Anyone who ran pip install litellm or an upgrade command while the bad versions were live may have pulled one. Local machines, notebooks and virtual environments often hold .env files with API keys, SSH keys and cloud credentials, which makes them rich targets.
CI/CD pipelines
Unpinned installs in build systems were the highest-risk case. A runner that installed a bad version had access to deployment tokens, cloud roles and registry credentials, and those usually carry more power than a developer laptop.
Frameworks that depend on LiteLLM
Agent frameworks, guardrail libraries and LLM tooling may include LiteLLM as a dependency. Searching only for direct installs will miss them, so inspect the full dependency tree.
Containers and Kubernetes
Images built during the window with an unpinned install may contain a bad version, and the same goes for images derived from them. Rebuilding is not enough unless you also rotate any secrets those workloads could reach, since stolen data is already gone. In clusters, review privileged pods and service accounts for anything unfamiliar.
Organizations that ran Trivy in CI
Even if you never used LiteLLM, any pipeline that ran Trivy v0.69.4 or a trivy-action tag during the affected period should be treated as potentially exposed, and its logs reviewed.
How to check if you were affected
Start with discovery, then confirm with logs. These commands are defensive checks and change nothing on your system.
pip show litellm # installed version in the current environment
pip freeze | grep -i litellm # pinned version in requirements
pipdeptree -r -p litellm # which packages pull litellm in
find / -name "litellm_init.pth" 2>/dev/null # the 1.82.8 trigger file
pip cache list litellm # cached copies of the package
grep -ri "litellm==1.82" requirements*.txt poetry.lock Pipfile.lock 2>/dev/null
For containers and CI, inspect build logs and image layers from March 24 for litellm==1.82.7 or 1.82.8, and compare image build dates with the incident window.
Signs of compromise
- DNS, proxy or firewall logs showing connections to
models.litellm.cloudor the associated attacker infrastructure - A
litellm_init.pthfile in anysite-packagesdirectory - Unexpected privileged pods, new service accounts or unfamiliar workloads in Kubernetes
- Cloud API activity from unfamiliar locations using your access keys
- Sudden spikes in AI provider usage or billing, which can indicate stolen API keys being used
- For Trivy users: workflow runs around March 19 to 20 that executed the affected actions, and a repository named
tpcp-docsappearing in your GitHub organization
No single indicator is definitive. Combine several signals before deciding that an environment is clean, and remember that the absence of evidence is weak when the stolen data left through a single encrypted request.
Incident response checklist
- Isolate. Stop and quarantine any environment that ran 1.82.7 or 1.82.8. Preserve disk images or logs if you may need a forensic review.
- Remove. Uninstall the package, delete
litellm_init.pth, clear pip caches and rebuild from a clean base rather than cleaning in place. - Rotate everything reachable. This includes AI provider API keys, cloud credentials, SSH keys, Git tokens, database passwords, Kubernetes secrets, CI/CD secrets, signing keys and any PyPI or registry tokens. Revoke the old ones, do not just add new ones.
- Review logs. Look at cloud audit trails, Kubernetes audit logs, Git hosting logs and AI provider usage from March 24 onward for activity on affected credentials.
- Check your Trivy usage. If pipelines ran an affected Trivy version or action tag around March 19 to 20, treat those runs as potentially compromised and rotate the secrets they held.
- Pin and verify. Pin LiteLLM to a known-good release, ideally with hash checking, and verify checksums for any build you adopt.
- Rebuild and redeploy. Replace images and workloads from clean sources, and confirm that old layers are removed from registries.
- Rescan and document. Run dependency and secret scans again, record what you found, and note which secrets were rotated and when.
Long-term hardening for AI and Python supply chains
- Pin everything, including security tools. Pin Trivy, GitHub Actions and base images to exact versions or full commit hashes, never to floating tags that can be moved.
- Use hash-checked installs.
pip install --require-hashesprevents a surprise upload from silently replacing what you tested. - Add a cooldown for new releases. A delay of a day or two before adopting brand-new versions would have avoided most installs during a window this short.
- Shrink CI/CD secrets. Use short-lived, narrowly scoped credentials and PyPI Trusted Publishing instead of long-lived tokens stored in the pipeline.
- Rotate completely. After any incident, rotate every secret, confirm the old ones are dead, and check automation and service accounts that are easy to forget.
- Separate AI keys. Do not keep every provider key in one
.envfile on one machine. Use per-service keys with spend limits and alerts. - Isolate build steps. Run security scans in an environment that does not hold publishing credentials.
- Monitor egress. Alert on outbound connections from build runners and servers to newly registered or lookalike domains.
- Maintain a software bill of materials. You cannot respond quickly to a package you did not know you had.
- Audit transitive dependencies. Review lock files regularly and watch for high-privilege packages arriving through other libraries.
Lessons for maintainers and security teams
For open-source maintainers
Treat your publishing token as the most valuable secret you own. Prefer short-lived, identity-based publishing, require a separate approval step for releases, and keep release jobs away from third-party actions that run with secrets. Publish checksums so users can verify what they download.
For security teams
Security tooling is part of your attack surface. Inventory the tools that run in your pipelines, decide which of them genuinely need access to secrets, and review how each is versioned. Build incident playbooks that cover “a trusted dependency was poisoned,” because the right first moves are different from a normal vulnerability patch: find, isolate, rotate, then upgrade.
For AI and platform teams
AI infrastructure is now a first-class target. Gateways, agent frameworks and model-routing libraries run with powerful credentials and often move faster than traditional review processes. Apply the same pinning, scanning and secret-management discipline you use for production services.
Frequently asked questions
I only use the official LiteLLM Docker proxy image. Am I affected?
LiteLLM states that the official proxy image was not impacted, since it pins its dependencies. Verify the image tag and build date you used, and confirm that nobody on your team separately installed the package from PyPI in the same environment.
The bad versions were live for a short time. Do I still need to worry?
Yes, if anything installed them. Automated pipelines and dependency updaters can pull a new release within minutes, so a short window can still reach many systems.
I never installed LiteLLM. Could I still have it?
Possibly. Other AI libraries pull it in as a dependency. Run pipdeptree or inspect your lock files to find out.
Is upgrading enough?
No. Upgrading removes the malicious code, but any secrets already collected are in the attacker’s hands. Rotation is the step that closes the risk.
Does this mean LiteLLM is unsafe now?
The maintainers paused releases, audited their pipeline and rebuilt it with isolated environments. Whether that meets your risk tolerance is your decision, but the underlying lesson applies to every open-source dependency, not just this one.
Was my AI provider account hacked?
Not directly. The theft happened on machines that ran the bad package. If API keys were on those machines, they may have been taken, so rotate them and review usage on your provider dashboards.
How do I know if a tool in my pipeline is trustworthy?
You often cannot know for certain. Reduce what it can do instead: pin its version, run it without access to publishing secrets, and watch its network behavior.
Conclusion
The LiteLLM incident shows how modern attacks work: skip the target’s defenses and compromise something it already trusts. One scanner, one unpinned install and one stolen token were enough to put credential-stealing code inside a package that sits beside the most valuable keys in an AI stack. The response is clear even if it is tedious: find every place the bad versions could have run, isolate and rebuild, rotate every reachable secret, and then fix the pipeline habits that made the chain possible. Pin your tools, verify your downloads, shorten the life of your credentials, and treat your build system with the same care as the production systems it deploys.
Disclaimer: This article is provided for educational and defensive purposes. Details were compiled from public advisories and vendor research, and some figures, such as how long the malicious versions were available, differ between sources. Always consult the official LiteLLM and PyPI advisories and your own security team before making changes to production systems.
Sources and further reading
- LiteLLM GitHub issue #24518: PyPI package compromised, timeline and status
- LiteLLM security update, March 2026
- CSA Singapore Advisory AD-2026-001: Ongoing TeamPCP supply-chain campaign
- PYSEC-2026-2: PyPI advisory database entry for litellm
- Aviatrix: TeamPCP poisoned a security tool
- Wiz Research: Trivy compromised, TeamPCP supply chain attack
Comments
Post a Comment