A 40-minute window in March was all it took. In that span, a poisoned version of LiteLLM, an open source AI proxy gateway, sat on PyPI, the official Python package registry, and any developer whose build touched it had their environment scraped for secrets. The fallout, disclosed this week by CloudSEK and Hudson Rock, is a credential dump that touches Microsoft, Amazon, Cisco, Samsung, Salesforce, ServiceNow, Siemens, Epic Games, X Corp, and roughly 2,500 organizations in total (Ars Technica, CloudSEK, Hudson Rock).
I am a Senior Software Engineer II at BS23 in Dhaka, and I have been building production AI systems with Spring Boot and Spring AI for over a year. My stack is Java, not Python, and this article is not about a Python bug. It is about a supply chain pattern that applies to every language, including the Java tools most of us reach for every day. If you run an AI gateway, a Maven mirror, or any proxy between your code and a model provider, this story is about your environment.
The attack chain: a scanner was the door
The attack did not start with LiteLLM. It started with Trivy, a widely used open source vulnerability scanner. TeamPCP, the group that took credit, compromised the GitHub Actions pipeline that builds Trivy. Because LiteLLM's developers used Trivy in their own CI, the poisoned scanner had legitimate read access to their build environment. That access let the attackers exfiltrate LiteLLM's PyPI publishing tokens (CloudSEK, Snyk analysis).
With those tokens, the group published malicious versions of LiteLLM, 1.82.7 and 1.82.8, to PyPI. The payload used a .pth file, a Python startup hook. Here is the detail that matters: a .pth file executes when the Python interpreter starts, whether or not the LiteLLM package is ever imported. The malware ran on any machine that merely installed the package, which sidesteps the --ignore-scripts flag developers use to block install-time code.
The payload then did three things (Hudson Rock):
- Harvested environment variables. Cloud keys, API tokens, and every secret the runner had in scope.
-
Read local config files. Including
.kube/configand.aws/credentials, the files that grant access to Kubernetes clusters and cloud accounts. - Tried to move laterally. Attempting to reach other systems on the network and installing a persistent backdoor.
All of this happened for any build that installed the package during the 40-minute window, and the persistent backdoor kept exfiltrating data afterward. Hudson Rock obtained the raw dump: 153GB of pipeline environments from roughly 118,000 attributed CI/CD runs. CloudSEK counted about 434,000 CI/CD pipelines with exposed credentials. Alon Gal, Hudson Rock's CTO, put the scale in one sentence: "A window of roughly 40 minutes in which the LiteLLM dependency was hacked led to over 430,000 instances in which millions of secrets were harvested" (Ars Technica).
One detail makes the chain feel inevitable in hindsight: the Trivy token was rotated but never fully revoked. CloudSEK's analysis found Trivy's maintainers rotated the leaked automation token, then failed to invalidate it, which left attackers a roughly 20-day window to force-push malicious code over the scanner's published version tags (CloudSEK). Rotating a credential without verifying the old one is dead is the same mistake Beaumont caught the impacted organization making this week. It is a common way supply chain incidents stay alive.
The attribution trap: your secrets may be in there without your name
Here is the part that should make every engineering team run the audit even if their name is not on any list. Much of the dump cannot be attributed to a specific organization. Hudson Rock explains why with a concrete example: a pipeline run whose committer email belonged to @siriusxm.com turned out, on closer inspection of the environment dump, to be the infrastructure of AdsWizz, a SiriusXM subsidiary, identified by self-hosted endpoints like gitlab.adswizz.com and registry.adswizz.com (Hudson Rock).
Many CI/CD pipelines are configured generically. The dumped variables contain active database passwords, third-party API keys, and cloud credentials with no identifiable company email, custom domain, or internal server name attached. That means countless organizations currently have active secrets sitting in this database and have no idea they are there, because their name appears nowhere in it.
Why this matters for Java teams
Java developers might read "PyPI" and feel safe. Let me translate the pattern:
- Your AI gateway is a high-value target. LiteLLM is an AI proxy. It sits between your application and model providers, which means its environment carries API keys for OpenAI, Anthropic, and everything else. The Java equivalent is any service that holds model-provider credentials, whether you wrote it with Spring AI or wrapped a vendor SDK.
- Build-time trust is the attack surface. Trivy is a security scanner. Teams install it to feel safer, and it became the door. The Java world has the same dynamic: a Maven plugin, a Gradle wrapper, or a Docker base image can be the trusted component that is actually the attacker's entry.
- The chain multiplies the blast radius. One compromised scanner, one leaked publishing token, and 2,500 organizations had credentials exposed. This is the "one upstream, many downstream" shape that supply chain incidents keep taking.
-
Your secrets are where the attacker wants them. In a Spring AI application, the model provider keys usually live in
application.ymlor an environment variable, and they are loaded by any component that touches the AI client. An agent gateway in front of that, whether LiteLLM or a homegrown proxy, concentrates those keys in one process. That process is now a prime target, which is exactly the position LiteLLM occupied.
The lesson that keeps getting relearned
Kevin Beaumont, the independent security researcher who confirmed the data was real, said it plainly: this is a massive supply chain breach caused by poor AI security, not by AI being the threat. His words: "teens can run circles around orgs obsessed with rushing out AI and poor DevOps security" (Ars Technica).
The same Ars piece carries an update that should worry every engineering lead: an impacted organization said it had rotated all its credentials and called the breach a non-event. Beaumont checked their responsible disclosure policy, tried the old credentials anyway, and almost every one still worked. Rotation that is not verified is theater.
The 10-minute audit
You can check your exposure right now, and most of it does not require special tooling:
-
Check your dependency lockfiles for LiteLLM. If any lockfile pinned
litellm==1.82.7orlitellm==1.82.8during the exposure window, treat that environment as compromised. Hudson Rock maintains a free domain lookup tool for affected organizations (Hudson Rock's LiteLLM lookup). - Assume every secret in that environment is compromised. Cloud IAM keys, Kubernetes service account tokens, GitHub and GitLab personal access tokens, database passwords. Rotate them, and verify the rotation actually killed the old values instead of just issuing new ones.
- Review your AI infrastructure for the same shape. Do you run an AI proxy, a model gateway, or a CI job that holds provider keys? Treat it as a Tier 1 asset and apply the same checks.
- Audit your logs back to March. Hudson Rock advises reviewing CloudTrail and Kubernetes audit logs for anomalous activity dating back to March 24, 2026. Look for egress to unexpected endpoints and for signs of a persistent backdoor (Hudson Rock).
- Question your trusted tooling. The scanner that is supposed to protect you can be the compromise. Know what your CI pipeline installs, pin versions, and watch for unexpected updates to trusted tools.
- Check your lockfiles for other packages in the campaign. The same group also poisoned KICS, a Kubernetes security tool, and the Telnyx Python SDK. If your supply chain includes AI tooling, security scanners, or telecom SDKs, widen the audit beyond LiteLLM (Ars Technica).
What I changed after reading this
This incident pushed two changes in how I think about the Spring AI work I do. First, model provider keys are now treated like database credentials: least privilege, short-lived where possible, and never duplicated across environments. Second, I now look at every dependency in the build path as a potential entry point, not just the application code. The security scanner is no longer exempt from review, because Trivy taught the industry that the protector can be the door.
The uncomfortable truth is that this will happen again, because the economics favor it. One poisoned package, a 40-minute window, and millions of secrets harvested. The defenses are boring: verify your locks, and rotate with proof.
When did you last verify that a rotated credential actually stopped working, not just replaced itself? I read every response.
I write about Java, Spring Boot, and AI security every week. Subscribe, it's free.
Bookmark this one. The next time a trusted tool updates itself, this checklist is the difference between a routine deploy and an incident.