Introduction
In the world of software development, we often take for granted the security and reliability of the tools and platforms we rely on daily. We assume that the packages we download and the registries we use are safe and trustworthy. However, at Lupin & Holmes we've recently discovered a Cache Poisoning Attack on the npm registry, one of the largest package registry for JavaScript, potentially exposing the fragility of our Software Supply Chains and the potential for widespread disruption.
The npm registry is a critical component of the JavaScript ecosystem, serving as a central repository for over 2.1 million packages and relied upon by more than 17 million developers worldwide. It has become an indispensable resource, enabling them to easily share, reuse, and manage dependencies in their projects. With millions of downloads per day, the npm registry is the backbone of countless applications and websites.
In this article, we will discuss the details of the cache poisoning attack on npm and explore its potential impact on the broader software ecosystem. By disclosing publicly this vulnerability, we aim to show the importance of security and availability in our Software Supply Chains.
The Cache Poisoning Attack: Exploiting a Vulnerability
This story is actually really funny. While we were developing a Research Submodule for Depi aiming to test Cache Poisoning on Artifactories, we were surprised to find that the most famous registry in the world was vulnerable: npmjs.com.
As part of the development of Depi, we often test several submodules in Research Mode. Essentially, we explore theoretical attack vectors that could disrupt the Software Supply Chain of our clients and develop a module to test and validate at a larger scale. Once the results are triaged, we focus on all the agnostic and odd results until we have an attack vector that is worth implementing into Depi. Anything that could impact the integrity, confidentiality but also availability of our clients is worth testing.
Our theory was simple: What if the Artifactories used by our clients could be vulnerable to a Cache Poisoning Denial of Service (CPDoS) condition ?
CPDoS, is a web attack technique described in this research by PortSwigger. It involves exploiting vulnerabilities in web caching systems to deliver malicious content to users. The attack works by manipulating how web caches store and serve content, effectively poisoning the cache with harmful data. When users request the affected resources, the poisoned content is served instead of the legitimate content, potentially leading to a denial of service or other malicious outcomes. This technique leverages subtle flaws in the caching mechanisms and how they interact with various web protocols and headers.
With that in mind, we thought it would be interesting if an attacker can poison a page to redirect the content of a .tar.gz file to a malicious one or deny the package entirely. We developed a module to perform this action, implemented into Depi, and we had a match in the first run:
registry.npmjs.org is vulnerable.
Obviously we thought this was a false positive and that the vulnerability didn't actually trigger on the first run, however we later managed to confirm that we indeed had a full CPDoS on the npmjs registry !
The attack relied on a series of different headers, which were found to trigger an error in the backend systems. By sending a specially crafted request containing those headers, an attacker could manipulate the registry's caching system into storing a "Not Found" response for a targeted package.
Note: We are not providing the exact series of headers that will trigger this issue, since it's still exploitable at the time of writing the article
Here's an example of the malicious request where we've set a cache buster parameter to avoid DoSing real users:
GET /safe-regex/-/safe-regex-1.1.0.tgz?lupin_E7A812DE-E09A-4906-A9E3-530E54AAEB41=cpdos_test HTTP/1.1
Host: registry.npmjs.org
User-Agent: lupin_test_cpdos
lupin: E7A812DE-E09A-4906-A9E3-530E54
By following up with a subsequent request without the header and the same cachekey, the attacker could then retrieve the cached "Not Found" response, effectively rendering the package inaccessible to other users.
GET /safe-regex/-/safe-regex-1.1.0.tgz?lupin_E7A812DE-E09A-4906-A9E3-530E54AAEB41=cpdos_test HTTP/1.1
Host: registry.npmjs.org
User-Agent: lupin_test_cpdos
The second request would lead in the following response:
HTTP/1.1 404 Not Found
Date: Tue, 21 May 2024 08:40:06 GMT
Content-Type: application/json
Content-Length: 21
Connection: keep-alive
CF-Ray: 8873427f1e301222-MRS
Access-Control-Allow-Origin: *
Vary: Accept-Encoding
Server: cloudflare
{
"error": "Not found"
}
We discovered that the cached response was temporary (a few minutes), requiring the attacker to continuously send requests once in a while to maintain the CPDoS. We also observed that the registry employed multiple backend servers, necessitating the attack to be repeated to poison the cache on different servers.
While this attack was tested directly from Burp Suite, we've created a dummy package to test if we could poison from a npm install command. We proxified the npm cli to see what was happening, and then cleaned the cache from the current system. After running the attack in a loop for a couple of seconds, we managed to confirm that we could CPDoS a package in production after it was uploaded:
In Red: Before running the attack
In Green: While running the attack
The implications of this cache poisoning attack are significant. An attacker can target popular packages, such as "express" which has over 30 million downloads per week, and effectively render them unavailable to developers. This could lead to widespread disruptions in software development pipelines, causing builds to fail and applications to break.
We also conducted thorough testing of the attack using various IPs and clients, confirming that the poisoning affected not only the attacker but also other users attempting to access the targeted package. This highlights the potential for widespread impact if the vulnerability were to be exploited maliciously.
By successfully exploiting this vulnerability, an attacker could cause a denial of service on the npm registry, making it unavailable to users disrupting the software development processes of countless organizations worldwide. The attack could be particularly devastating if targeted at widely-used packages, as it could cause a ripple effect throughout the software supply chain.
However we've also found that there was an inconsistency of which backend server would be poisoned and that if the cache is already registered the attacker couldn't use a PURGE command to poison it back. We've also found out while digging into this behaviour that an attacker could possibly start the poisoning before a new version of a popular package is registered in order to poison the installations at the moment the version exists or when a package's cache is renewed.
The Ripple Effect: Consequences for the Software Supply Chain
The cache poisoning attack on the npm registry has far-reaching consequences that extend beyond the immediate availability of a single package. When a critical component like the npm registry is compromised, it can have a domino effect on the entire software supply chain.
Consider the scenario where a widely-used package like express is targeted by an attacker. With over 12 million downloads per week on its latest version, this package is a fundamental building block for countless web applications.
If the package becomes unavailable due to a cache poisoning attack, it can have a cascading impact on all the projects and services that depend on it.
The ripple effect can be felt across multiple organizations.
Github's Response to this vulnerability
GitHub, the owner of npm, conducted an investigation when receiving the Bug Bounty Report of this vulnerability. After assessing the findings, GitHub determined that the issue did not present a significant security risk and closed the report as informative while not fixing the issue.
GitHub acknowledged the complexity and limited reliability of reproducing the cache poisoning attack consistently. They also noted that the attack relied on using a problematic header, which was already known to their team. Furthermore, GitHub emphasized that executing the attack as described would require a DDoS or volumetric attack (we are going to discuss about that), which falls under the category of abuse and is not within the scope of their bug bounty program. Despite the report being ineligible for a full bounty reward following their claims, GitHub recognized our efforts and offered a small award of $500 as a token of appreciation.
Now we are bothered by their following claim:
To further execute this attack as you've described in your latest follow up comment, a DDoS or volumetric attack is needed, in which is an abuse issue. We take abuse and spam seriously and have a dedicated team that tracks down spammy users. Please also note that submissions around volumetric DoS attacks are not in the scope of our bounty program.
But as we showed in the report and this article, we successfully compromised a production package from just one machine. The attack had only two issues affecting its reliability:
- We can't predict which backend will be impacted
- We have to send a request every couple of minutes to maintain the poisoning
Based on our tests, a quarter of the requests were poisoned, which is already a big deal. We only mentioned that if an attacker sends more requests, the attack could be more reliable. But the resources needed for this wouldn't be anywhere close to a DDoS.
At the time of writing this article, this vulnerability is still exploitable on the npmjs registry. However we sent a first draft of the article to Github's security team to have an overall feedback on the technical part. Upon receiving the draft, Github's team sent us the following comment:
Thank you again for sharing with us the draft of your intended blogpost and for sending this report in, providing us the opportunity to harden our DoS protections. As we've discussed in the history of this issue, the reproducibility has been consistently difficult and requires a DDoS or volumetric attack to successfully exploit. In the effort to continue to improve the security of npm and our products, we are aiming to ship a fix this coming Monday that should address the issue despite its complexity. We thank you again for your contribution to our Bug Bounty program and are excited to continue to collaborate in the future.
It's necessary to state that we don't have anything against Github's security team. We actually met them and they are awesome people and super interesting to talk too. That being said, we are still disagreeing with the overall assessment of the situation. We strongly believe that those kind of CPDoS needs to be taken more seriously from a company that is at the center of the Software Supply Chain. Claiming that the described attack is volumetric and that falls into the abuse category is far fetched in our humble opinion.
We believe that in the future, this kind of availability attacks on the Software Supply Chain will be taken more seriously. More and more researchers and bug hunters are shedding the light on CPDoS issues and their implication in our ecosystem.
We strongly hope that all of this work done by the security community can be a real wake up call for the Software Supply Chain ecosystem.
Conclusion
The cache poisoning attack on the npm registry, discovered by our security research team at Lupin & Holmes, let us ponder the question about the fragility of our software supply chains.
The potential impact of such vulnerabilities cannot be understated. A compromised registry can have huge consequences, affecting developers, organizations, and end-users. The ripple effect of a single package becoming unavailable can disrupt development pipelines, cause application downtime, and lead to financial losses.
However, this vulnerability shows an opportunity for growth and improvement. It highlights the need for organizations like GitHub and npm to continuously evaluate and strengthen their security practices.
Moreover, the cache poisoning attack underscores the importance of collaboration and shared responsibility within the software supply chain. Developers, registry providers, and the wider community must work together to ensure the security, integrity and availability of the packages we rely on.
Timeline
- March 3, 2023, 10:14pm UTC Vulnerability reported by Lupin
- March 6, 2023, 5:11pm UTC Github Acknowledged the report
- March 9, 2023, 7:48pm UTC Github closes the issue as Internal Duplicate
- May 4, 2023, 9:42pm UTC Github reopen the report
- June 6, 2023, 6:49pm UTC Github triages the report
- August 30, 2023, 10:24pm UTC Github closes the report as Informative and pays 500$
- May 15, 2024, 9:44am UTC Lupin started the coordinate disclosure process
- May 16, 2024, 9:22pm UTC Github agrees to disclosue the issue
- May 24, 2024, 2:21pm UTC First draft sent to Github
- June 1, 2024, 2:15am UTC Github confirms that the issue will be fixed
