Exploiting Fortune 500 Through Hidden Supply Chain Links

Oct 28, 2024

RONI CARTA | LUPIN

npm, supply chain attack, dependency confusion, hack, bounty, hashicorp

Introduction

Software Supply Chains have become increasingly complex, with countless dependencies forming the backbone of modern applications. Despite this complexity, many attacks targeting these dependencies are surprisingly simple. While significant time is spent ensuring the security and functionality of individual packages, the connections between them often go unnoticed.

The industry’s focus tends to be on the inner workings of dependencies—verifying that the code is secure and up to date. However, this approach overlooks the external links these dependencies establish with third-party sources, which can introduce vulnerabilities. By neglecting these relationships, we risk exposing our systems to attacks that exploit the trust placed in these external connections.

This is where Depi, our Software Supply Chain Security tool, steps in. Instead of concentrating only on individual components, Depi highlights the overlooked weak points in the links between them. It pinpoints how attackers might breach these connections, giving you a clearer view of potential threats within your software supply chain. Depi helps shift the focus from internal code security to protecting the broader network of dependencies.

In today’s article, we’ll explore how Depi identified a Software Supply Chain Attack (SCA) targeting Consul, HashiCorp’s open-source tool for service discovery and network automation, and the impact it had on a major Fortune 500 company with a 17,000$ Bug Bounty Reward.

How it all started

During Depi's development, it was important for us to reach out to a wide range of industry experts to shape the tool to meet the needs of companies. One of the key figures we consulted was Caleb Sima.

Caleb is a well-known figure in the field of cybersecurity, with decades of experience in ethical hacking, application security, and security strategy. He has co-founded and led several security-focused companies and has held executive positions in major technology firms. If you're planning to launch a SaaS in cybersecurity, Caleb is the person to talk to.

He provided valuable feedback and asked us to demonstrate the tool’s capabilities. To test its effectiveness, he suggested scanning HashiCorp’s open-source repositories to see what Depi could uncover.

We accepted the challenge and began the scan. That’s when we came across something interesting:

consul-peerings vulnerable

The vulnerable package was defined in the consul/ui/packages/consul-ui/package.json file as follows:

{
  "devDependencies": {
    "consul-lock-sessions": "*",
  }
}

This package, marked as a devDependency, is set to pull any version from the registry. However, in this scenario, the expected behavior is for the package manager to first check the ../package.json file, as this is a workspace directory:

{
  "private": true,
  "description": "Monorepo for Consul UI, packages and addons.",
  "license": "MPL-2.0",
  "author": "HashiCorp",
  "workspaces": {
    "packages": [
      "packages/*"
    ]
  },
  "scripts": {
    "doc:toc": "doctoc README.md",
    "compliance": "npm-run-all compliance:*",
    "compliance:licenses": "license-checker --summary --onlyAllow 'Python-2.0;Apache*;Apache License, Version 2.0;Apache-2.0;Apache 2.0;Artistic-2.0;BSD;BSD-3-Clause;CC-BY-3.0;CC-BY-4.0;CC0-1.0;ISC;MIT;MPL-2.0;Public Domain;Unicode-TOU;Unlicense;WTFPL' --excludePackages '[email protected];[email protected];[email protected];[email protected];[email protected];[email protected]'"

  },
  "devDependencies": {
    "doctoc": "^2.0.0",
    "license-checker": "^25.0.1",
    "npm-run-all": "^4.1.5"
  },
  "resolutions": {
    "xmlhttprequest-ssl": "^1.6.3",
    "ember-basic-dropdown": "3.0.21"
  },
  "engines": {
    "node": "18"
  }
}

In this case, the consul-lock-sessions package should be pulled from ../packages/consul-lock-sessions. Some package managers, like yarn Classic or npm, correctly detect that we are in a workspace by checking the parent directories. However, this is not true for pnpm. If someone were to execute the following:

git clone [email protected]:hashicorp/consul.git

cd consul/ui/packages/consul-ui/

pnpm install

The package manager would search the upstream registry (registry.npmjs.org) for consul-lock-sessions. This opens up a possible attack scenario, particularly for developers working with pnpm in a monorepo setup—at least that’s what we initially thought. Spoiler alert: things got more interesting.

Before jumping into exploitation, we noticed a weird thing. Several packages were defined the same way, but weren’t flagged by Depi:

{
  "devDependencies": {
    "consul-acls": "*",
    "consul-lock-sessions": "*",
    "consul-nspaces": "*",
    "consul-partitions": "*",
    "consul-peerings": "*",
  }
}

The reason for this is that the registry had previously published these packages, which were later unpublished. When querying the registry’s API, we received a 200 response (and not the expected 404), but with no version data. The result looked like this:

$ curl https://registry.npmjs.org/consul-lock-sessions | jq

{
  "_id": "consul-lock-sessions",
  "name": "consul-lock-sessions",
  "time": {
    "created": "YYYY-MM-DDTHH:mm:ss.xx1Z",
    "3000.0.0": "YYYY-MM-DDTHH:mm:ss.xx1Z",
    "modified": "YYYY-MM-DDTHH:mm:ss.xx1Z",
    "3001.0.0": "YYYY-MM-DDTHH:mm:ss.xx1Z",
    "unpublished": {
      "time": "YYYY-MM-DDTHH:mm:ss.xx1Z",
      "versions": [
        "3000.0.0",
        "3001.0.0"
      ]
    }
  }
}

This indicated that someone had previously attempted to exploit this but didn’t succeed, as the package was later removed from the registry.

We’ve discovered that in this state the package is still claimable.

We’ve learned from this behavior and changed the way Depi was detecting claimable packages on npm. After running a new scan Depi found the following exploits:

all packages vulnerable

Perfect, we were now ready to exploit this vulnerability.

Exploit

We got in touch with Caleb to set up a demo to showcase the pnpm exploit on Consul. At the time we thought that only users browsing the specific directory with pnpm were vulnerable to this attack. Given the level of user interaction required, we decided to claim the package in order to build a complete end-to-end demo to present to HashiCorp. We claimed the package on the registry, setting up a pingback to our servers upon installation to demonstrate remote code execution.

This was done by adding a preinstall command that exfiltrates the hostname, whoami and path of the target. These details allow ethical hackers to verify that the correct system has executed the command and help companies identify the exact location of the misconfiguration allowing the Dependency Confusion to happen. This probe is useful in incident response, especially when there are multiple potential failure points, such as developer machines, CI pipelines, or production servers.

We showed the demo to Caleb on a Friday, and he offered to connect us with HashiCorp's team the following Monday. Since the exploit wasn't highly critical (at least from what we knew at the time) and couldn't be abused by bad actors because we’ve already taken over the packages, we didn't think much of waiting over the weekend.

A few hours after the demo, however, we got an unexpected notification on our pingback server:

{
  "pwd"=>"/Users/REDACTED/src/consul/ui/packages/consul-ui/node_modules/consul-peerings/node_modules/axios", 
  "whoami"=>"REDACTED", 
  "hostname"=>"REDACTED.internal.fortune500.tld"
}

My first reaction was to message my brother Holmes, excited by what we had discovered. It turned out we had received a pingback from an internal machine coming with the hostname of a Fortune 500 company after claiming HashiCorp’s packages.

What made this even more intriguing was that the User-Agent in the pingback wasn’t from pnpm, as we expected, but from npm:

{ 
  "headers": {
        "accept": "*/*",
        "accept-encoding": "gzip, br",
        "connection": "Keep-Alive",
        "npm-in-ci": "false",
        "npm-scope": "",
        "pacote-pkg-id": "registry:undefined@https://redacted.tld/consul-lock-sessions",
        "pacote-req-type": "tarball",
        "referer": "install",
        "user-agent": "npm/6.14.15 node/v14.19.0 darwin arm64",
      }
}

This showed that our initial assumption—that only pnpm was vulnerable—was incorrect. Other package managers, like npm, were also susceptible to default dependency confusion in workspaces.

The Plot Thickens

Disclosure to the Fortune 500 company

Our next move was to responsibly disclose the issue to the affected company. The Fortune 500 affected by this vulnerability is running bug bounty programs out there. One of the best things with their program is the transparency and support they offer to bug hunters, helping to enhance their findings. This level of transparency and collaboration reflects the maturity of a well-run bug bounty program. We were confident that they would handle this Dependency Confusion disclosure quickly and fairly.

The report was triaged in less than a day, and within two weeks, we received the maximum reward of $17,000.

It's crucial for companies to actively engage with ethical hackers and maintain open channels for receiving responsible disclosures when vulnerabilities are found. This collaboration helps identify potential threats early and allows organizations to address them before they can be exploited maliciously. In our case, the partnership with this company was a great example of how companies should work with ethical hackers to ensure thorough assessments and prompt resolutions.

Equally important is properly evaluating supply chain attacks. These types of vulnerabilities can have far-reaching impacts, and it's essential to work closely with ethical hackers to fully understand the scope. This company facilitated an insightful discussion on Software Supply Chain Attacks (SCA), emphasizing the need for careful assessment in bug bounty programs. This open dialogue ensures that these complex threats are handled appropriately, with both the company and the hackers collaborating to mitigate risks effectively.

Funnily enough, none of the security scanners caught our small-scale data exfiltration, but npm still took control of the packages and banned our account. What happened was that the affected company emailed npm, asking them to take down the packages we had claimed, even before we sent in our report, as part of their security process. By doing so, they missed the chance to use the pingback probe to investigate which developer, pipeline, or server was misconfigured, leaving a blind spot in the fix. The npm team eventually claimed the packages with this message:

Security holding package: This package contained malicious code and was removed from the registry by the npm security team. A placeholder was published to ensure users are not affected in the future.

This vulnerability is not exploitable anymore even for older versions of Consul because of this.

Disclosure to HashiCorp

Shortly after our first disclosure, Caleb connected us with HashiCorp's security team. They received a redacted version of the report and quickly moved to address the vulnerability. The team implemented a patch, replacing the wildcard dependencies with local file references in their package.json files. The original configuration:

"consul-acls": "*",
"consul-lock-sessions": "*",
"consul-nspaces": "*",
"consul-partitions": "*",
"consul-peerings": "*",

was updated to:

"consul-acls": "file:../consul-acls",
"consul-lock-sessions": "file:../consul-lock-sessions",
"consul-nspaces": "file:../consul-nspaces",
"consul-partitions": "file:../consul-partitions",
"consul-peerings": "file:../consul-peerings",

When asked why they opted for the file: protocol instead of a namespaced solution like @namespace/package, HashiCorp explained that many files contained hard coded references. Switching to namespaced packages would have required significant refactoring, making the file protocol a simpler and faster fix. You can find more details in their security bulletin here.

In summary, the vulnerability affected the development workflows of HashiCorp Consul and Consul Enterprise, from versions 1.12.0 through 1.19.0. While it didn’t impact production environments, the issue was resolved in Consul 1.17.7, 1.18.3, and 1.19.1. The problem stemmed from an ambiguous dependency resolution introduced in March 2022, which allowed the Lupin & Holmes team to publish a “malicious” package to the npm registry and run arbitrary code in affected development environments. This vulnerability has since been fixed by explicitly using the file: protocol to reference local packages.

Customers using pre-built Consul binaries aren’t affected, but those building Consul UI from source should upgrade to a patched version.

An interesting point for Security Research is that, although HashiCorp doesn’t offer a Bug Bounty Program with financial rewards, they still value efforts to identify and coordinate the disclosure of security vulnerabilities. They provide guidance on how to report vulnerabilities to them. See HashiCorp's Vulnerability Reporting.

Conclusion

Depi plays a key role in safeguarding software supply chains by identifying vulnerabilities in the often-overlooked links between dependencies. In this case, Depi not only flagged an attack vector in HashiCorp’s Consul development environment but also highlighted how simple misconfigurations can lead to significant risks. By focusing on the broader network of dependencies, Depi helps organizations stay ahead of Supply Chain Attacks and better protect their systems.

Bug bounty programs are invaluable to our research and development process. They provide a real-world testing ground for our team at Lupin & Holmes to discover and validate new types of vulnerabilities, which we can then incorporate into Depi’s detection capabilities. The collaboration with ethical hackers through these programs ensures that we remain at the top of software supply chain security, continuously improving our tools based on the experience gained from real-world scenarios.

We would like to say a big thank you to Caleb Sima for his guidance, as well as the teams at HashiCorp for their quick response, transparency, and open communication throughout this process. Especially the HashiCorp ProdSec team and the Consul R&D team. Their willingness to collaborate ensured that the vulnerability was addressed swiftly, reinforcing the importance of cooperation between researchers, ethical hackers, and security teams in keeping software ecosystems safe.