How we got hit by Shai-Hulud: A complete post-mortem

On November 25th, 2025, we were on a routine Slack huddle debugging a production issue when we noticed something strange: a PR in one of our internal repos was suddenly closed, showed zero changes, and had a single commit from... Linus Torvalds?

The commit message was just "init."

Within seconds, our #git Slack channel exploded with notifications. Dozens of force-pushes. PRs closing across multiple repositories. All attributed to one of our engineers.

We had been compromised by Shai-Hulud 2.0, a sophisticated npm supply chain worm that compromised over 500 packages, affected 25,000+ repositories, and spread across the JavaScript ecosystem. We weren't alone: PostHog, Zapier, AsyncAPI, Postman, and ENS were among those hit.

This is the complete story of what happened, how we responded, and what we've changed to prevent this from happening again.

No Trigger.dev packages were ever compromised. The @trigger.dev/* packages and trigger.dev CLI were never infected with Shai-Hulud malware. This incident involved one of our engineers installing a compromised package on their development machine, which led to credential theft and unauthorized access to our GitHub organization. Our published packages remained safe throughout.

The Attack Timeline

The compromise

On the evening of November 24th, around 20:27 UTC (9:27 PM local time in Germany), one of our engineers was experimenting with a new project. They ran a command that triggered pnpm install. At that moment, somewhere in the dependency tree, a malicious package executed.

We don't know exactly which package delivered the payload. The engineer was experimenting at the time and may have deleted the project directory as part of cleanup. By the time we investigated, we couldn't trace back to the specific package. The engineer checked their shell history and they'd only run install commands in our main trigger repo, cloud repo, and one experimental project.

This is one of the frustrating realities of these attacks: once the malware runs, identifying the source becomes extremely difficult. The package doesn't announce itself. The pnpm install completes successfully. Everything looks normal.

What we do know is that the Shai-Hulud malware ran a preinstall script that:

1. Downloaded and executed TruffleHog, a legitimate security tool repurposed for credential theft
2. Scanned the engineer's machine for secrets: GitHub tokens, AWS credentials, npm tokens, environment variables
3. Exfiltrated everything it found

When the engineer later recovered files from their compromised laptop (booted in recovery mode), they found the telltale signs:

The .trufflehog-cache directory and trufflehog_3.91.1_darwin_amd64.tar.gz file found on the compromised machine. The extract directory was empty, likely cleaned up by the malware to cover its tracks.

17 hours of reconnaissance

The attacker had access to our engineer's GitHub account for 17 hours before doing anything visible. According to our GitHub audit logs, they operated methodically.

Just over two hours after the initial compromise, the attacker validated their stolen credentials and began mass cloning:

The simultaneous activity from US and India confirmed we were dealing with a single attacker using multiple VPNs or servers, not separate actors.

While our engineer slept in Germany, the attacker continued their reconnaissance. More cloning at 02:56-02:59 UTC (middle of the night in Germany), sporadic activity until 05:32 UTC. Total repos cloned: 669 (527 from US infrastructure, 142 from India).

Here's where it gets unsettling. Our engineer woke up and started their normal workday:

The attacker was monitoring our engineer's activity while they worked, unaware they were compromised.

During this period, the attacker created repositories with random string names to store stolen credentials, a known Shai-Hulud pattern:

• github.com/[username]/xfjqb74uysxcni5ztn
• github.com/[username]/ls4uzkvwnt0qckjq27
• github.com/[username]/uxa7vo9og0rzts362c

They also created three repos marked with "Sha1-Hulud: The Second Coming" as a calling card. These repositories were empty by the time we examined them, but based on the documented Shai-Hulud behavior, they likely contained triple base64-encoded credentials.

10 minutes of destruction

At 15:27 UTC on November 25th, the attacker switched from reconnaissance to destruction.

The attack began on our cloud repo from India-based infrastructure:

The attack continued on our main repository:

At 15:32:43-46 UTC, 12 PRs on jsonhero-web were closed in 3 seconds. Clearly automated. PRs #47, #169, #176, #181, #189, #190, #194, #197, #204, #206, #208 all closed within a 3-second window.

Our critical infrastructure repository was targeted next:

The final PR was closed on json-infer-types at 15:37:13 UTC.

Detection and response

We got a lucky break. One of our team members was monitoring Slack when the flood of notifications started:

Our #git Slack channel during the attack. A wall of force-pushes, all with commit message "init."

Every malicious commit was authored as:

An attacked branch: a single "init" commit attributed to Linus Torvalds, thousands of commits behind main.

We haven't found reports of other Shai-Hulud victims seeing this same "Linus Torvalds" vandalism pattern. The worm's documented behavior focuses on credential exfiltration and npm package propagation, not repository destruction. This destructive phase may have been unique to our attacker, or perhaps a manual follow-up action after the automated worm had done its credential harvesting.

Within 4 minutes of detection we identified the compromised account, removed them from the GitHub organization, and the attack stopped immediately.

Our internal Slack during those first minutes:

"Urmmm guys? what's going on?"

"add me to the call @here"

"Nick could you double check Infisical for any machine identities"

"can someone also check whether there are any reports of compromised packages in our CLI deps?"

Within the hour:

The damage

Repository clone actions: 669 (public and private), including infrastructure code, internal documentation, and engineering plans.

Branches force-pushed: 199 across 16 repositories

Pull requests closed: 42

Protected branch rejections: 4. Some of our repositories have main branch protection enabled, but we had not enabled it for all repositories at the time of the incident.

npm packages were not compromised. This is the difference between "our repos got vandalized" and "our packages got compromised."

Our engineer didn't have an npm publishing token on their machine, and even if they did we had already required 2FA for publishing to npm. Without that, Shai-Hulud would have published malicious versions of @trigger.dev/sdk, @trigger.dev/core, and others, potentially affecting thousands of downstream users.

Production databases or any AWS resources were not accessed. Our AWS CloudTrail audit showed only read operations from the compromised account:

These were confirmed to be legitimate operations by our engineer.

One nice surprise: AWS actually sent us a proactive alert about Shai-Hulud. They detected the malware's characteristic behavior (ListSecrets, GetSecretValue, BatchGetSecretValue API calls) on an old test account that hadn't been used in months, so we just deleted it. But kudos to AWS for the proactive detection and notification.

The recovery

GitHub doesn't have server-side reflog. When someone force-pushes, that history is gone from GitHub's servers.

But we found ways to recover.

Push events are retained for 90 days via the GitHub Events API. We wrote a script that fetched pre-attack commit SHAs:

Public repository forks still contained original commits. We used these to verify and restore branches.

Developers who hadn't run git fetch --prune (all of us?) still had old SHAs in their local reflog.

Within 7 hours, all 199 branches were restored.

GitHub app private key exposure

During the investigation, our engineer was going through files recovered from the compromised laptop and discovered something concerning: the private key for our GitHub App was in the trash folder.

When you create a private key in the GitHub App settings, GitHub automatically downloads it. The engineer had created a key at some point, and while the active file had been deleted, it was still in the trash, potentially accessible to TruffleHog.

Our GitHub App has the following permissions on customer repositories:

To generate valid access tokens, an attacker would need both the private key (potentially compromised) and the installation ID for a specific customer (stored in our database which was not compromised, not on the compromised machine).

We immediately rotated the key:

We found no evidence of unauthorized access to any customer repositories. The attacker would have needed installation IDs from our database to generate tokens, and our database was not compromised as previously mentioned.

However, we cannot completely rule out the possibility. An attacker with the private key could theoretically have called the GitHub API to enumerate all installations. We've contacted GitHub Support to request additional access logs. We've also analyzed the webhook payloads to our GitHub app, looking for suspicious push or PR activity from connected installations & repositories. We haven't found any evidence of unauthorized activity in these webhook payloads.

We've sent out an email to potentially effected customers to notify them of the incident with detailed instructions on how to check if they were affected. Please check your email for more details if you've used our GitHub app.

Technical deep-dive: how Shai-Hulud works

For those interested in the technical details, here's what we learned about the malware from Socket's analysis and our own investigation.

When npm runs the preinstall script, it executes setup_bun.js:

1. Detects OS/architecture
2. Downloads or locates the Bun runtime
3. Caches Bun in ~/.cache
4. Spawns a detached Bun process running bun_environment.js with output suppressed
5. Returns immediately so npm install completes successfully with no warnings

The malware runs in the background while you think everything is fine.

The payload uses TruffleHog to scan $HOME for GitHub tokens (from env vars, gh CLI config, git credential helpers), AWS/GCP/Azure credentials, npm tokens from .npmrc, environment variables containing anything that looks like a secret, and GitHub Actions secrets (if running in CI).

Stolen credentials are uploaded to a newly-created GitHub repo with a random name. The data is triple base64-encoded to evade GitHub's secret scanning.

Files created:

• contents.json (system info and GitHub credentials)
• environment.json (all environment variables)
• cloud.json (cloud provider credentials)
• truffleSecrets.json (filesystem secrets from TruffleHog)
• actionsSecrets.json (GitHub Actions secrets if any)

If an npm publishing token is found, the malware validates the token against the npm registry, fetches packages maintained by that account, downloads each package, patches it with the malware, bumps the version, and re-publishes, infecting more packages.

This is how the worm spread through the npm ecosystem, starting from PostHog's compromised CI on November 24th at 4:11 AM UTC. Our engineer was infected roughly 16 hours after the malicious packages went live.

If no credentials are found to exfiltrate or propagate, the malware attempts to delete the victim's entire home directory. Scorched earth.

File artifacts to look for: setup_bun.js, bun_environment.js, cloud.json, contents.json, environment.json, truffleSecrets.json, actionsSecrets.json, .trufflehog-cache/ directory.

Malware file hashes (SHA1):

• bun_environment.js: d60ec97eea19fffb4809bc35b91033b52490ca11
• bun_environment.js: 3d7570d14d34b0ba137d502f042b27b0f37a59fa
• setup_bun.js: d1829b4708126dcc7bea7437c04d1f10eacd4a16

We've published a detection script that checks for Shai-Hulud indicators.

What we've changed

We disabled npm scripts globally:

This prevents preinstall, postinstall, and other lifecycle scripts from running. It's aggressive and some packages will break, but it's the only reliable protection against this class of attack.

We upgraded to pnpm 10. This was significant effort (had to migrate through pnpm 9 first), but pnpm 10 brings critical security improvements. Scripts are ignored by default. You can explicitly whitelist packages that need to run scripts via pnpm.onlyBuiltDependencies. And the minimumReleaseAge setting prevents installing packages published recently.

To whitelist packages that legitimately need build scripts:

This prompts you to select which packages to allow (like esbuild, prisma, sharp).

For your global pnpm config:

We switched npm publishing to OIDC. No more long-lived npm tokens anywhere. Publishing now uses npm's trusted publishers with GitHub Actions OIDC. Even if an attacker compromises a developer machine, they can't publish packages because there are no credentials to steal. Publishing only happens through CI with short-lived, scoped tokens.

We enabled branch protection on all repositories. Not just critical repos or just OSS repos. Every repository with meaningful code now has branch protection enabled.

We've adopted Granted for AWS SSO. Granted encrypts SSO session tokens on the client side, unlike the AWS CLI which stores them in plaintext.

Based on PostHog's analysis of how they were initially compromised (via pull_request_target), we've reviewed our GitHub Actions workflows. We now require approval for external contributor workflow runs on all our repositories (previous policy was only for public repositories).

Lessons for other teams

The ability for packages to run arbitrary code during installation is the attack surface. Until npm fundamentally changes, add this to your ~/.npmrc:

Yes, some things will break. Whitelist them explicitly. The inconvenience is worth it.

pnpm 10 ignores scripts by default and lets you set a minimum age for packages:

Newly published packages can't be installed for 3 days, giving time for malicious packages to be detected.

Branch protection takes 30 seconds to enable. It prevents attackers from pushing to a main branch, potentially executing malicious GitHub action workflows.

Long-lived npm tokens on developer machines are a liability. Use trusted publishers with OIDC instead.

If you don't need a credential on your local machine, don't have it there. Publishing should happen through CI only.

Our #git Slack channel is noisy. That noise saved us.

A note on the human side

One of the hardest parts of this incident was that it happened to a person.

"Sorry for all the trouble guys, terrible experience"

Our compromised engineer felt terrible, even though they did absolutely nothing wrong. It could have happened to any team member.

Running npm install is not negligence. Installing dependencies is not a security failure. The security failure is in an ecosystem that allows packages to run arbitrary code silently.

They also discovered that the attacker had made their GitHub account star hundreds of random repositories during the compromise. Someone even emailed us: "hey you starred my repo but I think it was because you were hacked, maybe remove the star?"

Summary

Resources

About the Attack:

• Socket.dev: Shai-Hulud Strikes Again V2 - Technical deep-dive into the malware
• PostHog Post-Mortem - Another company's experience with Shai-Hulud
• Wiz Blog: Shai-Hulud 2.0 Supply Chain Attack
• The Hacker News Coverage
• Endor Labs Analysis
• HelixGuard Advisory (referenced in AWS alert)

Mitigation Resources:

• npm Trusted Publishers - OIDC-based publishing
• pnpm onlyBuiltDependencies - Whitelist packages allowed to run scripts
• pnpm minimumReleaseAge - Delay installation of new packages
• Granted - AWS SSO credential management

Have questions about this incident? Reach out on Twitter/X or Discord.