On October 6, 2026, a change we enabled caused our Vercel integration to overwrite some environment variable values with empty strings. Tasks that needed those values started failing.
We stopped the overwrites at 15:10 UTC, after 2 hours and 49 minutes. Recovering the values took longer: the bulk restoration finished at 19:52 UTC.
We're sorry. For affected customers, this meant broken workflows and time spent hunting down credentials that had been working earlier that day.
What happened
Vercel doesn't let integrations read variables marked Sensitive. Its list API returns those variables with value: "".
Our before-build sync treated that empty string as a value to save. It already skipped Vercel's legacy secret type, but it missed the Sensitive type.
That bug had been hidden by another behavior: Trigger.dev's import endpoint discarded empty values. We'd added support for storing empty strings because some applications need a variable to be present but blank. Enabling that feature removed the behavior that had been masking the sync bug.
We tested the feature on a test organization first. Our examples didn't contain Sensitive variables, so they never exercised Vercel's empty placeholders. The tests passed, and we expanded the rollout.
The integration setup and manual-pull paths already handled Sensitive variables correctly. We missed that the separate before-build implementation behaved differently.
What customers experienced
Our investigation identified affected variables across 121 customer organizations, in production, staging, and preview environments.
To be affected, a project needed Sensitive variables in Vercel and a before-build environment sync between 12:21 and 15:10 UTC.
Existing values were cleared, and some previously absent variables were created empty. The dashboard still showed the variable names, so you could see a missing-credential error, check your configuration, and find the variable apparently sitting right there.
If you re-entered a value during the incident, another build could clear it again. That made recovery particularly frustrating.
One of our own workflows started failing at 12:53 UTC. We traced those failures to the sync and identified the cause at 15:02 UTC.
Getting the values back
Disabling the feature stopped further overwrites. The values already cleared stayed empty, and we couldn't fetch them from Vercel because Sensitive values aren't available through its API.
We recovered a database copy from immediately before the incident and used it to restore encrypted values. We copied the encrypted data without decrypting customer secrets.
We also had to protect the work customers had already done. Restoring an old credential over a newly rotated one would have caused another problem.
The restore checked that each value was still empty and had no writes beyond the incident update. Those checks ran atomically with the restoration, and anything changed afterward was left alone.
The bulk restoration finished at 19:52 UTC. Variables without a value in the pre-incident backup needed manual recovery.
Timeline
All times are UTC on October 6, 2026.
| Time | Event |
|---|---|
| 12:21 | We enable empty-string support globally. Affected syncs begin. |
| 12:53 | An internal workflow starts failing with empty credentials. |
| 15:02 | We identify the cause. |
| 15:10 | We disable the feature, stopping further overwrites. |
| 16:13 | We send the first customer email. |
| 18:26 | The recovered database copy is ready, and restoration testing begins. |
| 19:08 | We email customers before starting the bulk restoration. |
| 19:52 | Bulk restoration of eligible values is complete. |
What we're changing
The fix starts with skipping Sensitive variables in the before-build sync. We're also planning receiving-side protection so an integration can't silently replace a populated value with an empty one.
Our tests need to follow the whole path, from Vercel's response to the value stored in Trigger.dev. That includes Sensitive variables alongside plain, encrypted, and intentionally empty values. We'll use those cases before expanding rollout and consolidate the rules used by our separate sync paths.
Recovery took too long. We're documenting the restore procedure and evaluating encrypted value history so getting a previous value back doesn't require recovering a database.
We also need to make empty values clearer in the dashboard and get affected-project details into customer emails faster. The first notification went out about an hour after we stopped the overwrites.
If you still have failing runs
Check that your Trigger.dev variables contain the current credentials, especially if you rotated them during the incident.
Restoring the values doesn't replay failed runs. Before replaying, check whether a run already sent messages, wrote records, or completed other actions you don't want repeated.
Reply to the incident email or contact Trigger.dev support if you need help. We're sorry our change broke your workflows and left you with recovery work.
- Dan