Zod 4 got rid of a bunch of CPUs for us

Upgrading to Zod 4 cut worker CPU by almost a third. And we became Zod's first Platinum Sponsor.

Chris Arderne

Chris Arderne

Software Engineer, Trigger.dev

Image for Zod 4 got rid of a bunch of CPUs for us

We (finally) upgraded to Zod 4.5 (from 3.x!) and our worker CPU demand plummeted, our batch throughput increased by 14% and our task trigger p99 dropped by 18%.

Aaaand in unrelated news… Trigger.dev is now the platinum sponsor for Zod! (jump to the end).


We parse a lot of JSON. Task payloads, batch items, queue messages, retry wrappers. Piles of our own schemas and piles of wrapped-up user schemas too.

Zod sits on those paths all day, parsing and safeParsing away.

It's hard to measure stuff when you're constantly growing. Marginal improvements are immediately swallowed by increased load. Plus we run extremely variable customer workloads. A bad database spike could just as easily be a missing index or a whale customer running some experiments.

But this was a change that was immediately noticeable: one of our main services dropped its CPU demand by nearly a third.

Fleet CPU after Zod 4
24h mean, indexed to Zod 3 = 100. Workers handled more throughput after.
Zod 3Zod 4
025507510069Engine workers−31%85Batch workers−15%90API−10%

How much Zod is that?

It's hard to say exactly how much Zod we do. I loaded some data for one of our worker services, and it was doing about 1,100 parses a second, nearly 100 million a day. We do a lot more Zod than this, but I needed a chart and this is what I could find.

Worker parses / second98M / day
Estimated Zod activity for Monday 14 September on one hot path.
05001,0001,5001,48009:0013:0017:0021:0001:0005:00

Each job does one safeParse (it's a tiny schema with a key, run id, that kind of thing). About half the jobs then parse every message - maybe another 10 parses per job. Not to mention the various API surfaces that get a job going in the first place, plus retry wrappers, waitpoints, heartbeats, etc.

The upgrade also bought us a bit of Event Loop Utilisation (ELU) breathing room. Have we talked about how ELU is the thing to watch? We spend a lot of time watching our ELU dashboards, and it was satisfying to see them tick gently down. Less time parsing on the hot path, less time blocking the main loop, more headroom to grow.

Event loop utilisation
Mean per-process ELU. Lower is more headroom.
Zod 3Zod 4
0%20%40%60%52%47%Engine workers−10%36%35%Batch workers−3%30%27%API−9%

Did compiling schemas help?

Zod recently added schema compilation, which creates a flat loop-free validator that should theoretically be even faster. For now we just experimented with our simple (and not user-defined) server schemas: queue messages, batch items, trigger bodies.

I tracked another day of production telemetry and compared like-for-like: about 5% less worker CPU, lower ELU. Nothing huge, but nothing to sniff at either. We'll experiment with some more schema compilation and see if it moves the needle again.

We're Zod's first platinum sponsor

Zod has been great for us and our users, so it felt like it was time to pay it forward. We're now Zod's first Platinum Sponsor!

Building a platform for agents that serves tens of thousands of users is complex and there are tons of moving parts, and all of it is getting bigger and beastlier every day.

We rely on loads of incredible open source tech. We're proudly open source ourselves, and we benefit hugely from the community.

Zod is one of those bits that we use and love and rely on every day and that's why we wanted to support the work Colin is doing.

Addendum

It was almost pain-free. But we hit a little "race condition" of sorts. I had the PR up for a good couple of weeks, waiting for the opportune moment to cut a minor release forcing the new minimum Zod version. I had been rebasing and rebasing, and re-testing and re-testing. The trickiest part of the upgrade was in our "core" package, which is used by our internal systems as well as our published SDK and CLI. And it's often the seam (I swear a human wrote this) between Zod v3-shaped and Zod v4-shaped stuff. In this core my PR changed the imports to explicitly use zod/v4 for all the imports, so that users on end-of-life v3 would get the correct packages.

But just after merging, a colleague added a module to core that added a bare zod import, and I didn't catch it in my sweeps. This caused a tricky issue where users who had (a) updated to this released version and (b) were still using Zod v3 and (c) were using npm (instead of pnpm etc) would get delayed warm starts on their task runs. It took us a while to debug, but eventually we could ignore our alerts and warnings no longer, and we fixed it at our offsite in Lisbon. Our hotel was playing horrible covers of 90s hits, which set the mood perfectly. We also added an oxlint check so it can't happen again.

Ready to start building?

Build and deploy your first task in 3 minutes.

Get started now