Middays's Automated Bank Synchronization, powered by Trigger.dev

Product background

We are a team of 2 and have scaled to 11,500+ customers in the last 12 months while building in public together with our users.

At Midday, we use background jobs extensively and for our most important feature our bank connections, this is a deep dive into how use background jobs to make this as reliable as possible.

But first a little background on Midday, we are a all-in-one tool for Freelancers to manage their business. We help them with invoicing, time tracking, expense tracking, file reconciliation and much more.

We are a team of 2 and have scaled to 11,500+ customers in the last 12 months while building in public together with our users.

With that said it's important for us to choose the right tools for the job, we are fully serverless, using Supabase as our backend, and Vercel for our Frontend hosting.

We quickly realised that we needed a scalable and more reliable way to fetch transactions and accounts from our 3 different banking providers (GoCardLess in EU, Teller and Plaid in US).

While we where searching we found Trigger.dev and we were able to build a first solution in a couple of days!

We just recently updated everything to v3 with over 30 tasks, and are super happy with the results. Let's get into the details for our bank connections...

Bank connections

A user can connect their bank account to Midday, this is done via our own Engine API that connects to our 3 providers so we can consume transactions and accounts in one structured format, this is the first step in our bank connection process.

On success we trigger an initial-bank-setup task that will register a scheduled cron job to run every 24 hours on a random time to avoid rate limiting.

Then we fire of the sync-connection task (view the code here) that will do a fan out pattern for syncing accounts and transactions, on initial sync we run everything in parallel, but if it's a sync task we run them with a delay to avoid the banks rate limiting.

For sync tasks we also send notifications about new transactions by triggering a transaction-notifications task.

Because we run this in Trigger.dev tasks we have the ability to run retries and backoffs, together with the Slack alert integration we are always on top of any issues regarding the bank connections for our customers.

Sync tasks

While Plaid and Teller support webhooks, GoCardLess does not, so we need to fetch the transactions from the bank every 24 hours.

We do that for all providers, but with webhooks we also fire the sync-connection task on an incoming webhook to make sure we get the latest transactions instantly.

This task is the same as the initial-bank-setup task, but in this case we only fetch the last few days of transactions.

New transaction notifications

We also send notifications about new transactions by triggering a new-transaction task, we generate it dynamically using React Email, and send it to our notification service over at Novu, from there we send the email via Resend, and in app notifications using their built-in solution. Soon we will also send Push Notifications to our Mobile app.

Manual sync

We also have a manual sync button in the app, this will trigger a manual-sync task, this task runs at the same time as the sync-connection task, but we also added the Realtime hook to show the user the status of the bank connection on the frontend.

Reconnect flow

In the EU we need to reconnect our bank connections every 180 days due to regulatory reasons, so we need to trigger a reconnect-bank task. The user will go through the same process as the initial bank setup, but we also need to update the bank connection in our database, through the process we use the Realtime feature to show the user the status of the bank connection and in this task we rely heavily on retries due to bank API restrictions.

Conclusion

While this is our most complex task, it's also the most important one, and we are super happy with the results, we are able to keep our customers bank connections running smoothly.

If you where about to ask for a good way to handle background jobs, I would recommend Trigger.dev, it's a game changer for us.

If you'd like to learn more about Midday and how we use Trigger.dev, you can check out our open source Github Repo or reach out to me via X on pontusab.