Self-hosting
You can self-host Trigger.dev on your own infrastructure.
Overview
The self-hosting guide covers two alternative setups. The first option uses a simple setup where you run everything on one server. With the second option, the webapp and worker components are split on two separate machines.
You’re going to need at least one Debian (or derivative) machine with Docker and Docker Compose installed. We’ll also use Ngrok to expose the webapp to the internet.
Support
It’s dangerous to go alone! Join the self-hosting channel on our Discord server.
Caveats
This guide outlines a quick way to start self-hosting Trigger.dev for evaluation purposes - it won’t result in a production-ready deployment. Security, scaling, and reliability concerns are not fully addressed here.
As self-hosted deployments tend to have unique requirements and configurations, we don’t provide specific advice for securing your deployment, scaling up, or improving reliability.
Should the burden ever get too much, we’d be happy to see you on Trigger.dev cloud where we deal with these concerns for you.
Requirements
- 4 CPU
- 8 GB RAM
- Debian or derivative
- Optional: A separate machine for the worker components
You will also need a way to expose the webapp to the internet. This can be done with a reverse proxy, or with a service like Ngrok. We will be using the latter in this guide.
Option 1: Single server
This is the simplest setup. You run everything on one server. It’s a good option if you have spare capacity on an existing machine, and have no need to independently scale worker capacity.
Server setup
Some very basic steps to get started:
Trigger.dev setup
- Clone the Trigger.dev docker repository
git clone https://github.com/triggerdotdev/docker
cd docker
- Run the start script and follow the prompts
./start.sh # hint: you can append -d to run in detached mode
Manual
Alternatively, you can follow these manual steps after cloning the docker repo:
- Create the
.env
file
cp .env.example .env
- Generate the required secrets
echo MAGIC_LINK_SECRET=$(openssl rand -hex 16)
echo SESSION_SECRET=$(openssl rand -hex 16)
echo ENCRYPTION_KEY=$(openssl rand -hex 16)
echo PROVIDER_SECRET=$(openssl rand -hex 32)
echo COORDINATOR_SECRET=$(openssl rand -hex 32)
-
Replace the default secrets in the
.env
file with the generated ones -
Run docker compose to start the services
. lib.sh # source the helper function
docker_compose -p=trigger up
Tunnelling
You will need to expose the webapp to the internet. You can use Ngrok for this. If you already have a working reverse proxy setup and a domain, you can skip to the last step.
- Start Ngrok. You may get prompted to sign up - it’s free.
./tunnel.sh
-
Copy the domain from the output, for example:
1234-42-42-42-42.ngrok-free.app
-
Uncomment the
TRIGGER_PROTOCOL
andTRIGGER_DOMAIN
lines in the.env
file. Set it to the domain you copied.
TRIGGER_PROTOCOL=https
TRIGGER_DOMAIN=1234-42-42-42-42.ngrok-free.app
- Quit the start script and launch it again, or run this:
./stop.sh && ./start.sh
Registry setup
If you want to deploy v3 projects, you will need access to a Docker registry. The CLI deploy command will push the images, and then the worker machine can pull them when needed. We will use Docker Hub as an example.
-
Sign up for a free account at Docker Hub
-
Edit the
.env
file and add the registry details
DEPLOY_REGISTRY_HOST=docker.io
DEPLOY_REGISTRY_NAMESPACE=<your_dockerhub_username>
- Log in to Docker Hub both locally and your server. For the split setup, this will be the worker machine. You may want to create an access token for this.
docker login -u <your_dockerhub_username> docker.io
- Required on some systems: Run the login command inside the
docker-provider
container so it can pull deployment images to run your tasks.
docker exec -ti \
trigger-docker-provider-1 \
docker login -u <your_dockerhub_username> docker.io
- Restart the services
./stop.sh && ./start.sh
- You can now deploy v3 projects using the CLI with these flags:
npx trigger.dev@latest deploy --self-hosted --push
Option 2: Split services
With this setup, the webapp will run on a different machine than the worker components. This allows independent scaling of your workload capacity.
Webapp setup
All steps are the same as for a single server, except for the following:
- Startup. Run the start script with the
webapp
argument
./start.sh webapp
- Tunnelling. This is now required. Please follow the tunnelling section.
Worker setup
- Environment variables. Copy your
.env
file from the webapp to the worker machine:
# an example using scp
scp -3 root@<webapp_machine>:docker/.env root@<worker_machine>:docker/.env
- Startup. Run the start script with the
worker
argument
./start.sh worker
-
Tunnelling. This is not required for the worker components.
-
Registry setup. Follow the registry setup section but run the last command on the worker machine - note the container name is different:
docker exec -ti \
trigger-worker-docker-provider-1 \
docker login -u <your_dockerhub_username> docker.io
Additional features
Large payloads
By default, payloads over 512KB will be offloaded to S3-compatible storage. If you don’t provide the required env vars, runs with payloads larger than this will fail.
For example, using Cloudflare R2:
OBJECT_STORE_BASE_URL="https://<bucket>.<account>.r2.cloudflarestorage.com"
OBJECT_STORE_ACCESS_KEY_ID="<r2 access key with read/write access to bucket>"
OBJECT_STORE_SECRET_ACCESS_KEY="<r2 secret key>"
Alternatively, you can increase the threshold:
# size in bytes, example with 5MB threshold
TASK_PAYLOAD_OFFLOAD_THRESHOLD=5242880
Version locking
There are several reasons to lock the version of your Docker images:
- Backwards compatibility. We try our best to maintain compatibility with older CLI versions, but it’s not always possible. If you don’t want to update your CLI, you can lock your Docker images to that specific version.
- Ensuring full feature support. Sometimes, new CLI releases will also require new or updated platform features. Running unlocked images can make any issues difficult to debug. Using a specific tag can help here as well.
By default, the images will point at the latest versioned release via the v3
tag. You can override this by specifying a different tag in your .env
file. For example:
TRIGGER_IMAGE_TAG=v3.0.4
Auth options
By default, magic link auth is the only login option. If the RESEND_API_KEY
env var is not set, the magic links will be logged by the webapp container and not sent via email.
All email addresses can sign up and log in this way. If you would like to restrict this, you can use the WHITELISTED_EMAILS
env var. For example:
# every email that does not match this regex will be rejected
WHITELISTED_EMAILS="authorized@yahoo\.com|authorized@gmail\.com"
It’s currently impossible to restrict GitHub OAuth logins by account name or email like above, so this method is not recommended for self-hosted instances. It’s also very easy to lock yourself out of your own instance.
Your GitHub OAuth app needs a callback URL https://<your_domain>/auth/github/callback
and you will have to set the following env vars:
AUTH_GITHUB_CLIENT_ID=<your_client_id>
AUTH_GITHUB_CLIENT_SECRET=<your_client_secret>
Checkpoint support
This requires an experimental Docker feature. Successfully checkpointing a task today, does not mean you will be able to restore it tomorrow. Your data may be lost. You’ve been warned!
Checkpointing allows you to save the state of a running container to disk and restore it later. This can be useful for long-running tasks that need to be paused and resumed without losing state. Think fan-out and fan-in, or long waits in email campaigns.
The checkpoints will be pushed to the same registry as the deployed images. Please see the registry setup section for more information.
Requirements
- Debian, NOT a derivative like Ubuntu
- Additional storage space for the checkpointed containers
Setup
Underneath the hood this uses Checkpoint and Restore in Userspace, or CRIU in short. We’ll have to do a few things to get this working:
- Install CRIU
sudo apt-get update
sudo apt-get install criu
- Tweak the config so we can successfully checkpoint our workloads
mkdir -p /etc/criu
cat << EOF >/etc/criu/runc.conf
tcp-close
EOF
- Make sure everything works
sudo criu check
- Enable Docker experimental features, by adding the following to
/etc/docker/daemon.json
{
"experimental": true
}
- Restart the Docker daemon
sudo systemctl restart docker
- Uncomment
FORCE_CHECKPOINT_SIMULATION=0
in your.env
file. Alternatively, run this:
echo "FORCE_CHECKPOINT_SIMULATION=0" >> .env
- Restart the services
# if you're running everything on the same machine
./stop.sh && ./start.sh
# if you're running the worker on a different machine
./stop.sh worker && ./start.sh worker
Updating
Once you have everything set up, you will periodically want to update your Docker images. You can easily do this by running the update script and restarting your services:
./update.sh
./stop.sh && ./start.sh
Sometimes, we will make more extensive changes that require pulling updated compose files, scripts, etc from our docker repo:
git pull
./stop.sh && ./start.sh
Occasionally, you may also have to update your .env
file, but we will try to keep these changes to a minimum. Check the .env.example
file for new variables.
From beta
If you’re coming from the beta CLI package images, you will need to:
- Stash you changes. If you made any changes, stash them with
git stash
. - Switch branches. We moved back to main. Run
git checkout main
in your docker repo. - Pull in updates. We’ve added a new container for Electric and made some other improvements. Run
git pull
to get the latest updates. - Apply your changes. If you stashed your changes, apply them with
git stash pop
. - Update your images. We’ve also published new images. Run
./update.sh
to pull them. - Restart all services. Run
./stop.sh && ./start.sh
and you’re good to go.
In summary, run this wherever you cloned the docker repo:
# if you made changes
git stash
# switch to the main branch and pull the latest changes
git checkout main
git pull
# if you stashed your changes
git stash pop
# update and restart your services
./update.sh
./stop.sh && ./start.sh
Troubleshooting
- Deployment fails at the push step. The machine running
deploy
needs registry access:
docker login -u <username> <registry>
# this should now succeed
npx trigger.dev@latest deploy --self-hosted --push
- Prod runs fail to start. The
docker-provider
needs registry access:
# single server? run this:
docker exec -ti \
trigger-docker-provider-1 \
docker login -u <your_dockerhub_username> docker.io
# split webapp and worker? run this on the worker:
docker exec -ti \
trigger-worker-docker-provider-1 \
docker login -u <your_dockerhub_username> docker.io
CLI usage
This section highlights some of the CLI commands and options that are useful when self-hosting. Please check the CLI reference for more in-depth documentation.
Login
To avoid being redirected to the Trigger.dev Cloud login page when using the CLI, you can specify the URL of your self-hosted instance with the --api-url
or -a
flag. For example:
npx trigger.dev@latest login -a http://trigger.example.com
Once you’ve logged in, the CLI will remember your login details and you won’t need to specify the URL again with other commands.
Custom profiles
You can specify a custom profile when logging in. This allows you to easily use the CLI with our cloud product and your self-hosted instance at the same time. For example:
npx trigger.dev@latest login -a http://trigger.example.com --profile my-profile
You can then use this profile with other commands:
npx trigger.dev@latest dev --profile my-profile
To list all your profiles, use the list-profiles
command:
npx trigger.dev@latest list-profiles
Verify login
It can be useful to check you have successfully logged in to the correct instance. You can do this with the whoami
command, which will also show the API URL:
npx trigger.dev@latest whoami
# with a custom profile
npx trigger.dev@latest whoami --profile my-profile
Deploy
On Trigger.dev Cloud, we build deployments remotely and push those images for you. When self-hosting you will have to do that locally yourself. This can be done with the --self-hosted
and --push
flags. For example:
npx trigger.dev@latest deploy --self-hosted --push
CI / GitHub Actions
When running the CLI in a CI environment, your login profiles won’t be available. Instead, you can use the TRIGGER_API_URL
and TRIGGER_ACCESS_TOKEN
environment
variables to point at your self-hosted instance and authenticate.
For more detailed instructions, see the GitHub Actions guide.
Telemetry
By default, the Trigger.dev webapp sends telemetry data to our servers. This data is used to improve the product and is not shared with third parties. If you would like to opt-out of this, you can set the TRIGGER_TELEMETRY_DISABLED
environment variable in your .env
file. The value doesn’t matter, it just can’t be empty. For example:
TRIGGER_TELEMETRY_DISABLED=1
Was this page helpful?