Errors & Retrying
How to deal with errors and write reliable tasks.
When an uncaught error is thrown inside your task, that task attempt will fail.
You can configure retrying in two ways:
- In your trigger.config file you can set the default retrying behavior for all tasks.
- On each task you can set the retrying behavior.
By default when you create your project using the CLI init command we disabled retrying in the DEV environment. You can enable it in your trigger.config file.
A simple example with OpenAI
This task will retry 10 times with exponential backoff.
openai.chat.completions.create()
can throw an error.- The result can be empty and we want to try again. So we manually throw an error.
import { task } from "@trigger.dev/sdk/v3";
import OpenAI from "openai";
const openai = new OpenAI({
apiKey: process.env.OPENAI_API_KEY,
});
export const openaiTask = task({
id: "openai-task",
//specifying retry options overrides the defaults defined in your trigger.config file
retry: {
maxAttempts: 10,
factor: 1.8,
minTimeoutInMs: 500,
maxTimeoutInMs: 30_000,
randomize: false,
},
run: async (payload: { prompt: string }) => {
//if this fails, it will throw an error and retry
const chatCompletion = await openai.chat.completions.create({
messages: [{ role: "user", content: payload.prompt }],
model: "gpt-3.5-turbo",
});
if (chatCompletion.choices[0]?.message.content === undefined) {
//sometimes OpenAI returns an empty response, let's retry by throwing an error
throw new Error("OpenAI call failed");
}
return chatCompletion.choices[0].message.content;
},
});
Combining tasks
One way to gain reliability is to break your work into smaller tasks and trigger them from each other. Each task can have its own retrying behavior:
import { task } from "@trigger.dev/sdk/v3";
export const myTask = task({
id: "my-task",
retry: {
maxAttempts: 10,
},
run: async (payload: string) => {
const result = await otherTask.triggerAndWait("some data");
//...do other stuff
},
});
export const otherTask = task({
id: "other-task",
retry: {
maxAttempts: 5,
},
run: async (payload: string) => {
return {
foo: "bar",
};
},
});
Another benefit of this approach is that you can view the logs and retry each task independently from the dashboard.
Retrying smaller parts of a task
Another complimentary strategy is to perform retrying inside of your task.
We provide some useful functions that you can use to retry smaller parts of a task. Of course, you can also write your own logic or use other packages.
retry.onThrow()
You can retry a block of code that can throw an error, with the same retry settings as a task.
import { task, logger, retry } from "@trigger.dev/sdk/v3";
export const retryOnThrow = task({
id: "retry-on-throw",
run: async (payload: any) => {
//Will retry up to 3 times. If it fails 3 times it will throw.
const result = await retry.onThrow(
async ({ attempt }) => {
//throw on purpose the first 2 times, obviously this is a contrived example
if (attempt < 3) throw new Error("failed");
//...
return {
foo: "bar",
};
},
{ maxAttempts: 3, randomize: false }
);
//this will log out after 3 attempts of retry.onThrow
logger.info("Result", { result });
},
});
If all of the attempts with retry.onThrow
fail, an error will be thrown. You can catch this or
let it cause a retry of the entire task.
retry.fetch()
You can use fetch
, axios
, or any other library in your code.
But we do provide a convenient function to perform HTTP requests with conditional retrying based on the response:
import { task, logger, retry } from "@trigger.dev/sdk/v3";
export const taskWithFetchRetries = task({
id: "task-with-fetch-retries",
run: async ({ payload, ctx }) => {
//if the Response is a 429 (too many requests), it will retry using the data from the response. A lot of good APIs send these headers.
const headersResponse = await retry.fetch("http://my.host/test-headers", {
retry: {
byStatus: {
"429": {
strategy: "headers",
limitHeader: "x-ratelimit-limit",
remainingHeader: "x-ratelimit-remaining",
resetHeader: "x-ratelimit-reset",
resetFormat: "unix_timestamp_in_ms",
},
},
},
});
const json = await headersResponse.json();
logger.info("Fetched headers response", { json });
//if the Response is a 500-599 (issue with the server you're calling), it will retry up to 10 times with exponential backoff
const backoffResponse = await retry.fetch("http://my.host/test-backoff", {
retry: {
byStatus: {
"500-599": {
strategy: "backoff",
maxAttempts: 10,
factor: 2,
minTimeoutInMs: 1_000,
maxTimeoutInMs: 30_000,
randomize: false,
},
},
},
});
const json2 = await backoffResponse.json();
logger.info("Fetched backoff response", { json2 });
//You can additionally specify a timeout. In this case if the response takes longer than 1 second, it will retry up to 5 times with exponential backoff
const timeoutResponse = await retry.fetch("https://httpbin.org/delay/2", {
timeoutInMs: 1000,
retry: {
timeout: {
maxAttempts: 5,
factor: 1.8,
minTimeoutInMs: 500,
maxTimeoutInMs: 30_000,
randomize: false,
},
},
});
const json3 = await timeoutResponse.json();
logger.info("Fetched timeout response", { json3 });
return {
result: "success",
payload,
json,
json2,
json3,
};
},
});
If all of the attempts with retry.fetch
fail, an error will be thrown. You can catch this or let
it cause a retry of the entire task.
Advanced error handling and retrying
We provide a handleError
callback on the task and in your trigger.config
file. This gets called when an uncaught error is thrown in your task.
You can
- Inspect the error, log it, and return a different error if you’d like.
- Modify the retrying behavior based on the error, payload, context, etc.
If you don’t return anything from the function it will use the settings on the task (or inherited from the config). So you only need to use this to override things.
OpenAI error handling example
OpenAI calls can fail for a lot of reasons and the ideal retry behavior is different for each.
In this complicated example:
- We skip retrying if there’s no Response status.
- We skip retrying if you’ve run out of credits.
- If there are no Response headers we let the normal retrying logic handle it (return undefined).
- If we’ve run out of requests or tokens we retry at the time specified in the headers.
import { task } from "@trigger.dev/sdk/v3";
export const openaiTask = task({
id: "openai-task",
retry: {
maxAttempts: 1,
},
run: async (payload: { prompt: string }) => {
const chatCompletion = await openai.chat.completions.create({
messages: [{ role: "user", content: payload.prompt }],
model: "gpt-3.5-turbo",
});
return chatCompletion.choices[0].message.content;
},
handleError: async (payload, error, { ctx, retryAt }) => {
if (error instanceof OpenAI.APIError) {
if (!error.status) {
return {
skipRetrying: true,
};
}
if (error.status === 429 && error.type === "insufficient_quota") {
return {
skipRetrying: true,
};
}
if (!error.headers) {
//returning undefined means the normal retrying logic will be used
return;
}
const remainingRequests = error.headers["x-ratelimit-remaining-requests"];
const requestResets = error.headers["x-ratelimit-reset-requests"];
if (typeof remainingRequests === "string" && Number(remainingRequests) === 0) {
return {
retryAt: calculateISO8601DurationOpenAIVariantResetAt(requestResets),
};
}
const remainingTokens = error.headers["x-ratelimit-remaining-tokens"];
const tokensResets = error.headers["x-ratelimit-reset-tokens"];
if (typeof remainingTokens === "string" && Number(remainingTokens) === 0) {
return {
retryAt: calculateISO8601DurationOpenAIVariantResetAt(tokensResets),
};
}
}
},
});
Preventing retries
Using AbortTaskRunError
You can prevent retries by throwing an AbortTaskRunError
. This will fail the task attempt and disable retrying.
import { task, AbortTaskRunError } from "@trigger.dev/sdk/v3";
export const openaiTask = task({
id: "openai-task",
run: async (payload: { prompt: string }) => {
//if this fails, it will throw an error and stop retrying
const chatCompletion = await openai.chat.completions.create({
messages: [{ role: "user", content: payload.prompt }],
model: "gpt-3.5-turbo",
});
if (chatCompletion.choices[0]?.message.content === undefined) {
// If OpenAI returns an empty response, abort retrying
throw new AbortTaskRunError("OpenAI call failed");
}
return chatCompletion.choices[0].message.content;
},
});
Using try/catch
Sometimes you want to catch an error and don’t want to retry the task. You can use try/catch as you normally would. In this example we fallback to using Replicate if OpenAI fails.
import { task } from "@trigger.dev/sdk/v3";
export const openaiTask = task({
id: "openai-task",
run: async (payload: { prompt: string }) => {
try {
//if this fails, it will throw an error and retry
const chatCompletion = await openai.chat.completions.create({
messages: [{ role: "user", content: payload.prompt }],
model: "gpt-3.5-turbo",
});
if (chatCompletion.choices[0]?.message.content === undefined) {
//sometimes OpenAI returns an empty response, let's retry by throwing an error
throw new Error("OpenAI call failed");
}
return chatCompletion.choices[0].message.content;
} catch (error) {
//use Replicate if OpenAI fails
const prediction = await replicate.run(
"meta/llama-2-70b-chat:02e509c789964a7ea8736978a43525956ef40397be9033abf9fd2badfe68c9e3",
{
input: {
prompt: payload.prompt,
max_new_tokens: 250,
},
}
);
if (prediction.output === undefined) {
//retry if Replicate fails
throw new Error("Replicate call failed");
}
return prediction.output;
}
},
});
Was this page helpful?