Increased performance

We've redesigned the way we queue and execute job runs in order to increase the speed of job execution.

  • Fixed the cached task miss issue in the which should speed up resumed runs by A LOT
  • Allow setting graphile worker concurrency settings through env vars WORKER_CONCURRENCY and EXECUTION_WORKER_CONCURRENCY
  • Allow settings prisma pool settings through env vars DATABASE_CONNECTION_LIMIT and DATABASE_POOL_TIMEOUT
  • You can now selectively enable/disable the workers through WORKER_ENABLED=false and EXECUTION_WORKER_ENABLED=false. This means the image can be deployed as 2 or 3 separate services:
    • A WebApp service that serves the API and the Dashboard
    • A Worker service that runs tasks that have been added the standard worker
    • An Execution Worker service that only runs "run execution" tasks
  • Deprecated the JobOptions.queue options as we are no longer using that to control job concurrency. We'll add proper queue support in the future.