Skip to main content

Multi-Provider Object Storage

The object storage system supports multiple S3-compatible providers (R2, S3, GCS, MinIO, etc.) using protocol prefixes. This enables migrating between providers without breaking existing runs.

How It Works

When data exceeds the configured threshold (TASK_PAYLOAD_OFFLOAD_THRESHOLD), it’s uploaded to object storage. The storage location is saved in the database with an optional protocol prefix:
  • With protocol: s3://run_abc/payload.json or r2://batch_123/item_0/payload.json
  • Without protocol (legacy): batch_123/item_0/payload.json (uses default provider)

Configuration

Default Provider (Backward Compatible)

The default provider is used for data without a protocol prefix:

Named Providers

Named providers are accessed via protocol-prefixed URIs. Configure them using OBJECT_STORE_{PROTOCOL}_* variables:

Default Protocol for New Uploads

Set OBJECT_STORE_DEFAULT_PROTOCOL to specify which provider to use for new uploads:

Migration Guide

To migrate from R2 to S3 without breaking existing runs:
1

Configure S3 provider

Add S3 credentials as a named provider:
Keep your existing OBJECT_STORE_* variables (R2) as the default provider.
2

Test the configuration

Restart the webapp and verify both providers work:
  • Old runs (no prefix) should still access R2
  • New runs with s3:// prefix should use S3
3

Switch to S3 for new uploads

Set the default protocol to use S3 for new uploads:
After this change:
  • New data uses s3:// prefix and goes to S3
  • Old data (no prefix) still uses R2
  • Data with explicit protocol uses the corresponding provider
4

Optionally decommission R2

Once all active runs using R2 data have completed (check your data retention policies), you can remove the R2 credentials. Keep OBJECT_STORE_DEFAULT_PROTOCOL=s3 to ensure new data continues using S3.