S3 / Blob Storage

Langfuse uses S3 or another S3-compatible blob storage (referred to as S3 going forward) to store raw events, multi-modal inputs, batch expors, and other files. You can use a managed service on AWS, or GCP, or host it yourself using MinIO. We use it as a scalable and durable storage solution for large files with strong read-after-write guarantees. This guide covers how to configure S3 within Langfuse and how to connect your own S3-compatible storage.

Configuration

Langfuse has multiple use-cases for S3 and allows you to configure them individually. That way, you can use separate buckets for each case, or combine information in a single bucket using prefixes.

Mandatory Configuration

Langfuse needs an S3 bucket to upload raw event information. The following environment variables are mandatory for every deployment. They need to be provided for the Langfuse Web and Langfuse Worker containers.

Optional Configuration

Langfuse also uses S3 for batch exports and for multi-modal tracing. Those use-cases are opt-in and can be configured separately. Use the following information to enable them.

Multi-Modal Tracing

Batch Exports

Langfuse allows to export table data via batch exports. We upload intermediate results to S3 and provide a presigned URL for users to download their exports in CSV or JSON format. To configure batch exports in your environment, configure the following environment variables:

Deployment Options

This section covers different deployment options and provides example environment variables. We will focus on the EVENT_UPLOAD case, as the other cases are similar.

Amazon S3

Amazon S3 is a globally available object storage. Langfuse uses the AWS SDK internally to connect to blob storages, as most providers provide an S3-compatible interface.

If Langfuse is running on an AWS instance, we recommend to use an IAM role on the Langfuse container to access S3. Otherwise, create an IAM user and generate an Access Key pair for Langfuse. Ensure that both entities have the necessary permissions to access the bucket:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": ["s3:PutObject", "s3:ListBucket", "s3:GetObject"],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::<my-bucket-name>/*",
        "arn:aws:s3:::<my-bucket-name>"
      ],
      "Sid": "EventBucketAccess"
    }
  ]
}

Example Configuration

Set the following environment variables if you authenticate using an IAM role:

LANGFUSE_S3_EVENT_UPLOAD_BUCKET=my-bucket-name

If you authenticate using an Access Key pair:

LANGFUSE_S3_EVENT_UPLOAD_BUCKET=my-bucket-name
LANGFUSE_S3_EVENT_UPLOAD_REGION=my-bucket-region
LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

MinIO

MinIO is an open-source object storage server that is compatible with the S3 API. It is a popular choice for on-premise deployments and local development. Langfuse uses it for local development and as a default in our Docker Compose and Kubernetes (Helm) deployment options.

Example Configuration

Start a local MinIO container with Docker using:

docker run --name minio \
  -p 9000:9000 \
  -p 9001:9001 \
  -e MINIO_ROOT_USER=minio \
  -e MINIO_ROOT_PASSWORD=miniosecret \
  minio/minio server /data --console-address ":9001"

Navigate to http://localhost:9001 to access the MinIO console and create a bucket named langfuse. Now, you can start Langfuse using the following environment variables:

LANGFUSE_S3_EVENT_UPLOAD_BUCKET=langfuse
LANGFUSE_S3_EVENT_UPLOAD_REGION=us-east-1
LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID=minio
LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY=miniosecret
LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT=http://minio:9000
LANGFUSE_S3_EVENT_UPLOAD_FORCE_PATH_STYLE=true
LANGFUSE_S3_EVENT_UPLOAD_PREFIX=events/

This example setup uses an ephemeral volume, i.e. on restarts MinIO will discard all event data. Please follow the MinIO documentation or use a cloud provider managed blob store for persistent data storage.

Cloudflare R2

Cloudflare R2 is globally available, S3 compatible object storage by Cloudflare. Create a new bucket within the Cloudflare UI and generate an Access Key pair. Ensure that the Access Key pair has the necessary permissions to access the bucket.

Example Configuration

Set the following environment variables to connect Langfuse with your Cloudflare R2 bucket:

LANGFUSE_S3_EVENT_UPLOAD_BUCKET=my-bucket-name
LANGFUSE_S3_EVENT_UPLOAD_REGION=auto
LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID=<access-key-id>
LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY=<secret-access-key>
LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT=https://${ACCOUNT_ID}.r2.cloudflarestorage.com

Google Cloud Storage

Google Cloud Storage is a globally available object storage by Google Cloud. It offers S3-compatibility through its interoperability interface. To get started, create a new bucket within Google Cloud Storage. Navigate to Settings > Interoperability to create a service account HMAC key. Ensure that the HMAC key has the necessary permissions to access the bucket.

Example Configuration

Set the following environment variables to connect Langfuse with your Google Cloud Storage bucket:

LANGFUSE_S3_EVENT_UPLOAD_BUCKET=my-bucket-name
LANGFUSE_S3_EVENT_UPLOAD_REGION=auto
LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID=<HMAC Access Key>
LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY=<HMAC Secret Key>
LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT=https://storage.googleapis.com
LANGFUSE_S3_EVENT_UPLOAD_FORCE_PATH_STYLE=true
LANGFUSE_S3_EVENT_UPLOAD_PREFIX=events/

Azure Blob Storage (Beta)

Azure Blob Storage is a globally available object storage by Microsoft Azure. It does not offer an S3-compatible API and requires slightly different configurations. We implemented a dedicated storage interface for experimental Azure Blob Storage support.

Example Configuration

You will need an Azure Blob Storage container and a static account key with the necessary permissions.

Set the following environment variables to connect Langfuse with your Azure Blob Storage container. This example uses sample credentials from Azurite.

# Special flag to enable the Azure Blob Storage interface
LANGFUSE_USE_AZURE_BLOB=true
 
LANGFUSE_S3_EVENT_UPLOAD_BUCKET=langfuse # Container name - If it does not exists, Langfuse will attempt to create it
LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID=devstoreaccount1 # ABS Account
LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw== # ABS Account Key
LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT=http://localhost:10000/devstoreaccount1 # URL, e.g. `https://${account}.blob.core.windows.net`

Other Providers

Langfuse supports any S3-compatible storage provider. Please refer to the provider’s documentation on how to create a bucket and generate Access Key pairs. Ensure that the Access Key pair has the necessary permissions to access the bucket. If you believe that other providers should be documented here, please open an issue or a pull request to contribute to this documentation.

Backups

Langfuse does not manage backups for S3. We recommend that you create a backup strategy for your buckets. Those may involve versioning, cross-region replication, or regular exports to another storage provider.