All docs pages ↓

Keep a bucket on S3

A bucket keeps its objects in a store: local, the server’s own disk, or an S3-compatible bucket you declare, such as Cloudflare R2, AWS S3, Garage, versitygw or MinIO. You choose the store when you create a bucket, and can move the bucket later.

Commands are written with the odu shorthand.

Compare the stores

Every store holds the whole protocol: the NAR, the narinfo beside it, and the bucket’s nix-cache-info. What differs is who hands a reader the NAR, odu or the store: see Reads leave odu.

local an S3 store
Declared never: always there services.odu.cache.stores.<name>, or $ODU_OBJECT_STORES
Objects at <state>/cache/objects/<bucket>/nar/ <s3-bucket>/<bucket>/nar/
A put that finds the key refused, first kept wins (link) refused, first kept wins (If-None-Match: *)
FileHash/FileSize kept beside the object (objects/.facts/) as object metadata (x-amz-meta-odu-*)
A large upload streamed to a temp file multipart, 8 MiB parts, four at a time, bounded memory
A read odu does itself (verifying, copying, collecting) the file, from any offset ranged GETs, 1 MiB at a time with the next one prefetched; no HEAD first
A name the world may read it at never publicUrl, an R2 custom domain bound to the bucket
A NAR a client pushes odu takes the bytes into staging the client PUTs it to presigned URLs; odu reads it back, verifies it and copies it into place

A put is put-if-absent within one service on every store. Between two services writing one S3 bucket’s prefix, it holds where the store honours If-None-Match on the write that finishes the object.

Set up an R2 bucket

Either way, you end with four facts: the Account ID, a private bucket, and an Access Key ID and Secret Access Key that may read and write only that bucket.

From the Cloudflare dashboard

  1. R2 object storage → Create bucket, say odu-cache. Leave public access off unless you mean to serve this store’s public buckets from a domain of their own (see A public URL for a store). With it off, every read goes through odu, which checks the bucket’s cache tokens.
  2. R2 object storage → Account Details → API Tokens → Manage → Create Account API token. Permissions: Object Read & Write. Under the bucket scope, pick only odu-cache.
  3. Copy the Access Key ID and the Secret Access Key. The secret is shown once. The page also shows the endpoint, https://<ACCOUNT_ID>.r2.cloudflarestorage.com.

From the cf CLI

The cf CLI runs as npx -y cf@latest, written cf below.

  1. Create the bucket and look up the two permission groups:

    cf auth login                                   # approve in the browser
    export CLOUDFLARE_ACCOUNT_ID=<ACCOUNT_ID>       # the account cf acts on
    cf r2 buckets create --name odu-cache
    # the ids of the two bucket-scoped permission groups (the name is URL-encoded):
    cf user tokens permission-groups list --name 'Workers%20R2%20Storage%20Bucket%20Item%20Write'
    cf user tokens permission-groups list --name 'Workers%20R2%20Storage%20Bucket%20Item%20Read'
    cf user tokens create --name odu-cache --policies @policy.json
  2. Before the last command, write policy.json naming the one bucket. default is the jurisdiction of a bucket created without one:

    [{
      "effect": "allow",
      "resources": { "com.cloudflare.edge.r2.bucket.<ACCOUNT_ID>_default_odu-cache": "*" },
      "permission_groups": [{ "id": "<Item Write group id>" }, { "id": "<Item Read group id>" }]
    }]
  3. The new token’s reply carries its id and its value, shown once. The Access Key ID is the id, and the Secret Access Key is the SHA-256 of the value, in hex, as Cloudflare documents:

    printf %s '<token value>' | sha256sum | cut -d' ' -f1

This makes a user token, because that is what an OAuth login may create: cf accounts tokens create is refused under cf auth login (9109 Unauthorized, 403). A user token stops working if that user leaves the account. An account token from the dashboard does not.

With keys made this way, R2 returned 200 for a list and a put, 412 for a second put of the same key with If-None-Match: *, and 204 for a delete: the put-if-absent odu relies on.

Declare the store

Write the credentials file on the server, readable by root only. An agenix or sops secret is the usual home:

# /run/secrets/odu-r2
AWS_ACCESS_KEY_ID=<Access Key ID>
AWS_SECRET_ACCESS_KEY=<Secret Access Key>

On NixOS, declare the store and create the bucket on it:

services.odu.cache.stores.r2 = {
  endpoint = "https://<ACCOUNT_ID>.r2.cloudflarestorage.com";
  bucket = "odu-cache";
  region = "auto";
  credentialsFile = "/run/secrets/odu-r2"; # AWS_ACCESS_KEY_ID=… / AWS_SECRET_ACCESS_KEY=…
};
odu surface cache bucket_create --input '{"rootToken":"'"$ODU_ROOT_TOKEN"'","name":"main","public":true,"retentionDays":30,"store":"r2"}' --json
  • On the board, New bucket offers the declared stores under where its objects live, and a bucket’s page says where its objects are, as in objects on r2.
  • The credentials file is a systemd credential: never in the Nix store, never in odu’s database, never logged. The key needs object read, write, delete and list.
  • Without the module, point $ODU_OBJECT_STORES at a JSON file of the same shape before odu web starts: {"r2": {"kind": "s3", "endpoint": …, "bucket": …, "region": …, "credentialsFile": …, "publicUrl": …}}.

A NAR on an S3 store is a redirect to a presigned URL built from endpoint. A loopback, VPC-only or tailnet-only endpoint breaks every NAR read on the client. odu logs cache.store.endpoint.unreachable at start only for a loopback or link-local one.

What else to know about a store

  • One S3 bucket can hold many odu buckets. Each odu bucket is a key prefix of its own name, and odu lists, sweeps and deletes nothing outside those prefixes, so the S3 bucket may hold other things.
  • A store a bucket still names cannot be removed. The cache refuses to start, naming the bucket and the store, rather than return 404 for every path.
  • A push aborts the multipart uploads it abandons. One whose service was killed mid-upload is aborted at the next start, which aborts the bucket’s incomplete uploads begun before it. A lifecycle rule (“abort incomplete multipart uploads” on R2 and AWS) is a second net.

A public URL for a store

A store may name where the world reads it:

services.odu.cache.stores.r2 = {
  endpoint = "https://<ACCOUNT_ID>.r2.cloudflarestorage.com";
  bucket = "odu-cache";
  region = "auto";
  publicUrl = "https://cache.example.com";   # the custom domain bound to odu-cache
  credentialsFile = "/run/secrets/odu-r2";
};

publicUrl is an origin and nothing more, with no path, query or fragment, because odu appends /<bucket>/<key> to it. Anything else fails the rebuild. One S3 bucket still serves many odu buckets, each under its own name: https://cache.example.com/main/nix-cache-info.

  • A public bucket on such a store is advertised there. Its substituter URL on its page, in bucket_describe and in odu cache use becomes <publicUrl>/<bucket>, and its page says reads served from cache.example.com rather than reads served by odu.
  • Reading it never touches odu, so the cache keeps serving while odu is down, redeployed or scaled to zero.
  • The public name is for readers only. odu signs its own requests against endpoint, and odu cache push still addresses /cache/<bucket> on the server.
  • A public bucket on such a store may still have a domain of its own, which odu honours for writes and the missing query. Readers are sent to the store’s URL, because that one answers when odu does not.

Private buckets on a public store

A private bucket may not live on a store with a publicUrl. The domain serves every prefix in the S3 bucket, so anyone who guessed <publicUrl>/<bucket>/<hash>.narinfo could read it.

  • cache_bucket_create and cache_bucket_migrate refuse public: false onto such a store.
  • A private bucket already there when you bound the domain is named at every start with cache.bucket.exposed. Move it to a store with no public URL.

Bind the domain

odu does not bind the domain: you do. For R2, one command:

cf r2 buckets domains custom create <r2-bucket> --domain <host> --zone-id <zone>

odu checks your work at start. For each store with a publicUrl and a public bucket on it, it fetches <publicUrl>/<bucket>/nix-cache-info. Anything but a 200 with the expected body is one cache.store.public.unreachable {store, url, status} line, never a refused start, because a domain may still be propagating.

Move a bucket to another store

odu surface cache bucket_migrate --input '{"rootToken":"'"$ODU_ROOT_TOKEN"'","bucket":"main","store":"r2"}' --json

On the board, it is a bucket’s Move to another store panel: pick a store under where to move it, then click Start move. A service that declares only one store says so instead.

The bucket keeps serving from the old store the whole time:

  1. odu copies every object (the NARs, each narinfo, nix-cache-info) to the new store, two at a time. The bytes pass through odu.
  2. When the copy is complete, the bucket changes stores under the same lock a push takes, after a last pass for paths pushed during the move.
  3. odu deletes the old copies and purges their URLs at the edge.

bucket_migrate returns the bucket as soon as the move is recorded, and copies in the background. Its migration field is the progress, which the board follows live and bucket_list returns:

Field What it says
to the store it is moving to. The store beside it is still where reads come from.
state copying fills the new store, flipping is the moment the bucket changes stores, deleting lets the old copies go
copied / total objects done, out of what the move found
bytes bytes copied so far

migration is null when nothing is moving. The panel says it in a sentence, such as “Copying to r2: 12 of 40 objects, 3.0 MiB.”

  • A move resumes after a restart. An object already in the new store with the same length is skipped.
  • It refuses a store this service does not declare, the store the bucket is on, a second move of the same bucket, and anyone who is not an admin.