Keep a bucket on S3
A bucket keeps its objects in a store: local, the server’s own disk, or an S3-compatible bucket you declare, such as Cloudflare R2, AWS S3, Garage, versitygw or MinIO. You choose the store when you create a bucket, and can move the bucket later.
Commands are written with the odu shorthand.
Compare the stores
Every store holds the whole protocol: the NAR, the narinfo beside it, and the bucket’s nix-cache-info. What differs is who hands a reader the NAR, odu or the store: see Reads leave odu.
local |
an S3 store | |
|---|---|---|
| Declared | never: always there | services.odu.cache.stores.<name>, or $ODU_OBJECT_STORES |
| Objects at | <state>/cache/objects/<bucket>/nar/ |
<s3-bucket>/<bucket>/nar/ |
| A put that finds the key | refused, first kept wins (link) |
refused, first kept wins (If-None-Match: *) |
FileHash/FileSize kept |
beside the object (objects/.facts/) |
as object metadata (x-amz-meta-odu-*) |
| A large upload | streamed to a temp file | multipart, 8 MiB parts, four at a time, bounded memory |
| A read odu does itself (verifying, copying, collecting) | the file, from any offset | ranged GETs, 1 MiB at a time with the next one prefetched; no HEAD first |
| A name the world may read it at | never | publicUrl, an R2 custom domain bound to the bucket |
| A NAR a client pushes | odu takes the bytes into staging | the client PUTs it to presigned URLs; odu reads it back, verifies it and copies it into place |
A put is put-if-absent within one service on every store. Between two services writing one S3 bucket’s prefix, it holds where the store honours If-None-Match on the write that finishes the object.
- AWS honours it on both
PutObjectandCompleteMultipartUpload. - R2 documents it on
PutObject. Measured in November 2026, it also honours it onCompleteMultipartUpload(412), but ignores it on the one-requestCopyObject. odu’s server-side copy is therefore a multipart copy, conditional at its completion.
Set up an R2 bucket
Either way, you end with four facts: the Account ID, a private bucket, and an Access Key ID and Secret Access Key that may read and write only that bucket.
From the Cloudflare dashboard
- R2 object storage → Create bucket, say
odu-cache. Leave public access off unless you mean to serve this store’s public buckets from a domain of their own (see A public URL for a store). With it off, every read goes through odu, which checks the bucket’s cache tokens. - R2 object storage → Account Details → API Tokens → Manage → Create Account API token. Permissions: Object Read & Write. Under the bucket scope, pick only
odu-cache. - Copy the Access Key ID and the Secret Access Key. The secret is shown once. The page also shows the endpoint,
https://<ACCOUNT_ID>.r2.cloudflarestorage.com.
From the cf CLI
The cf CLI runs as npx -y cf@latest, written cf below.
-
Create the bucket and look up the two permission groups:
cf auth login # approve in the browser export CLOUDFLARE_ACCOUNT_ID=<ACCOUNT_ID> # the account cf acts on cf r2 buckets create --name odu-cache # the ids of the two bucket-scoped permission groups (the name is URL-encoded): cf user tokens permission-groups list --name 'Workers%20R2%20Storage%20Bucket%20Item%20Write' cf user tokens permission-groups list --name 'Workers%20R2%20Storage%20Bucket%20Item%20Read' cf user tokens create --name odu-cache --policies @policy.json -
Before the last command, write
policy.jsonnaming the one bucket.defaultis the jurisdiction of a bucket created without one:[{ "effect": "allow", "resources": { "com.cloudflare.edge.r2.bucket.<ACCOUNT_ID>_default_odu-cache": "*" }, "permission_groups": [{ "id": "<Item Write group id>" }, { "id": "<Item Read group id>" }] }] -
The new token’s reply carries its
idand itsvalue, shown once. The Access Key ID is theid, and the Secret Access Key is the SHA-256 of thevalue, in hex, as Cloudflare documents:printf %s '<token value>' | sha256sum | cut -d' ' -f1
This makes a user token, because that is what an OAuth login may create: cf accounts tokens create is refused under cf auth login (9109 Unauthorized, 403). A user token stops working if that user leaves the account. An account token from the dashboard does not.
With keys made this way, R2 returned 200 for a list and a put, 412 for a second put of the same key with If-None-Match: *, and 204 for a delete: the put-if-absent odu relies on.
Declare the store
Write the credentials file on the server, readable by root only. An agenix or sops secret is the usual home:
# /run/secrets/odu-r2
AWS_ACCESS_KEY_ID=<Access Key ID>
AWS_SECRET_ACCESS_KEY=<Secret Access Key>
On NixOS, declare the store and create the bucket on it:
services.odu.cache.stores.r2 = {
endpoint = "https://<ACCOUNT_ID>.r2.cloudflarestorage.com";
bucket = "odu-cache";
region = "auto";
credentialsFile = "/run/secrets/odu-r2"; # AWS_ACCESS_KEY_ID=… / AWS_SECRET_ACCESS_KEY=…
};
odu surface cache bucket_create --input '{"rootToken":"'"$ODU_ROOT_TOKEN"'","name":"main","public":true,"retentionDays":30,"store":"r2"}' --json
- On the board, New bucket offers the declared stores under where its objects live, and a bucket’s page says where its objects are, as in
objects on r2. - The credentials file is a systemd credential: never in the Nix store, never in odu’s database, never logged. The key needs object read, write, delete and list.
- Without the module, point
$ODU_OBJECT_STORESat a JSON file of the same shape beforeodu webstarts:{"r2": {"kind": "s3", "endpoint": …, "bucket": …, "region": …, "credentialsFile": …, "publicUrl": …}}.
A NAR on an S3 store is a redirect to a presigned URL built from
endpoint. A loopback, VPC-only or tailnet-onlyendpointbreaks every NAR read on the client. odu logscache.store.endpoint.unreachableat start only for a loopback or link-local one.
What else to know about a store
- One S3 bucket can hold many odu buckets. Each odu bucket is a key prefix of its own name, and odu lists, sweeps and deletes nothing outside those prefixes, so the S3 bucket may hold other things.
- A store a bucket still names cannot be removed. The cache refuses to start, naming the bucket and the store, rather than return 404 for every path.
- A push aborts the multipart uploads it abandons. One whose service was killed mid-upload is aborted at the next start, which aborts the bucket’s incomplete uploads begun before it. A lifecycle rule (“abort incomplete multipart uploads” on R2 and AWS) is a second net.
A public URL for a store
A store may name where the world reads it:
services.odu.cache.stores.r2 = {
endpoint = "https://<ACCOUNT_ID>.r2.cloudflarestorage.com";
bucket = "odu-cache";
region = "auto";
publicUrl = "https://cache.example.com"; # the custom domain bound to odu-cache
credentialsFile = "/run/secrets/odu-r2";
};
publicUrl is an origin and nothing more, with no path, query or fragment, because odu appends /<bucket>/<key> to it. Anything else fails the rebuild. One S3 bucket still serves many odu buckets, each under its own name: https://cache.example.com/main/nix-cache-info.
- A public bucket on such a store is advertised there. Its substituter URL on its page, in
bucket_describeand inodu cache usebecomes<publicUrl>/<bucket>, and its page says reads served from cache.example.com rather than reads served by odu. - Reading it never touches odu, so the cache keeps serving while odu is down, redeployed or scaled to zero.
- The public name is for readers only. odu signs its own requests against
endpoint, andodu cache pushstill addresses/cache/<bucket>on the server. - A public bucket on such a store may still have a domain of its own, which odu honours for writes and the
missingquery. Readers are sent to the store’s URL, because that one answers when odu does not.
Private buckets on a public store
A private bucket may not live on a store with a publicUrl. The domain serves every prefix in the S3 bucket, so anyone who guessed <publicUrl>/<bucket>/<hash>.narinfo could read it.
cache_bucket_createandcache_bucket_migraterefusepublic: falseonto such a store.- A private bucket already there when you bound the domain is named at every start with
cache.bucket.exposed. Move it to a store with no public URL.
Bind the domain
odu does not bind the domain: you do. For R2, one command:
cf r2 buckets domains custom create <r2-bucket> --domain <host> --zone-id <zone>
odu checks your work at start. For each store with a publicUrl and a public bucket on it, it fetches <publicUrl>/<bucket>/nix-cache-info. Anything but a 200 with the expected body is one cache.store.public.unreachable {store, url, status} line, never a refused start, because a domain may still be propagating.
Move a bucket to another store
odu surface cache bucket_migrate --input '{"rootToken":"'"$ODU_ROOT_TOKEN"'","bucket":"main","store":"r2"}' --json
On the board, it is a bucket’s Move to another store panel: pick a store under where to move it, then click Start move. A service that declares only one store says so instead.
The bucket keeps serving from the old store the whole time:
- odu copies every object (the NARs, each narinfo,
nix-cache-info) to the new store, two at a time. The bytes pass through odu. - When the copy is complete, the bucket changes stores under the same lock a push takes, after a last pass for paths pushed during the move.
- odu deletes the old copies and purges their URLs at the edge.
bucket_migrate returns the bucket as soon as the move is recorded, and copies in the background. Its migration field is the progress, which the board follows live and bucket_list returns:
| Field | What it says |
|---|---|
to |
the store it is moving to. The store beside it is still where reads come from. |
state |
copying fills the new store, flipping is the moment the bucket changes stores, deleting lets the old copies go |
copied / total |
objects done, out of what the move found |
bytes |
bytes copied so far |
migration is null when nothing is moving. The panel says it in a sentence, such as “Copying to r2: 12 of 40 objects, 3.0 MiB.”
- A move resumes after a restart. An object already in the new store with the same length is skipped.
- It refuses a store this service does not declare, the store the bucket is on, a second move of the same bucket, and anyone who is not an admin.