Operate the cache
This page is for whoever runs the server: where the cache’s files are, how big they get, what to back up, and what each log line means. To set up a server, see Self-host on NixOS.
Where the files are
<state> is the first of these that applies. The run history lives there too.
| Where | <state> |
|---|---|
$ODU_STATE_DIR is set |
$ODU_STATE_DIR |
| Linux | $XDG_STATE_HOME/odu, which is ~/.local/state/odu by default |
| macOS | ~/Library/Application Support/odu |
| the NixOS module | /var/lib/odu |
<state>/cache/cache.sqlite metadata, bucket signing keys, token hashes, pins
<state>/cache/objects/<bucket>/ nar/<narHash>.<ext> (immutable), <hash>.narinfo,
nix-cache-info
<state>/cache/staging/ uploads between their NAR and their narinfo
- An upload waits in
staging/until its narinfo arrives. One that never gets it is swept after an hour, so a client killed between the two PUTs cannot leak a NAR. - On an S3 store, the objects are in the store’s bucket instead of
objects/. - Cache tokens are stored only as hashes, so a copy of the database cannot push. The signing keys are in it, so treat it as a secret.
The service’s own home
The service’s lock and control socket are not under <state> by default. Their directory is named for the service’s origin: odu-web at the default origin, and odu-web-<hash> at any other, so two services never share one.
- Without
$ODU_STATE_DIR, it is$HOME/.local/state/odu-web*, besideodurather than inside it. - It is never under
$XDG_STATE_HOME, which differs between a login shell, a bare ssh and a systemd unit. Two launches that computed two homes would split one service. - Set
$ODU_STATE_DIR, and the home moves inside it. A container with a read-only$HOMEwants that.
How much disk and memory
- Disk is the compressed objects, plus staging for the largest NAR being uploaded.
- Memory per NAR is constant, not in proportion to its size.
- A bucket created with
upstreamKeys: []also stores every upstream-signed closure it is pushed. Budget that disk on purpose.
What to back up
Back up the database and the objects: <state>/cache/cache.sqlite, objects/, and each S3 store’s bucket. The bytes cannot be rebuilt from anywhere else.
What the service does at start
The cache serves at once, then does its upkeep in the background:
- It sweeps objects that no path names, such as one left by a crash between upload and record, or by a collection that stopped between two deletes. The sweep knows all three kinds of key, so it never takes a narinfo for an orphan.
- It writes any narinfo object a path lacks, two at a time. An object already there with the right length is skipped, so a restart halfway costs a walk, not a rewrite. When the
nix-cache-infotext changes, such as after a move to a new public origin, it rewrites that object and purges its address at the edge. - It proves
CA:once for paths that have not been proved, with two workers, and keeps each verdict. A path that refers to itself is proved by reading its object from the store, and once proved, its narinfo in the store is rewritten with theCA:line. An upgrade that adds a way to prove such a path sets every one not proved back to unproved, so the first start after it reads each once. - It aborts S3 multipart uploads that a killed service left behind.
The one-time migration
A cache upgraded from the store-backed design has paths with no object. The steps are on Self-host on NixOS. What the service does:
- It copies each path from the old store (
$ODU_CACHE_STORE, or the host’s/nix/storewhen unset) into the object store, two at a time. - It pauses below 10 GiB free and checks again every minute, resuming without a restart.
- Until a path’s object lands, its NAR is served from the old store, logged as
source: transition. - A path the old store lacks returns 404 and is named once. A path whose object is missing or unreadable is retried at the next start, not on a request.
cache.migration.completesays every path has its object, and the old store can go. On a module host, delete the abandoned roots under<state>/cache/gcroots, then letnix-collect-garbagereclaim the paths. Do not delete/nix/store.
Log lines
A service in the foreground (odu web) prints these lines, and keeps its other logging quiet.
| Line | Fields | What it means |
|---|---|---|
cache.migration.start, .progress, .pause, .resume, .complete, .incomplete |
objects, NAR bytes, free space | The one-time migration. After incomplete, the paths it names have no object and no copy in the old store, and return 404 until pushed again. |
cache.migration.object_failed |
bucket, path, reason |
One path the migration could not move, such as one the old store does not have, or has with another NarHash. |
cache.self_ca.start, .progress, .complete |
paths, bytes, proved, refused, unreadable, skipped | Proves CA: for paths that refer to themselves: once per path, and again after an upgrade that adds a way to prove them. skipped: paths a push or a collection changed meanwhile. |
cache.nar.served |
path, bytes written, ms, source, cancelled |
A NAR served. source is object from the bucket’s store, transition from the old Nix store, or redirect for a 307 to an S3 store, always with bytes: 0. |
cache.nar.uploaded |
bucket, file, mode, parts, bytes |
A NAR upload completed, direct or staged, with the bytes odu measured itself. |
cache.narinfo.served |
bucket, hash, hit |
A narinfo asked of odu, and whether the bucket had it. |
cache.nar.unavailable |
path | No object, and the old store lacks it. Served as 404, logged once per path. |
cache.boot.orphans, cache.boot.ca |
counts | The sweep and the metadata CA: counts at start. |
cache.import.recorded, cache.request.refused, cache.object.put_failed |
— | A path recorded, a request refused, an object that could not be written. |
cache.nar.verify_failed |
bucket, path, file, mode, reason |
An upload odu would not vouch for, refused with 400. |
cache.gc |
evicted, kept, orphans, bytes, stopped |
One line per collection, scheduled or asked for. kept counts paths whose bytes a twin still names. stopped appears when the store refused a delete. |
cache.gc.purge |
per bucket | How many URLs a collection announced to the edge. |
cache.gc.failed |
— | A scheduled collection hit an error. The next one is still scheduled. |
cache.narinfo.backfill |
bucket, written, remaining |
The start-up pass that writes missing narinfo objects, one line per five hundred paths and one at the end. |
cache.narinfo.object_failed |
bucket, path, reason |
A narinfo object could not be written. |
cache.narinfo.purge |
bucket, urls |
A push rewrote a narinfo object, so its address is announced to the edge. A NAR never is: it is named by its own bytes. |
cache.store.public.unreachable |
store, url, status |
A store’s publicUrl did not answer nix-cache-info at start. odu keeps serving, but readers sent to that domain fail. |
cache.store.endpoint.unreachable |
store, endpoint, host |
A store holding a bucket has a loopback or link-local endpoint. Clients follow redirects to it, so every NAR read fails on the client. |
cache.bucket.exposed |
bucket, store, url |
A private bucket is on a store with a public URL, so anyone who knows its name can read it. New ones are refused; this names those already there. |
cache.bucket.migrate.start, .progress, .flipped, .done, .failed |
copied, total, bytes |
A move to another store. |
cache.bucket.migrate.purge |
— | The old store’s URLs a move asked the edge to drop. |
cache.bucket.deleted |
bucket, objects, bytes |
A bucket was deleted: the store objects removed under its prefix, and their size. It is the only record left of the bucket. |
cache.bucket.delete.purge |
bucket, urls |
The deleted bucket’s addresses, announced to the edge. |
cache.nar.verify_failed
The reason says why odu refused an upload:
reason |
What happened |
|---|---|
absent |
The narinfo names a NAR nothing uploaded. |
unreadable |
The bytes would not decompress, or would not read. |
nar_hash_mismatch |
The bytes hash to something else. |
unversioned_source, source_changed |
The version of the object odu read back could not be pinned, or was gone by the copy. |
short_read |
A direct upload’s read-back got fewer bytes than promised: a window returned less than its range, or a later window returned 412 or 416. |
short_read is not a verdict on the NAR. Push the path again.