Self-host on NixOS
The board, the CI runs and the cache are one process, and odu’s flake ships a NixOS module for all three. This page sets up a server with the module, then lists what a host without the module sets instead.
Commands are written with the odu shorthand.
Add the module
Add the flake and import its module:
{
inputs.odu.url = "github:juspay/odu";
}
{ config, inputs, ... }:
{
imports = [ inputs.odu.nixosModules.default ];
# `cache.enableACME` is on by default, and Let's Encrypt wants both of these.
# Accepting the terms is your act, not the module's.
security.acme = {
acceptTerms = true;
defaults.email = "you@example.com";
};
# Any secret file works. agenix is the example.
age.secrets.odu-root-token = {
file = ./secrets/odu-root-token.age;
owner = "odu";
mode = "0400";
};
services.odu = {
enable = true;
cache.domain = "cache.example.com";
rootTokenFile = config.age.secrets.odu-root-token.path;
openFirewall = true;
};
}
What the module sets up
- The service listens on loopback and runs as the
odusystem user. - Its state is in
/var/lib/odu, at mode 0700, because the cache’s signing keys are there. - The root token arrives as a systemd credential and is exported into the unit’s environment. It is never written into the Nix store.
- nginx serves one virtual host for
cache.domain. It proxies/cache/,/_odu/v1/and/metrics, and returns 404 for everything else. cache.domainalso sets the origin a bucket is published under, so the URLodu cache useprints is one your clients can reach.
Without server mode, the board and
/mcphave no authentication, so the module keeps them off the public internet. Withservices.odu.githubset, nginx serves them too, behind GitHub sign-in.
The module requires the service to listen on loopback, because it must sit behind a proxy. Without sign-in nothing authenticates a request, and with sign-in the session cookie needs TLS.
Create the first bucket
Reach the board over a tailnet or an SSH forward to 127.0.0.1:18440, then create the bucket and mint a cache token:
export ODU_ROOT_TOKEN=...
odu surface cache bucket_create --input '{"rootToken":"'"$ODU_ROOT_TOKEN"'","name":"main","public":true,"retentionDays":30}' --json
odu surface cache token_mint --input '{"rootToken":"'"$ODU_ROOT_TOKEN"'","bucket":"main","scope":"write"}' --json
On each client machine, put the bucket’s address and the key bucket_create printed in the machine’s Nix configuration. On NixOS:
{
nix.settings = {
extra-substituters = [ "https://cache.example.com/cache/main" ];
extra-trusted-public-keys = [ "main-1:<public-key>" ];
};
}
On a machine with the Nix daemon, these lines belong in the machine’s configuration, where they work for every user. Use it has the other forms and who each one works for.
What a bucket keeps, and for how long, is on Keep and reclaim.
Keep objects on S3, or give a bucket a domain
A bucket can keep its objects on an S3 store (Cloudflare R2, AWS, Garage or versitygw) instead of the server’s disk. Declare the store in the module and name it at creation:
services.odu.cache.stores.r2 = {
endpoint = "https://<account>.r2.cloudflarestorage.com";
bucket = "odu-cache";
region = "auto";
credentialsFile = config.age.secrets.odu-r2.path; # AWS_ACCESS_KEY_ID=… / AWS_SECRET_ACCESS_KEY=…
};
Then add "store":"r2" to the bucket_create input. The credentials file is a systemd credential, like the root token. Keep a bucket on S3 has the rest.
A bucket can also have a domain of its own, served through a Cloudflare tunnel with no port opened. It needs a Cloudflare API token (services.odu.cloudflare.apiTokenFile), and cache.bucketDomains gives each name a virtual host and a certificate on a module host. A domain for a bucket has the steps.
Without the module
A container, or any host without systemd, runs the same one process from the environment the module sets, and needs nothing else.
| Variable | What it sets |
|---|---|
ODU_STATE_DIR |
where everything is kept: the run history, the cache, and the service’s own lock and socket. Set it when $HOME is read-only. |
ODU_WEB_ORIGIN |
the address the service listens on |
ODU_PUBLIC_ORIGIN |
the address a bucket’s URL is published under when a proxy stands in front. The module derives it from cache.domain and cache.enableACME. |
ODU_ROOT_TOKEN |
the root token, the secret that administers the cache |
ODU_OBJECT_STORES |
a JSON file of declared S3 stores, the shape the module writes. See Keep a bucket on S3. |
ODU_WEB_ALLOWED_ORIGINS |
browser origins the service accepts besides its own. The module’s allowedOrigins. |
ODU_CLOUDFLARE_API_TOKEN_FILE |
a file holding the Cloudflare API token. The module’s cloudflare.apiTokenFile. |
ODU_CACHE_STORE |
the old Nix store, only while upgrading from a store-backed cache |
Sign-in is set with ODU_GITHUB_*: see Server mode. Where the files go, how big they get and what to back up is on Operate the cache.
Upgrade from a store-backed cache
An older cache kept its paths in a Nix store. The service moves them into its object store in the background, and keeps serving while it does.
- On a module host, leave
ODU_CACHE_STOREunset: the old paths are in the host’s/nix/store. In a container, pointODU_CACHE_STOREat the old chroot store. - Start the service. It serves at once, and runs the migration and the
CA:pass in the background. Watchcache.migration.*andcache.self_ca.*. - Wait for
cache.migration.complete. - On a module host, delete the abandoned roots under
<state>/cache/gcroots, then runnix-collect-garbage. In a container, delete the old store directory. Never delete/nix/storeitself.
A path the old store lacks has no object, and returns 404 until it is pushed again. Each is named by a cache.migration.object_failed line, and cache.migration.incomplete carries the counts. Operate the cache describes the migration.