Running a host

The production server: hardening, reverse proxies, limits and recovery.

crisptastic host is the production server. One process serves many sites, choosing the site by the request's domain. It runs behind a reverse proxy that owns public TLS.

Audience and status. This chapter is for whoever operates the server. The single-node setup is used in production by the project's own website. Clustering, Kismet ingress and sleeping are preview; see Clustering. Production certification of the whole system is still open.

What the host does

  • Listens for plain HTTP/1.1 on --bind. There is no TLS, HTTP/2 or WebSocket support on this port.
  • Loads Capsules at start-up (from paths or digests on the command line) and later when told to by crisptastic notify on the management port.
  • Routes by exact domain. A request for a domain no site claims receives 421 Misdirected Request. It never falls back to another site.
  • Keeps decrypted secrets only in locked memory.

Secrets in memory and the routing table are lost on restart; a restart must re-activate sites, either with start-up arguments or by notify. A site's visitors are not: its session files live in <host-root>/sessions/<site>/, named for the site's first domain, never inside a generation's unpacked directory, and a [login] site's signed-in sessions are shared by its generations and kept, encrypted to the host's age identity, in login-*.age beside them. So neither a redeploy nor a restart signs anyone out of a [login] site. The exception is the experimental Wasm [arbax] capability: its tokens stay in that generation's memory, so a redeploy or a restart signs its users out. A Capsule's sessions setting is ignored on a host.

Minimal production command

crisptastic host \
  --identity /etc/crisptastic/server.agekey \
  --trusted-signer /etc/crisptastic/release.pub \
  --host-root /var/lib/crisptastic \
  --store-dir /var/lib/crisptastic/capsule-store \
  --bind 127.0.0.1:6174 \
  --proxy-mode x-forwarded --trusted-cidr 127.0.0.1/32 --trusted-hops 1 \
  --management-bind 127.0.0.1:7443 \
  --management-cert /etc/crisptastic/management.crt \
  --management-key /etc/crisptastic/management.key \
  --management-client-ca /etc/crisptastic/client-ca.crt \
  --digest <sha256-of-a-capsule-to-start-with>

Notes on the options:

OptionMeaning
--identity FILEThe host's age private key. First non-comment line must be AGE-SECRET-KEY-1..., which is what age-keygen -o writes.
--trusted-signer FILEPublic signing key; repeat for each trusted signer. At least one is required.
--host-root DIRAbsolute directory the host owns. Created if missing.
--deployments-root DIRWhere sites are unpacked. Default <host-root>/deployments. Must live inside the host root.
--bind ADDRPublic listener. Default 127.0.0.1:6174; if PORT is set and --bind is not, 0.0.0.0:$PORT. Must be a numeric ip:port.
--store-dir DIR / --s3Where to fetch Capsules by digest.
CAPSULE..., --digest SHA256...Capsules to activate at start-up, as file paths or as digests found in the store. At least one of these, --management-bind or --fabric-config is required.
--management-bind + --management-cert/--management-key/--management-client-caThe mTLS listener for notify and deploy --peer.
--locked-secret-memory-bytes NMemory reserved for secrets (default 64 MiB).

Run crisptastic help host for the full list.

Local trial without hardening

On a laptop you can skip the hardening checks:

crisptastic host my-site.age \
  --identity ./server.agekey --trusted-signer ./release.pub \
  --development-process-profile --bind 127.0.0.1:6174 \
  --locked-secret-memory-bytes 65536

The default of 64 MiB of locked memory is more than most desktop limits allow (ulimit -l), and the host refuses to start without it, so a trial asks for less.

--development-process-profile prints a "not production conformant" warning. It cannot be combined with --host-root, --management-bind, a cluster configuration or Kismet options, and it turns off the HTTPS-metadata requirement below, so plain http://localhost works.

The production process profile

By default the host refuses to start unless it is configured safely. These checks run, in order, before any Capsule is read or socket opened:

  1. Every configured path is absolute.
  2. The effective user is not root.
  3. The host sets its own umask to 077.
  4. The host root is a real directory owned by the service user and not writable by group or others. The deployments root is inside it.
  5. The public, management and cluster listeners are numeric and do not overlap.
  6. Inherited file descriptors are closed.
  7. An exclusive lock is taken on <host-root>/.crisptastic-host.lock. A second host cannot use the same host root.
  8. Enough memory can be locked for the secret budget. This needs RLIMIT_MEMLOCK at least equal to --locked-secret-memory-bytes (LimitMEMLOCK=64M in systemd).

Core dumps are disabled. The practical target is Linux; the profile requires a Unix system.

Reverse proxy requirements

The host trusts the proxy to describe the original request, and it rejects every tenant request unless that description arrives, over a trusted connection, with scheme https:

  • Pass --proxy-mode x-forwarded (or forwarded), at least one --trusted-cidr containing the proxy's address, and --trusted-hops N (default 1).
  • With x-forwarded, the proxy must send exactly one each of X-Forwarded-For, X-Forwarded-Host and X-Forwarded-Proto: https, and no Forwarded header. X-Forwarded-Port is ignored.
  • With forwarded, exactly one RFC 7239 Forwarded header with for, host and proto in each element, and no X-Forwarded-* headers.
  • Requests from an address outside --trusted-cidr have their forwarding headers ignored and are treated as plain HTTP, which is rejected.

A common mistake: the default --proxy-mode none under the production profile answers every request except /health with 400 Production public origin must use trusted HTTPS metadata. Some older examples omit the proxy flags and look like they should work. They do not.

The matching nginx essentials:

location / {
    proxy_pass http://127.0.0.1:6174;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Proto https;
    client_max_body_size 1m;
    proxy_read_timeout 35s;
}

Other points:

  • Use a catch-all virtual host. Adding a domain needs DNS and TLS at the proxy but no proxy edit.
  • The domain match is case-insensitive, ignores a trailing dot and the default port. A non-default port is kept, so example.com:8443 does not match example.com.
  • Do not expose the management port to the internet. The reference deployment reaches it through an SSH tunnel.

Fixed limits

The public listener has built-in limits that are not configurable: 1,024 connections (100 per client address), 256 concurrent requests, 1,000 requests per connection, 1 MiB request body, 8 KiB request line, 32 KiB and 100 headers, 10-second header timeout, 15-second body idle timeout and a 30-second handler deadline. SIGTERM or SIGINT drains for up to 30 seconds. There is no per-site CPU, memory or request quota, and no way to forcibly cancel a runaway page, so a host is not a safe place for untrusted tenants.

Health and status

On the public listener (answered before site routing, so they shadow a site route at the same path):

RequestResponse
GET /health200 {"status":"live"} whenever the process is up
GET /__crisptastic/ready/<domain>200 if that exact domain is ready to serve, otherwise 503

On the management listener (mTLS): GET /live, GET /ready, GET /status and POST /v1/deployments/activate. There is no /metrics, no log endpoint and no audit log. Logs are plain lines on standard output and standard error, such as route example.com -> <deployment-id> at start-up; capture them with your service manager.

Use /__crisptastic/ready/<domain> for per-site readiness checks.

Storage options

  • --store-dir DIR keeps Capsules in DIR/objects/capsules/sha256/<aa>/<digest>.age.
  • --s3 takes its configuration only from the environment (see Deploying). The S3 client is minimal: static credentials, whole objects in memory, no listing. Hosts address Capsules by digest only, so nothing can ask "what is the latest".

Sites with an embedded store

A site whose crisptastic.toml declares [data] gets a Residiuum store that the host, not the deployment, owns: <host-root>/stores/<application_id>/, created at the site's first activation and reused by every later generation. The rules:

  • One site, one store. The store is named by [data] application_id. The site that first activates it owns it; a different site (one whose domains do not overlap) claiming the same id is refused with data.application_id_owned. Sites never see each other's heaps.
  • Created once, with a key the Capsule carries. A store is sealed. The first activation needs its unseal key, named by [data] unseal_secret as one of the Capsule's secure variables ([env_from_process]); it is read into locked memory, used to create the store, and never written to disk. Every later activation and every restart opens the store sealed, without the key. Activating a site whose store does not exist yet and which names no unseal_secret fails with data.store_missing.
  • One writer. A new generation of the site adopts the store the running generation holds open; when the last generation is retired the store is closed, and the next activation reopens it sealed. A new generation may add record or content collections, which are created in the open store; one it drops stays in the store, unserved.
  • Its own memory. [data] memory_budget is that store's budget, counted within the host's MemoryMax. Size the host for the sum of its stores. The budget is fixed while the store is open: a generation that changes it is refused with data.memory_budget_changed, and takes effect after a host restart.
  • Single host only. A Fabric cluster refuses [data] with data.hosted_fabric_unsupported; the store would live on one member while the next request may be served by another. [data] root is used by dev and run (default .crisptastic/residiuum beside the site) and ignored by a host, so a hosted site may leave it out.

Backup and recovery

WhatSurvives a restart?Back it up?
Encrypted Capsules in the storeYesYes. This is your durable record, and it holds your secrets, encrypted
Embedded stores under <host-root>/stores/YesYes. A site's records live only here; stop the host or use Residiuum's dump before copying
The host age identityYesYes. Without it every stored Capsule is unreadable
Signing private key(on the deployer)Yes, offline
Trusted signer keys, management PKIYesYes
Unpacked public filesYes (not cleaned up)No, regenerable
Session files and kept login sessions under <host-root>/sessions/YesOptional; losing them only signs people out
Secrets in memory, routing tableNoNot applicable

To restore a host, start it with the identity, signers, store and the digests of the Capsules to activate. There is no export or restore command. After total loss the supported path is to redeploy your site with the original environment.

Upgrading

Replace the binary and restart. A Capsule whose schema the new binary does not understand is rejected, never guessed at. [login] visitors stay signed in across the restart, and a restored session refreshes its Arbax token on first use; Wasm [arbax] users are signed out.

What is not here

These appear in the older specifications but are not implemented: a host.toml file and host init, a challenge-authenticated deployment protocol and keys authorize, an operations listener with /metrics, rollback and retire commands, per-tenant audit sinks, and hosted telemetry. See Known gaps.