Running a host
The production server: hardening, reverse proxies, limits and recovery.
crisptastic host is the production server. One process serves many sites,
choosing the site by the request's domain. It runs behind a reverse proxy that
owns public TLS.
Audience and status. This chapter is for whoever operates the server. The single-node setup is used in production by the project's own website. Clustering, Kismet ingress and sleeping are preview; see Clustering. Production certification of the whole system is still open.
What the host does
- Listens for plain HTTP/1.1 on
--bind. There is no TLS, HTTP/2 or WebSocket support on this port. - Loads Capsules at start-up (from paths or digests on the command line) and
later when told to by
crisptastic notifyon the management port. - Routes by exact domain. A request for a domain no site claims receives
421 Misdirected Request. It never falls back to another site. - Keeps decrypted secrets only in locked memory.
Secrets in memory and the routing table are lost on restart; a restart must
re-activate sites, either with start-up arguments or by notify. A site's
visitors are not: its session files live in <host-root>/sessions/<site>/,
named for the site's first domain, never inside a generation's unpacked
directory, and a [login] site's signed-in sessions are shared by its
generations and kept, encrypted to the host's age identity, in
login-*.age beside them. So neither a redeploy nor a restart signs anyone
out of a [login] site. The exception is the experimental Wasm [arbax]
capability: its tokens stay in that generation's memory, so a redeploy or a
restart signs its users out. A Capsule's sessions setting is ignored on a
host.
Minimal production command
crisptastic host \
--identity /etc/crisptastic/server.agekey \
--trusted-signer /etc/crisptastic/release.pub \
--host-root /var/lib/crisptastic \
--store-dir /var/lib/crisptastic/capsule-store \
--bind 127.0.0.1:6174 \
--proxy-mode x-forwarded --trusted-cidr 127.0.0.1/32 --trusted-hops 1 \
--management-bind 127.0.0.1:7443 \
--management-cert /etc/crisptastic/management.crt \
--management-key /etc/crisptastic/management.key \
--management-client-ca /etc/crisptastic/client-ca.crt \
--digest <sha256-of-a-capsule-to-start-with>
Notes on the options:
| Option | Meaning |
|---|---|
--identity FILE | The host's age private key. First non-comment line must be AGE-SECRET-KEY-1..., which is what age-keygen -o writes. |
--trusted-signer FILE | Public signing key; repeat for each trusted signer. At least one is required. |
--host-root DIR | Absolute directory the host owns. Created if missing. |
--deployments-root DIR | Where sites are unpacked. Default <host-root>/deployments. Must live inside the host root. |
--bind ADDR | Public listener. Default 127.0.0.1:6174; if PORT is set and --bind is not, 0.0.0.0:$PORT. Must be a numeric ip:port. |
--store-dir DIR / --s3 | Where to fetch Capsules by digest. |
CAPSULE..., --digest SHA256... | Capsules to activate at start-up, as file paths or as digests found in the store. At least one of these, --management-bind or --fabric-config is required. |
--management-bind + --management-cert/--management-key/--management-client-ca | The mTLS listener for notify and deploy --peer. |
--locked-secret-memory-bytes N | Memory reserved for secrets (default 64 MiB). |
Run crisptastic help host for the full list.
Local trial without hardening
On a laptop you can skip the hardening checks:
crisptastic host my-site.age \
--identity ./server.agekey --trusted-signer ./release.pub \
--development-process-profile --bind 127.0.0.1:6174 \
--locked-secret-memory-bytes 65536
The default of 64 MiB of locked memory is more than most desktop limits allow
(ulimit -l), and the host refuses to start without it, so a trial asks for less.
--development-process-profile prints a "not production conformant" warning.
It cannot be combined with --host-root, --management-bind, a cluster
configuration or Kismet options, and it turns off the HTTPS-metadata
requirement below, so plain http://localhost works.
The production process profile
By default the host refuses to start unless it is configured safely. These checks run, in order, before any Capsule is read or socket opened:
- Every configured path is absolute.
- The effective user is not root.
- The host sets its own
umaskto077. - The host root is a real directory owned by the service user and not writable by group or others. The deployments root is inside it.
- The public, management and cluster listeners are numeric and do not overlap.
- Inherited file descriptors are closed.
- An exclusive lock is taken on
<host-root>/.crisptastic-host.lock. A second host cannot use the same host root. - Enough memory can be locked for the secret budget. This needs
RLIMIT_MEMLOCKat least equal to--locked-secret-memory-bytes(LimitMEMLOCK=64Min systemd).
Core dumps are disabled. The practical target is Linux; the profile requires a Unix system.
Reverse proxy requirements
The host trusts the proxy to describe the original request, and it rejects
every tenant request unless that description arrives, over a trusted
connection, with scheme https:
- Pass
--proxy-mode x-forwarded(orforwarded), at least one--trusted-cidrcontaining the proxy's address, and--trusted-hops N(default 1). - With
x-forwarded, the proxy must send exactly one each ofX-Forwarded-For,X-Forwarded-HostandX-Forwarded-Proto: https, and noForwardedheader.X-Forwarded-Portis ignored. - With
forwarded, exactly one RFC 7239Forwardedheader withfor,hostandprotoin each element, and noX-Forwarded-*headers. - Requests from an address outside
--trusted-cidrhave their forwarding headers ignored and are treated as plain HTTP, which is rejected.
A common mistake: the default
--proxy-mode noneunder the production profile answers every request except/healthwith400 Production public origin must use trusted HTTPS metadata. Some older examples omit the proxy flags and look like they should work. They do not.
The matching nginx essentials:
location / {
proxy_pass http://127.0.0.1:6174;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Proto https;
client_max_body_size 1m;
proxy_read_timeout 35s;
}
Other points:
- Use a catch-all virtual host. Adding a domain needs DNS and TLS at the proxy but no proxy edit.
- The domain match is case-insensitive, ignores a trailing dot and the default
port. A non-default port is kept, so
example.com:8443does not matchexample.com. - Do not expose the management port to the internet. The reference deployment reaches it through an SSH tunnel.
Fixed limits
The public listener has built-in limits that are not configurable: 1,024 connections (100 per client address), 256 concurrent requests, 1,000 requests per connection, 1 MiB request body, 8 KiB request line, 32 KiB and 100 headers, 10-second header timeout, 15-second body idle timeout and a 30-second handler deadline. SIGTERM or SIGINT drains for up to 30 seconds. There is no per-site CPU, memory or request quota, and no way to forcibly cancel a runaway page, so a host is not a safe place for untrusted tenants.
Health and status
On the public listener (answered before site routing, so they shadow a site route at the same path):
| Request | Response |
|---|---|
GET /health | 200 {"status":"live"} whenever the process is up |
GET /__crisptastic/ready/<domain> | 200 if that exact domain is ready to serve, otherwise 503 |
On the management listener (mTLS): GET /live, GET /ready, GET /status and
POST /v1/deployments/activate. There is no /metrics, no log endpoint and no
audit log. Logs are plain lines on standard output and standard error, such as
route example.com -> <deployment-id> at start-up; capture them with your service
manager.
Use /__crisptastic/ready/<domain> for per-site readiness checks.
Storage options
--store-dir DIRkeeps Capsules inDIR/objects/capsules/sha256/<aa>/<digest>.age.--s3takes its configuration only from the environment (see Deploying). The S3 client is minimal: static credentials, whole objects in memory, no listing. Hosts address Capsules by digest only, so nothing can ask "what is the latest".
Sites with an embedded store
A site whose crisptastic.toml declares [data] gets a Residiuum store that
the host, not the deployment, owns: <host-root>/stores/<application_id>/,
created at the site's first activation and reused by every later generation.
The rules:
- One site, one store. The store is named by
[data] application_id. The site that first activates it owns it; a different site (one whose domains do not overlap) claiming the same id is refused withdata.application_id_owned. Sites never see each other's heaps. - Created once, with a key the Capsule carries. A store is sealed. The
first activation needs its unseal key, named by
[data] unseal_secretas one of the Capsule's secure variables ([env_from_process]); it is read into locked memory, used to create the store, and never written to disk. Every later activation and every restart opens the store sealed, without the key. Activating a site whose store does not exist yet and which names nounseal_secretfails withdata.store_missing. - One writer. A new generation of the site adopts the store the running generation holds open; when the last generation is retired the store is closed, and the next activation reopens it sealed. A new generation may add record or content collections, which are created in the open store; one it drops stays in the store, unserved.
- Its own memory.
[data] memory_budgetis that store's budget, counted within the host'sMemoryMax. Size the host for the sum of its stores. The budget is fixed while the store is open: a generation that changes it is refused withdata.memory_budget_changed, and takes effect after a host restart. - Single host only. A Fabric cluster refuses
[data]withdata.hosted_fabric_unsupported; the store would live on one member while the next request may be served by another.[data] rootis used bydevandrun(default.crisptastic/residiuumbeside the site) and ignored by a host, so a hosted site may leave it out.
Backup and recovery
| What | Survives a restart? | Back it up? |
|---|---|---|
| Encrypted Capsules in the store | Yes | Yes. This is your durable record, and it holds your secrets, encrypted |
Embedded stores under <host-root>/stores/ | Yes | Yes. A site's records live only here; stop the host or use Residiuum's dump before copying |
| The host age identity | Yes | Yes. Without it every stored Capsule is unreadable |
| Signing private key | (on the deployer) | Yes, offline |
| Trusted signer keys, management PKI | Yes | Yes |
| Unpacked public files | Yes (not cleaned up) | No, regenerable |
Session files and kept login sessions under <host-root>/sessions/ | Yes | Optional; losing them only signs people out |
| Secrets in memory, routing table | No | Not applicable |
To restore a host, start it with the identity, signers, store and the digests of the Capsules to activate. There is no export or restore command. After total loss the supported path is to redeploy your site with the original environment.
Upgrading
Replace the binary and restart. A Capsule whose schema the new binary does not
understand is rejected, never guessed at. [login] visitors stay signed in
across the restart, and a restored session refreshes its Arbax token on first
use; Wasm [arbax] users are signed out.
What is not here
These appear in the older specifications but are not implemented: a host.toml
file and host init, a challenge-authenticated deployment protocol and
keys authorize, an operations listener with /metrics, rollback and retire
commands, per-tenant audit sinks, and hosted telemetry. See
Known gaps.