Clustering with Fabric

Status: preview. The Fabric has been exercised in disposable three-node trials, not in production. This chapter is an overview for authors and operators deciding whether they need it. A single host is a single point of failure: a restart loses secrets and routing until sites are re-activated. The Fabric is an optional private cluster of hosts that keeps sites running when a node is lost.

You do not change your pages to use it. The difference is how you deploy: you deploy to the cluster rather than to one machine.

What the Fabric does

Holds a catalog of which sites should run, replicated across voting nodes (OpenRaft consensus over QUIC with mutual TLS).
Assigns each site to desired_replicas nodes (default 2).
Has each assigned node download the exact Capsule by digest, verify it and report ready. Routes start only once enough replicas are ready.
Repairs assignments when a node is lost.

It is not the reverse proxy. A proxy (for example Kismet) still chooses a healthy node for each request.

What it does not do (yet)

Nothing is durable. The catalog, membership, assignments and secrets live in memory. After losing the whole cluster you start a new, empty one and redeploy every site. The Fabric never rescans S3 and never guesses "the latest" Capsule.
Sessions are not replicated. Each node keeps its own session files, so a visitor routed to a different node is signed out, and a redeploy signs everyone out. Do not use sign-in on a clustered site until this is delivered.
Nodes do not hand secrets to each other. Each node decrypts the Capsule itself with its own age identity. Consequently every node that may host the site must have been listed with --recipient when you built the Capsule.
Stateful sites using the embedded data store are not supported.
No export or restore command, no automated certificate tooling, and no credential rotation.

Topologies

Nodes	Behaviour
1	Not highly available. A restart needs re-activation.
2 serving	A site keeps serving after one node fails (reported `serving_degraded`), but deploys and changes freeze, because two voters cannot form a majority after losing one.
3 voters	The smallest self-contained production layout: tolerates the loss of one voter and still accepts deploys.

Deploying to a cluster

You encrypt one Capsule to all eligible nodes, publish it to a shared store (S3), and submit it to any node's management port:

crisptastic deploy ./my-site \
  --recipient age1node-a... --recipient age1node-b... --recipient age1node-c... \
  --signing-key release.pk8 --out ./site.capsule.age \
  --publish-s3 \
  --peer deploy.internal:7443 --client-cert operator.crt --client-key operator.key --server-ca management-ca.crt \
  --desired-replicas 3 --minimum-ready-replicas 2

The receiving node checks the Capsule against what you sent, including length, domains and secret names, then submits it to the catalog. The response is "state": "preparing". Check progress with:

crisptastic fabric status --peer deploy.internal:7443 \
  --client-cert operator.crt --client-key operator.key --server-ca management-ca.crt

Cluster states include healthy, serving_degraded, no_control_quorum, rebalancing and draining; application states include active_ha, active_under_replicated, active_single_replica, active_sleeping and failed.

deploy --peer needs the site to declare [deployment] domains. The default --application-id is derived from the domain set; pass --application-id explicitly if you expect the domains to change.

Operating nodes

Each node is started with host --fabric-config fabric.toml (together with the usual options), and the fabric.toml file defines the node's identity, certificates, peers and either a formation or an admission section. Key points:

Every node needs its own age identity, boot ID, certificate and credential serial. A restarted process that wants to rejoin needs a new boot ID, certificate and serial, and a new join grant if it joins dynamically. Nothing makes these automatic.
Start the non-forming initial voters first, then the one carrying the formation proof.
Add a node by issuing a short-lived grant: crisptastic fabric issue-join .... Newly admitted nodes replicate the catalog and can host sites, but only vote if the leader promotes them to replace a failed voter.
A host started with --fabric-config clears its deployments folder at every start and never activates routes on its own.
You must generate the certificates yourself. The files must be DER-encoded with specific subject alternative names; see the operator guide.

Sleeping sites (preview)

An opt-in memory-saving mode. When the process's memory use crosses a configured high mark, the host unloads the least recently used idle site, and reloads it on the next request from a verified cache or S3.

Enable per site with deploy --sleepable, and per host with --runtime-memory-high-bytes, --runtime-memory-low-bytes (and optionally --runtime-min-residence-seconds, --capsule-cache-dir).
Needs Linux, --fabric-config and --s3.
Only for stateless sites. A waking site responds 503 to at most 16 waiting requests for up to 8 seconds, so expect a cold-start delay.
No memory or latency capacity claims have been measured.

Kismet ingress (preview)

Kismet is a Frogfish reverse proxy. A host can register itself with it:

crisptastic host ... \
  --proxy-mode x-forwarded --trusted-cidr 127.0.0.1/32 --trusted-hops 1 \
  --ingress-kismet-credential /etc/crisptastic/kismet \
  --ingress-instance-key node-a \
  --ingress-health-path /__crisptastic/ready/example.com

The host registers its ready domains with Kismet every five seconds and withdraws them before retiring a site. Only /__crisptastic/ready/<domain> is a meaningful health path; the default /health reports only that the process is alive. A single health path names one domain, so how to configure a host that serves several domains is an open question for the maintainers.