Clustering with Fabric

What the Fabric is

The Fabric makes multi-tenant site runtimes highly available across replaceable Crisptastic nodes. A deployment targets the Fabric, not a machine. The Fabric continuously reconciles the desired state (which Capsules, how many replicas) against healthy replicas, and heals after a node is lost.

It is a private, volatile, in-memory cluster:

no plaintext application secret ever enters persistent Fabric state;
a total Fabric outage may lose volatile secrets permanently, by design;
nodes are cattle — a restarted process gets a fresh 128-bit boot ID and rejoins as a learner before it can vote.

The Fabric is not the proxy cluster

Two independent clusters:

Reverse proxy (e.g. Kismet)	Crisptastic Fabric
Terminates public TLS	Places Capsules on nodes
Accepts public connections	Runs and repairs site replicas
Picks a healthy Crisptastic origin	Owns readiness, secret continuity, sessions

The proxy must not be the only store of placement, deployment or session state — a working proxy cannot make a single site process highly available.

Topologies

Nodes	Profile	Behaviour on one loss
1	development / non-HA	site down until re-activated
2	minimum site HA	survivor keeps serving as `serving_degraded`; all control-plane changes freeze
3 voters	smallest production HA	tolerates one loss, keeps a majority for control changes
more	scale out	sites still default to 2 replicas each

Default: two replicas per site, two required readiness acknowledgements.

Running a clustered node

crisptastic host \
  --identity /run/credentials/capsule.agekey \
  --trusted-signer /run/credentials/release.pub \
  --host-root /var/lib/crisptastic \
  --fabric-config /etc/crisptastic/fabric.toml \
  --s3

--fabric-config is mutually exclusive with positional Capsules and --digest. With Fabric configured, the mTLS management listener becomes the authenticated clustered deployment entry point; it never activates a local route directly.

A joiner config names this process's identity and at least one live seed:

cluster_id       = "1111...."
placement_salt   = "0123....(64 hex)"
boot_id          = "aaaa....(32 hex)"
credential_serial = 1
bind             = "10.20.0.11:7443"
advertise        = "10.20.0.11:7443"
certificate_der  = "node-a.der"
private_key_der  = "node-a.pkcs8.der"
ca_certificate_der = "fabric-ca.der"
admission_authority_env = "CRISPTASTIC_FABRIC_ADMISSION_AUTHORITY"

[[peers]]
boot_id = "bbbb....(32 hex)"
address = "10.20.0.12:7443"
credential_serial = 2

Admitting a new node

New nodes are not trusted by CA validity alone. An operator on an authority-bearing machine issues a short-lived grant bound to that exact boot, certificate fingerprint and endpoint:

macrun run crisptastic authority -- \
  crisptastic fabric issue-join \
    --cluster-id 1111.... \
    --boot-id dddd.... \
    --credential-serial 4 \
    --endpoint quic://10.20.0.14:7443 \
    --certificate-der node-d.der \
    --request-id 3333.... \
  | macrun set crisptastic joiner CRISPTASTIC_FABRIC_JOIN_GRANT --stdin

The grant defaults to a 10-minute lifetime. The joining node presents it once, is added to the roster through consensus, then reconnects as a normal trusted member and catches up as a non-voting replica before being promoted.

Deploying to a Fabric

crisptastic deploy . \
  --recipient age1... \
  --signing-key /run/credentials/release.pk8 \
  --out ./site.capsule.age \
  --publish-s3 \
  --peer deploy.internal.example:7443 \
  --client-cert ./operator.crt --client-key ./operator.key \
  --server-ca ./management-ca.crt \
  --desired-replicas 3 --minimum-ready-replicas 2

The response reports preparing. Routing appears only after the assigned replicas independently verify the exact Capsule digest, byte length, domains and secret-name set, and activation commits through Fabric consensus. By default the stable application ID is derived from the Capsule's domain set; --application-id pins it when a site must change its whole domain set without becoming a new application.

Reading cluster state

crisptastic fabric status --peer <host:port> \
  --client-cert ... --client-key ... --server-ca ...

Returns secret-free membership, placement and readiness from the mTLS management endpoint.

Invariants you can rely on

A domain resolves to at most one active application per cluster epoch.
A replica serves only the exact Capsule digest and secret generation it advertises, and is not routable before local validation and health pass.
Losing one node does not interrupt a site whose availability policy was satisfied just before the loss.
Session correctness never depends on routing a client back to the same node.
A node that cannot prove safe state reports not-ready rather than serving a stale generation.