Fetching and shaping data

Pages get data from other services with @fetch, and reshape it with @shape or sda(...).

Crisptastic has no database server. Your records live in your own API, database service or content store, and the page reads them over HTTPS. (A site on a single host can also have a small embedded store of its own; see Capabilities and integrations.)

`@fetch`

@fetch makes an outbound HTTP request that returns JSON, and binds the result to a name for the rest of the page.

@fetch posts = get_json("https://api.example.com/v1/posts", query: { limit: 5 })

@if (posts.ok) {
  @for (post in posts.value) {
    <h2>{{ post.title }}</h2>
  }
} @else {
  <p>Could not load posts ({{ posts.error.code }}).</p>
}

Forms

@fetch name = get_json(url [, query: e] [, headers: e] [, timeout_ms: e])
@fetch name = fetch_json(method, url [, body: e] [, query: e] [, headers: e] [, timeout_ms: e])

method is one of get, post, put, patch, delete. The whole directive must be on one line.

Option	Meaning
`query`	An object of strings, numbers or arrays of them. `null` values are skipped.
`headers`	An object of scalar values.
`body`	Sent as JSON with `content-type: application/json`. A string body is sent as-is, e.g. `form_urlencode({...})` for form posts.
`timeout_ms`	Non-negative integer. There is no default timeout other than the 5-second page render limit.

The result

@fetch never throws on a network or HTTP failure. Always check .ok.

{ ok: true,  status: 200, headers: { ... }, value: <parsed JSON> }
{ ok: false, status: 404 or null, headers: { ... }, error: { code, message } }

`error.code`	Meaning
`transport`	Could not connect, DNS failure, TLS failure, timeout
`invalid_json`	The response was not JSON (including an empty 2xx body)
`http_status`	A non-2xx status. If the body is JSON with its own `code` / `message`, those are used instead
`security.ssrf_blocked`	The URL was refused by the outbound policy (below)
`security.safe_method_mutation`	A non-`GET` fetch inside a `GET` / `HEAD` request on a production host
`invalid_query`, `invalid_headers`, `invalid_timeout`	Bad option values

Redirects are never followed; a 3xx response is reported as http_status.

Parallel fetches

Consecutive @fetch lines (with only whitespace between them) run concurrently, unless one uses the result of an earlier one, in which case it waits.

@fetch user  = get_json("https://api.example.com/user")
@fetch stats = get_json("https://api.example.com/stats")

Keep related fetches together at the top of the page to benefit.

Outbound safety rules

Every outbound request is checked, and these rules cannot currently be loosened:

https only, port 443 only, no credentials in the URL.
Loopback, private, link-local and cloud-metadata addresses are refused. The resolved IP address is pinned, so DNS rebinding does not help an attacker.
Redirects are not followed.

Consequences for development: a page cannot @fetch http://localhost:.... Run your API behind an HTTPS name that resolves publicly, or fetch it from a page-less script instead.

A production host also refuses non-GET fetches during GET and HEAD requests (see Sessions and forms).

Default headers and API keys: `[fetch]`

The optional [fetch] table in crisptastic.toml adds the same headers or credentials to outgoing requests:

[fetch]
api_key = "key-id.secret"       # sent as Authorization: ApiKey key-id.secret
scope = "tenant-42"             # sent as x-scope
cache_ttl_ms = 60000            # in-memory GET cache; 0 disables
max_concurrent_requests = 8     # 0 means unlimited
headers = { x-client = "my-site" }

Warning: these values are attached to every @fetch the site makes, whichever host it targets. Do not set api_key if your pages fetch from more than one trusted service. Instead, pass credentials in headers: on the specific fetch.

Each of these can also be set by environment variable on the process: CRISPTASTIC_FETCH_API_KEY, CRISPTASTIC_FETCH_SCOPE, CRISPTASTIC_FETCH_CACHE_TTL_MS, CRISPTASTIC_FETCH_MAX_CONCURRENT_REQUESTS and CRISPTASTIC_FETCH_HEADERS_JSON.

Reshaping data: `@shape` and `sda()`

APIs often return data in a shape that is awkward for a page. SDA (Structured Data Access) is a small query language that reshapes a value.

@shape names = rows with '{ yield r<name> | r in A }'

or, as an expression:

{{~ sda(rows, "{ yield r<name> | r in A }") | json }}

The input value is bound to A. The @shape form must be on one line, and its query is a string in single or double quotes; use the other kind of quote for quotes inside the query, because backslash escapes break SDA's own tokenizer there. In the sda(...) expression form, the query must be a double-quoted string (expressions have no single-quoted strings), and a double quote inside it is written \".

What is implemented

Only a subset of the full SDA specification is implemented today:

Comprehensions: { yield EXPR | x in SOURCE | PREDICATE }. The source must be an array, and yield and the predicate are optional.
Field selectors: x<field> (required; a missing key is an error), x<field>! (required), x<field>? (optional; null when missing).
Predicates: =, !=, <, <=, >, >=, and, or, not, in, and set literals Set{"a", "b"}.
Pipe: A |> normalizeLast(_) with the placeholder _.
Functions: typeOf, keys, values, normalizeUnique, normalizeFirst, normalizeLast. The normalize* functions turn a list of {key, value} pairs into an object; normalizeUnique fails on a duplicate key.

Examples (verified):

{ yield r<name> | r in A }                     rows -> list of names
{ yield r<id> | r in A | r<city> = "la" }      filter then project
A |> normalizeLast(_)                          [{key,value}...] -> object
keys(A)

Errors are reported as sda() failed: t_sda_*: ..., for example t_sda_missing_key, t_sda_wrong_shape, t_sda_parse.

What is not implemented

The jq-like pipeline syntax shown in older pages (.items | sort_by(-created) | take(5)<code>, object projections like</code>{ id, total }) is not valid. When in doubt, test the query with crisptastic dev before depending on it. For sorting, limiting or complex joins, do the work in the API you are calling.

@fetch