Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Safety and runtime behaviour

The three runtimes (@elysra/tungsten-runtime for TypeScript, tungsten-runtime for Python and tungsten-runtime for Rust) implement identical semantics, checked by a shared contract suite. This page describes that behaviour.

Diagnostic error envelope

Every failure the SDK returns, from pre-flight validation to transport failure to an HTTP error, is one shape:

{
  "status": "error",
  "category": "VALIDATION_FAILED",
  "operation": "public.webhooks.deliveries.replay",
  "http_status": null,
  "code": null,
  "failed_parameter": "idempotencyKey",
  "received_value": "replay-1",
  "expected": "UUIDv4 string",
  "remediation": "Generate a random UUIDv4, persist it with the replay intent, and pass it as idempotencyKey. Reuse the exact same key on every retry of this replay.",
  "retryable": "never",
  "retry_after_ms": null,
  "next_action": null,
  "request_id": null,
  "trace": { "attempts": 0 }
}

Categories (closed set, shared by all runtimes):

CategorySourceDefault retryable
VALIDATION_FAILEDpre-flight or 400 with envelopenever
MALFORMED_REQUESTnon-JSON 400never
REQUEST_TOO_LARGE413never
AUTH_FAILED401, 403never
NOT_FOUND404 (or gated route)never
CONFLICT409never
PRECONDITION_FAILED402, 412, 422 by manifestnever
RATE_LIMITED429after_delay
UPSTREAM_UNAVAILABLE502, 503, 504 on a readafter_delay
OUTCOME_UNKNOWNtimeout, reset, ambiguous status or unexplained 5xx on a mutationsame_key_only (after_remediation without replay protection)
TRANSPORT_FAILEDDNS, TLS, connection refused before sendafter_delay
CONFIRMATION_REQUIREDdestructive/irreversible without tokennever (call preview)
GATE_DISABLEDstatus matches an x-runtime-gate disabled statusnever
UNEXPECTED_RESPONSEresponse failed strict validationnever

retryable values: never, after_delay, same_key_only, after_remediation. The envelope is designed to be pasted verbatim into a model's context: short, imperative, no stack traces.

received_value is redacted when the parameter is marked sensitive and truncated to 200 characters.

Pre-flight validation

Before any network call the runtime validates: required params present, types and formats (uuid, date-time, e164 via pattern), enum membership, string bounds, closed records (unknown keys rejected when the schema is additionalProperties: false), idempotency key format when the policy is caller_owned, confirmation token when the tier requires it. The first failure is returned as VALIDATION_FAILED with failed_parameter as a JSON path (body.recipient_e164).

Transport pipeline

call → preflight → auth profile → idempotency → [preview?] → send → classify → verify? → result
                                     ▲                           │
                                     └──── retry policy ◀────────┘

Middleware hooks: onRequest, onResponse, onDiagnostic, onRetry. The OpenTelemetry adapter is a middleware. Default headers: User-Agent: <api>-sdk/<ver> tungsten/<ver> (<target>), X-Tungsten-Operation.

Retry policy

  • Read-only: up to max attempts on RATE_LIMITED, UPSTREAM_UNAVAILABLE, TRANSPORT_FAILED, with full-jitter exponential backoff, honoring Retry-After (seconds or HTTP-date) when present, capped by max_ms.
  • Mutating with an idempotency key (auto, caller_owned, content_hash): same as read-only, always with the identical key and identical body.
  • Mutating without a key: never retried; a transport failure before the request was sent is TRANSPORT_FAILED; anything after is OUTCOME_UNKNOWN.
  • Redirects are never followed on mutations. Reads follow same-origin redirects only (as built: at most five, auth query parameters re-applied); fetch is never asked to follow, since it would forward custom credential headers to another origin.

Unknown outcome

Returned when the runtime cannot know whether the server applied a mutation: client timeout after the request was written, connection reset without a response, or a status listed in errors.ambiguous_statuses. The envelope carries next_action from the verification hook when one exists ("Call public.getAlphaMessageStatus with message_id if you have it, or list recent messages filtered by client_message_id"), and retryable: same_key_only. SDK docs state that a new key for the same intent is the one way to duplicate a side effect.

Idempotency store

Interface in each runtime: get(scope, logical_id) → key?, put(scope, logical_id, key). Default memory store; file store adapters for Node and Python; caller_owned + persist: required bypasses the store and refuses calls without a key. Keys are never logged.

Preview and confirmation

preview() returns:

{
  "operation": "public.webhooks.rotate",
  "safety": "destructive",
  "request": { "method": "POST", "url": "https://.../v1/webhooks/0b2e.../rotate", "headers": { "Origin": "https://app.example.test", "x-acme-csrf": "<redacted>" } },
  "effects": [
    "Disables webhook endpoint 0b2e... immediately.",
    "Permanently retires its pending deliveries.",
    "Returns a new signing secret exactly once."
  ],
  "confirmation_token": "tgc1.…",
  "expires_in_ms": 300000
}

, macros follow the same rule at macro level: ClientCore.previewMacro returns a token bound to the macro name and its exact input, and runMacro requires it (or confirm: true for a destructive macro) when the strictest of the macro's declared tier and its steps' tiers is destructive or irreversible; the run's steps then need no tokens of their own.

effects comes from the manifest's confirmation.message, remediation_note and the operation's compacted description. The token is HMAC(session key, operation, args digest) and is only valid for identical arguments. In the SDK the session key is per client instance; in the MCP server per connection; in the CLI it is replaced by --yes because the CLI has no session.

Verification hooks

Compiled from verify entries: after a successful call, optionally run the referenced read operation with mapped arguments and compare with expect; poll up to budget_ms when terminal is given. Results are attached to the success result as verification: { checked: true, state: "queued" }. The hook never changes the primary result and is off by default in the SDK ({verify: true} option), on by default in the MCP invoke for irreversible operations, and exposed as --verify in the CLI.

Sensitive data

Fields marked sensitive (manifest or x-agent-sensitive) are: excluded from received_value, excluded from traces and logs, redacted in previews, and flagged shown_once in MCP text renderings. The runtime never persists them.

Auth profiles

Profile kindRuntime behavior
bearerAuthorization: Bearer <token> from env, config or callback; optional required prefix check (ztw_, ApiDescriptor http_bearer prefix) before every request: a mismatch is AUTH_FAILED and nothing is sent
api_keyheader, query or cookie
basicheader
oauth2_client_credentialstoken fetch and refresh with single-flight
oauth2_authorization_codehelpers on the client (oauth): PKCE S256 (RFC 7636), authorizationUrl, exchangeCode, refresh; the tokens live in ClientOptions.tokenStore (memory by default) and are sent as bearer, refreshed single-flight when they expire within 30 s, and refreshed once, then the call is sent once more, after a 401; see "OAuth2 authorization code"
compositethe ordered parts: cookies added to the jar, derived headers computed (equals_cookie), conditional headers (when: mutation) added for non-GET; missing parts fail pre-flight with AUTH_FAILED and a remediation naming the missing piece

OAuth2 authorization code

An authorizationCode flow of an oauth2 scheme reaches the API descriptor as authorizationCode: {authorizationUrl, tokenUrl, refreshUrl, scopes} (Python authorization_code, Rust AuthSchemeDescriptor::Oauth2.authorization_code) and only then; generated clients expose the helpers as client.oauth[scheme] (Rust: client.oauth(scheme)). The credential of the scheme is {flow: "authorizationCode", clientId, clientSecret?, redirectUri?} (client_id, ... in Python and Rust); a bare string is still an access token. There is no interactive flow in the runtime: the application sends the user to authorizationUrl(...), receives the redirect, and calls exchangeCode(...).

  • pkce(): a 43 character URL-safe verifier from 32 random bytes and its S256 challenge, base64url(SHA-256(verifier)) without padding.
  • authorizationUrl: response_type=code, client_id, redirect_uri, scope (space separated; the flow's scopes by default), state, code_challenge with code_challenge_method=S256, then extra parameters in key order that never override these. RFC 3986 percent-encoding.
  • exchangeCode and refresh POST a form to the flow's tokenUrl (refreshUrl for a refresh, tokenUrl when absent): grant_type=authorization_code, code, redirect_uri, code_verifier; grant_type=refresh_token, refresh_token. A client with a secret authenticates with HTTP Basic (client_id:client_secret, percent-encoded); a public client adds client_id to the form. Both return metadata without secrets (expiresAt, scope, refreshable); the tokens go to the store. A refresh keeps the old refresh token when the answer has none.
  • A call reads the store: no token is AUTH_FAILED before anything is sent; a token that expires within 30 s is refreshed first, concurrent callers sharing one refresh request (a token without a refresh token is AUTH_FAILED asking to authorize again); a token endpoint answer of HTTP 4xx to a refresh deletes the stored tokens. A 401 from the API refreshes the token once (when it has a refresh token and was not replaced meanwhile) and sends the same request once more; a second 401 is the envelope. Streams and previews do not retry.
  • Client secret, code, verifier and tokens are scrubbed from envelopes and never appear in middleware contexts; the token endpoint's error text is not repeated (only an error code such as invalid_grant).

Results without a body

A success response without a body (204, or a success status declared without content) is {ok: true, value: undefined}; generated SDKs type such operations as void, or T | undefined when the body is optional.

Rules fixed by the contract suite

  • Retry tiers: defaults.retries reaches the SDK as ApiDescriptor.retries (readOnly, mutating). A call's policy is the runtime default, then its operation's tier, then the client's own retries option, field by field. A mutation is still retried only with a key or an identity body.
  • Unknown outcome: next_action comes from the verification hook (when it can be called without the lost response, see the review rules below); without one, a keyed mutation's names the repeat with the same key (and identical arguments), a content_identity mutation's the repeat with the identical body; otherwise it is null. An ambiguous status keeps the manifest's text for it (errors.non_json, errors.codes) in the remediation.
  • A read never reports OUTCOME_UNKNOWN: a manifest entry mapping a status to it reads as UPSTREAM_UNAVAILABLE, after_delay, with the entry's text.
  • Verification hooks and confirmation messages are written against the API: argument keys, $args.<name> references and {name} placeholders use wire names, resolved to the generated argument names. $response and $args references in expect and terminal are resolved before polling; one that does not resolve never matches.

Rules fixed by review

  • Replay protection decides retryable for an unknown outcome: with a key or an identity body it is same_key_only; without either (an unkeyed create, workflow.action.send, cancel, enable, rotate) it is after_remediation, so an envelope-only policy checks instead of resending.
  • On a mutation, any 5xx (a gateway's 502/504, an intermediary's bare or HTML 500, a JSON 500 with an unknown code) is OUTCOME_UNKNOWN, not "wait, then call again". A manifest entry for the answer's code or non-JSON shape that is a definite rejection (retryable: never / after_remediation, or a category such as PRECONDITION_FAILED, NOT_FOUND, RATE_LIMITED) wins over this and over ambiguous_statuses; one that maps to UPSTREAM_UNAVAILABLE (Acme's unavailable, which can report a failed commit) does not.
  • next_action uses the verification hook only when its arguments do not need $response: after a lost submit the caller has no message_id, so the same-key repeat is the next action.
  • A gate's disabled status is recognized only without the API's error code (the route is not mounted; the mock answers it bare). The mounted route's own JSON 404 is classified by its code.
  • Partial results: a failed result may carry partial (never inside the envelope): the decoded body of a mutation whose success response failed strict validation (the effect happened and the body can hold a one-time secret; strict-validation-keeps-a-created-secret), or a macro's completed step results by as name. A macro's envelope names the macro; when a completed mutating step has no replay protection the envelope says not to rerun the macro, downgrades after_delay / same_key_only to after_remediation and names the failed step as the next call.
  • A confirmation token authorizes one intent. The first call that sends spends it and binds it to that call's replay protection (the key sent, or the identity body); it is accepted again only for a retry under the same protection, and a call without a key can use it once. Macro tokens likewise.
  • A path parameter that would make an empty or dot segment ("", ., ..) is VALIDATION_FAILED before anything is rendered, confirmed or sent: URL parsing removes such segments, so DELETE /items/.. would reach the collection.
  • Sensitive request fields (AgentMeta.sensitiveRequestFields) are redacted in received_value (also below a parent value), in previews, in the body middleware sees (RequestContext.body is the redacted rendering whenever redaction changed a JSON or form body) and are scrubbed from diagnostics.
  • An optional argument set to undefined is absent (TypeScript callers without exactOptionalPropertyTypes pass {cursor: maybe}); the request schemas stay exact.

Logging and telemetry

Structured events through onDiagnostic and onRequest/onResponse, no default stdout logging. The OTel middleware maps operations to spans with http.* attributes and tungsten.safety, tungsten.idempotency.policy, tungsten.outcome.

Event streams (SSE)

ClientCore.stream (TypeScript), stream on ClientCore/AsyncClientCore (Python) and ClientCore::stream (Rust, with the stream's own descriptor) iterate the server-sent events of an operation. Generated SDKs get <method>Stream / <method>_stream; MCP tools whose operation only streams (or whose flag is true) return {events}.

  • The request is Accept: text/event-stream with the stream's request flag set to true. A caller's Accept (the call's, else the client's; any casing of the name) is merged with text/event-stream into one header, never duplicated: its media ranges in order without repeats, text/event-stream first when they do not name it; a q-value the caller wrote stays. Everything before the first byte of a 2xx event stream (validation, auth, status errors, retries by the usual tier rules) is the normal envelope and the only item. A 2xx answer of another type is UNEXPECTED_RESPONSE.
  • The body is parsed per WHATWG HTML 9.2 (LF/CRLF/CR, split chunks, comments, multi-line data, event, id and retry persisting across events, a leading BOM dropped, an event still open at the end discarded). data is JSON; the declared done value ends the stream without an item.
  • Each item carries value, event (default message), id, retry and the response meta. Events are checked against the event type per validateResponses: strict ends the stream with UNEXPECTED_RESPONSE (events[N].path), warn reports through onDiagnostic and delivers, off does not look. The Rust SDK decodes into generated types, so an event that does not decode ends the stream in every mode.
  • A started stream is not retried by the tier rules, but it can be resumed (below). A lost connection or a stream idle for the attempt timeout ends it with one final failure item: TRANSPORT_FAILED or UPSTREAM_UNAVAILABLE for a read, OUTCOME_UNKNOWN (with the usual check-before-repeating rule) for a mutation. Leaving the loop closes the connection.
  • Resuming: a connection that drops after the stream started (a transport error, or a clean end before the done event the operation declares; a stream that declares none ends cleanly) is reconnected with Last-Event-ID set to the last event id delivered, up to maxReconnects / max_reconnects times (default 3; 0 turns it off, and a clean end is then a clean end as before). Only a call that is safe to repeat is reconnected: a read, or a mutation with replay protection (the same request goes out again, idempotency key included). It needs an event id to resume from, or no event delivered yet (then the request is simply repeated). The wait before a reconnect is the server's last retry value (250 ms when it sent none) capped at reconnectMaxMs / reconnect_max_ms / reconnect_max (default 30 s) and shortened by up to 25 % at random (so it never exceeds the cap). A server that ignores Last-Event-ID and replays events is harmless: after a reconnect, events whose id was already delivered are skipped until the first new one. The reconnect is a new request through the usual pipeline (its failure, an error status or a body that is not an event stream, is the final item as an ordinary envelope); when the reconnects are used up the final item is the failure of the last drop and its remediation says how many reconnects were made. Items report reconnects in their meta (the stream so far), id and retry carry over a reconnect. The caller's abort signal / cancel / drop ends the stream for good, never reconnects.
  • Idle timeout: idleTimeoutMs / idle_timeout_ms / idle_timeout (default none) is the silence, bytes of comments included, after which a stream is idle. An idle stream is reconnected like a dropped one; when it cannot be, it ends with UPSTREAM_UNAVAILABLE (read) or OUTCOME_UNKNOWN (mutation) whose code is STREAM_IDLE. Without it the attempt timeout ends a silent stream as before, without a reconnect and without the code. The category set is unchanged: STREAM_IDLE is a code value, the one the runtime itself sets (envelope schema, code).
  • Helpers, in every runtime: the stream can be consumed with on (TypeScript and Python: a handler per event name, * for the others; Rust: EventHandlers), collect (TypeScript and Python; Rust has collect_values), reduce (fold the events, e.g. concatenate text deltas), first(predicate) (then close the stream) and cancellation (TypeScript cancel() or the AbortSignal; Python cancel(), also from another thread or task, which ends a blocked read; Rust: drop the stream). They return the result and reconnects, or the failure with what was gathered before it (partial, TypeScript and Python; Rust Folded). cancel() ends the stream without a failure item.
  • Size limit: one event holds at most maxEventBytes / max_event_bytes (client option, default 1 MiB): the UTF-8 bytes of its field lines, each counted with a one-byte terminator, plus the line being read; a finished comment line is not kept and does not count (an unfinished one counts like any line). Past it the parser stops, the events before it are delivered and the stream ends with UNEXPECTED_RESPONSE (failed_parameter events[N], expected an event of at most N bytes), never buffering more than the limit plus one chunk. The same in TypeScript, Python (sync and async) and Rust.
  • Collection budgets: collecting a stream into a list is bounded. The MCP tool of a streamed operation stops after 1000 events (truncated: true), and fails with UNEXPECTED_RESPONSE (failed_parameter events, the events so far as partial) after maxStreamBytes of event JSON (default 4 MiB) or maxStreamMs (default 60 s, whatever the idle timeout; the connection is closed); both are ServerOptions. The Rust dynamic Dispatch::stream (generated collect_events, EventStream::collect_values) ends the same way, last item an error, after ClientOptions::max_collect_bytes (4 MiB) or max_collect_time (60 s). Python's collect() is not bounded; its callers bound it themselves.
  • MCP: a streamed tool call reads the stream through ClientCore.stream, so it resumes like any other (the server's client options apply). When the tools/call request carries a progressToken, each event collected is a notifications/progress (progress is the number of events so far, message says how many were collected and how often the stream reconnected), so progress keeps flowing across a reconnect.
  • Mock: tungsten mock serves a generated stream from the event after the numeric Last-Event-ID of the request (the done sentinel has no id and stays), and the injections drop-after=<n>[;retry=<ms>] (the first n events, then the connection breaks) and stall-after=<n>[;ms=<ms>][;retry=<ms>] (then silence for ms, default 30 s) make a resumable and an idle stream; retry starts the stream with a retry: line. Programs can apply them (inject); a scripted body still replays whatever it says, which is how the contract tests a server that ignores Last-Event-ID.
  • Detection (build): the stream is carried by an exact success status, else a 2XX range, else default (a default that declares text/event-stream and has no sibling success response becomes the success response; next to a success response it stays an error response and carries no stream).