Case study: the ZROtext pilot
ZROtext is an open-source Android SMS gateway with a Rust server. Its contract is three OpenAPI 3.1 documents (the public, sealed and workflow planes, 44 operations in all), and it ships hand-written clients, which makes it a useful baseline. It was the first real API tungsten was pointed at: the TypeScript, Python and Rust SDKs, the generated CLI and the MCP server were generated from those documents and two manifests, then every scenario was run against a live instance on loopback. Nothing in the pilot sends a real SMS. This page summarises what the run showed.
Headline
- The generated TypeScript, Python (sync and async) and Rust SDKs pass all 26 live scenarios against the real server: 4 driver runs, 104 scenario runs.
- The parity check finds the four records identical (26/26, three times). The generated Rust CLI passes 20/20 live checks.
- All the live scenarios pass in every SDK and in the CLI.
- No runtime or emitter defect was found. The live run found four defects in
the agent manifest; all are fixed with offline regression tests, and
one led to the new
tungsten checkwarning TG0614. - ZROtext's hand-written alpha clients behave the same on the wire but give their caller only status, code and Retry-After.
Criteria
| Result | |
|---|---|
| Compile | 44 operations, 0 errors, 3 TG0301 warnings |
| Golden output | goldens regenerated |
| Generated CLI | unchanged |
| Contract suite | 64/64 on each of 4 runs, parity 64/64 ×3 |
| Tool schema size | largest tool schema 538 tokens, median 116, none over the budget |
| Webhook lifecycle | webhook lifecycle with the verification hook, the bootstrap macro, replay of a dead delivery (generation 2, created true then false, then 409 (not eligible)): pass in 4 SDK runs and the CLI |
| Submit and cancel a message | submit 202 created:true → identical resubmit 202 created:false → status 200 → cancel 204 → cancel again 409 never → status 200: pass in 4 SDK runs and the CLI |
| Missing idempotency key | missing key refused before sending: attempts 0, the proxy recorded 0 requests; CLI exit 2 |
| Disabled feature | gate off → bare 404 → GATE_DISABLED with the gate's text (also the sealed and workflow planes); CLI exit 3 |
| Pagination and revoke | 50 + 50 + 6 devices via the list-all macro; revoke needs confirmation; CLI exit 6 without --yes |
| Timeout with an unknown outcome | the 600 ms client timeout fires while the proxy holds the 202 for 2.5 s → OUTCOME_UNKNOWN / same_key_only; the same-key retry gives created:false; the owner message list holds one message; CLI exit 5 |
Extra live scenarios: 409 for the same key with another body; the await macro
reports timed_out honestly; a device queue holds 16, the 17th is 429 (queue full) with Retry-After 60 and same_key_only; a CSRF value the session
does not own and a foreign origin are 403; an unknown bearer is 401; the key
bound to one device is 403 for the other; an unknown endpoint is 404; limit
21 on the delivery page is refused before sending.
Findings
Each finding has a fix and an offline regression test, so the mock-based suite catches it next time.
- Reserved callback hosts. The server refuses callback hosts under
.test,.example,.invalid,.localand similar suffixes with a 400 error; the example host and the mock both accept them. Fix: theagent.ymlremediation text names the rule. - Expiry window. An expiry more than 15 minutes ahead gets a bare 400 that the spec never explains. Fix: an operation-specific
remediation with
retryable: after_remediation. - Ungated planes. The sealed plane's gate was declared but no
sealed operation named it, and the workflow plane had no gate, so a default
deployment's 404 came back as
NOT_FOUND. Fix: agateon all 15 sealed and 9 workflow operations, plus a gate for the workflow plane; newtungsten checkwarning TG0614 for agatesentry no operation uses. - Rate-limit text. A rate-limit answer on submit carried the "four requests in
flight" text when the live cause was the 60-per-minute budget, and had no
same-key rule. Fix: an operation-specific remediation with
same_key_only. - Harness. Live runs were order dependent because of the server's budgets and caps (8 webhook endpoints, 16 pending messages per device, 60 submissions per minute). Fix: the proxy's reset restores server state before every scenario.
- Policy, unchanged. The manifest requires a UUIDv4 submit key; the server
accepts any 1–128 character
[A-Za-z0-9._-]key.
Comparison baseline
| Hand-written (ZROtext's own clients) | Generated by tungsten | |
|---|---|---|
| Coverage | alpha plane only (no owner-webhook or device client) | public, sealed and workflow planes |
| Size | TypeScript 284 lines, Python 239; signature verification 96 / 80 | TS 6,461 lines, Python 8,535, Rust 13,140 + CLI 812 |
MCP tools/list | its MCP server 9,473 tokens, its send-only MCP server 539 | progressive 765, discrete 14,205 |
Run live through the same flows:
| Flow | Hand-written | Generated |
|---|---|---|
| submit, resubmit, status, cancel | same | same |
| cancel again | an API error with status 409 | envelope CONFLICT / never with remediation |
| queue full | an API error with status 429 | RATE_LIMITED / same_key_only, retry_after_ms 60000 |
| bad key | a type or value error before sending | VALIDATION_FAILED before sending |
| drop or timeout | an outcome-unknown error; same-key retry gives created:false | OUTCOME_UNKNOWN / same_key_only with next_action; same-key retry gives created:false |
Equal: wire behaviour and no automatic retry. Better in the generated SDKs: the envelope, preview tokens, secret handling and owner-plane coverage. Missing or worse in the generated SDKs: webhook signature verification, sealed envelope composition, and the stricter UUIDv4 key.