Tested 2026-11-15 · Server 1.8.1, TypeScript SDK · Self-hosted server and Node endpoint in a throwaway local environment.
| Approval gates | Built in An awakeable held the effect until an external resolve arrived; the effect then ran exactly once. The suspension is durable. |
|---|
| Restart recovery | Built in Server SIGKILL mid-work and endpoint SIGKILL mid-work both resumed from the journal with each effect exactly once. Replaying under the same idempotency key returned PreviouslyAccepted — no second execution. |
|---|
| Evidence trail | Built in The full ordered journal (input, steps, the approval signal, output) is queryable via `restate sql`. Ordered and inspectable, but not hash-chained. |
|---|
| Kill switch | Built in Cancel decided a gated invocation — a later resolve of its awakeable was accepted by the ingress but had no effect. Kill decided in-flight work. The CLI refuses without --yes, a footgun worth knowing. |
|---|
| Policy controls | Not claimed No policy layer observed; the server ran unauthenticated as-run. |
|---|
| Sandbox isolation | Not claimed Handlers run as host processes. |
|---|
Tested 2026-11-15 · Server 1.32.2 (dev server, file DB), Python SDK · Self-hosted dev server and worker in a throwaway local environment.
| Approval gates | Documented pattern A workflow waited on a Signal with wait_condition; the effect ran exactly once after approval. There is no built-in approval object — the gate is workflow code on native primitives. |
|---|
| Restart recovery | Built in Worker SIGKILL and server SIGKILL both resumed the workflow exactly-once from event history. A mid-activity kill re-ran that activity (at-least-once, as documented); the downstream effect still ran once. |
|---|
| Evidence trail | Built in Complete ordered event history per workflow via CLI and Web UI. Not hash-chained. |
|---|
| Kill switch | Built in Cancel and terminate both decide in-flight work; signals sent after a stop are refused. |
|---|
| Policy controls | Not claimed No policy layer in the open-source server as run. |
|---|
| Sandbox isolation | Not claimed Workers run as host processes. |
|---|
Findings
- Duplicate workflow IDs re-execute by default. Starting a second workflow with the same ID after the first completed ran the effect a second time; the REJECT_DUPLICATE reuse policy is opt-in per start request and refused it when set. Verified both ways.
Tested 2026-11-15 · SDK 3.5.0, PostgreSQL 16 · Library in a throwaway application backed by a local Postgres.
| Approval gates | Documented pattern A workflow blocked on DBOS.recv until DBOSClient.send delivered the approval; the effect then ran exactly once. |
|---|
| Restart recovery | Built in Executor SIGKILL mid-workflow: DBOS recovered it automatically (recovery_attempts = 2), completed steps were read back from stored outputs instead of re-executing, and the effect ran once. |
|---|
| Evidence trail | Built in The Postgres system schema is the trail: workflow_status plus one operation_outputs row per executed step, in order, queryable with plain SQL. Not hash-chained. |
|---|
| Kill switch | Built in Cancel decided both a gated workflow and a sleeping one. A later approval send was recorded in the database but the cancelled workflow never acted on it. |
|---|
| Policy controls | Not claimed No policy or authorization layer exists in the library as run. |
|---|
| Sandbox isolation | Not claimed In-process library; workflows run in your application process. |
|---|
Findings
- Re-enqueueing under the same workflow ID is deduplicated by default — the opposite default from Temporal, verified in the same battery.
Tested 2026-11-15 · Dev server 1.53.0 (--persist), SDK 4.26.2 · Self-hosted dev server and application in a throwaway local environment.
| Approval gates | Documented pattern A function paused in waitForEvent (CEL match on the event payload) until the approval event arrived; the effect then ran exactly once. |
|---|
| Restart recovery | Documented pattern Split result: killing the app process mid-run recovered exactly-once (step memoization is server-side), but killing the dev server mid-run lost the in-flight run despite --persist — live queue state is held in memory. Production self-host mode (external Postgres/Redis) was not checked. |
|---|
| Evidence trail | Built in Persisted step-level history in the product’s own store: one row per step event with attempts, latency and payloads, plus traces/spans tables. Not hash-chained. |
|---|
| Kill switch | Documented pattern Cancellation is expressed as a cancelOn event pattern in function configuration. |
|---|
| Policy controls | Not claimed No policy layer observed as run. |
|---|
| Sandbox isolation | Not claimed Functions run as host processes. |
|---|
Findings
- A replayed trigger event under the same key was deduplicated by the function’s declared idempotency — one run, one effect.
Tested 2026-11-15 · Library 1.2.22 with SQLite checkpointer; dev API server 0.4.43 · Library and dev API in a throwaway local environment.
| Approval gates | Built in interrupt() is a runtime primitive: the graph parked mid-node, persisted state, and resumed to exactly-once on the resume command — in the library and over the HTTP API. On resume the interrupted node re-executes from its top (documented, observed). |
|---|
| Restart recovery | Built in SIGKILL mid-node: a fresh process resumed the thread from the SQLite checkpointer at node granularity; completed nodes were not re-run and the effect ran once. The dev server’s in-memory runtime is a convenience, not the durable surface. |
|---|
| Evidence trail | Built in Full checkpoint history per thread (source, step, next nodes, checkpoint IDs) persisted alongside state snapshots. Not hash-chained. |
|---|
| Kill switch | Documented pattern The API surface can cancel a run; the library mode has no stop. |
|---|
| Policy controls | Not claimed No policy layer observed. |
|---|
| Sandbox isolation | Not claimed In-process library. |
|---|
Findings
- No submission idempotency: re-invoking a completed thread with the same input starts a new run. A thread_id is a checkpoint key, not an idempotency key.
Tested 2026-11-15 · agent-framework 1.26.0, FileCheckpointStorage · Workflow layer in a throwaway local environment.
| Approval gates | Built in request_info parked the run in IDLE_WITH_PENDING_REQUESTS; the driver process exited entirely, and a fresh process resumed from the checkpoint to exactly-once. |
|---|
| Restart recovery | Built in SIGKILL inside an executor: a fresh process resumed from the latest file checkpoint at executor granularity and completed the run. |
|---|
| Evidence trail | Built in Checkpoint JSON documents carry an explicit previous_checkpoint_id chain, timestamps, the full message log and executor state — the run history is reconstructible from the storage directory. The chain is structural, not cryptographic. |
|---|
| Kill switch | Not claimed No stop in library mode. |
|---|
| Policy controls | Not checked A ToolApprovalMiddleware with rules ships in the public API (verified present), but exercising it needs an LLM-client agent run, which this battery did not do. |
|---|
| Sandbox isolation | Not claimed In-process library. |
|---|
Findings
- Operator trap, verified: WorkflowBuilder without an explicit name= gets a fresh random UUID name on every build, and checkpoints are stored per workflow name — with defaults, a new process cannot find the previous process’s checkpoints. Setting a stable name makes cross-process resume work.
- No submission idempotency: re-running with identical input starts a new run.
Tested 2026-11-15 · v1.7.0 (release binary, sha256 verified) · Standalone gateway in front of a local MCP backend, throwaway configuration.
| Approval gates | Not claimed Requests are allowed or denied synchronously; there is no hold-for-approval primitive. |
|---|
| Restart recovery | Not claimed In-flight requests are not recovered: killing the gateway mid-call cut the client stream and the result was never delivered (work already dispatched to the backend completed anyway). MCP sessions do survive a restart — session state rides in the encrypted session ID. |
|---|
| Evidence trail | Built in Structured per-request logs, including denials with the reason (Authorization). |
|---|
| Kill switch | Documented pattern Killing the gateway stops routing; it does not cancel work already dispatched to a backend. |
|---|
| Policy controls | Built in The strongest policy surface in this battery: CEL rules filtered tools/list per identity, and an unauthorized call was refused before the backend ever saw it. Denials are logged with reasons. |
|---|
| Sandbox isolation | Not claimed The gateway is a proxy; it provides no execution sandbox. |
|---|
Findings
- jwtAuth defaults to mode: optional. Configured from the project’s own example, an anonymous caller could list and call any tool whose rule does not reference JWT claims (tools whose rules do reference claims stayed closed). The one-line fix — mode: strict — was verified to close it: anonymous and garbage tokens get 401, a valid token gets through.
Tested 2026-11-15 · 0.24.1 · Library exercised with the vendor’s own scripted FakeLLMModel test double — call counts are exact by construction.
| Approval gates | Not claimed Rails decide synchronously from model verdicts; there is no hold-for-human state. |
|---|
| Restart recovery | Not claimed Stateless per call — there is no run state to recover. A filter, not an engine. |
|---|
| Evidence trail | Not checked No structured rail-decision record was captured in this run. |
|---|
| Kill switch | Not claimed No operator stop; rails are synchronous checks. |
|---|
| Policy controls | Built in Input self-check refused a request before generation (exactly 1 LLM call — generation never attempted). Output self-check replaced a reply that leaked a secret with the refusal (3 calls). Clean traffic passed verbatim in both directions. |
|---|
| Sandbox isolation | Not claimed In-process library. |
|---|
Tested 2026-11-15 · v0.65.0 (release binary, sha256 verified) · Docker Desktop on a workstation; the registry fetch server as the test workload, driven through ToolHive’s proxy with raw JSON-RPC.
| Approval gates | Not claimed No human-approval primitive in the run surface. |
|---|
| Restart recovery | Built in Killing the server container mid-stable-run: the supervisor detected the exit, restarted the full workload (attempt counters, backoff, maxRetries 10), and the workload served real calls again on the same port in under a minute. During the window the egress proxy answers with its own error page, not an MCP error. A kill delivered during initial startup failed its restart attempt — startup-phase kills are handled worse. |
|---|
| Evidence trail | Documented pattern Structured per-workload logs record exits (with the server’s last logs embedded), restart attempts and auth warnings. Local log files; no queryable audit store observed. |
|---|
| Kill switch | Built in thv stop and rm tore workloads down cleanly in every phase (verified by container listing afterwards). A stop issued inside a restart window races the restart. |
|---|
| Policy controls | Documented pattern OIDC authentication and authorization policies exist as configuration, but the default run is unauthenticated — see the finding below. The enforcing configuration was not exercised. |
|---|
| Sandbox isolation | Built in Every workload is four containers: the MCP server, an ingress proxy, a Squid egress proxy and a DNS sidecar. The server has no direct network path; egress is proxied by construction. |
|---|
Findings
- Default posture, from the product’s own log on restart: “no authentication configured: the proxy will accept every request unauthenticated and assign each a synthetic local-user identity with no token or credential check; configure an OIDC provider to enforce authentication per request.”
- thv list reports “No MCP servers found” transiently during any (re)start — a list call that lands in a startup window looks like an empty registry.
Tested 2026-11-15 · Gateway v2.0.1 via Docker Desktop 29.4.3; catalog of 313 servers · Docker Desktop on a workstation; the catalog fetch server in a test profile.
| Approval gates | Not claimed The gateway is allow/deny by configuration; no human-approval primitive observed. |
|---|
| Restart recovery | Not claimed The gateway is a stateless stdio proxy and servers are per-session containers started with --rm; there is no in-flight run state to recover. Profiles and catalog state persist on disk across sessions. |
|---|
| Evidence trail | Not checked Verbose console logging was observed; no durable, queryable audit record of calls was exercised. |
|---|
| Kill switch | Documented pattern Ending the gateway process ends the session and its containers. |
|---|
| Policy controls | Built in Per-tool allowlists per profile, enforced at list time and verified in both directions: disabling fetch.fetch left 0 tools listed in a fresh session; re-enabling restored it, and a real call then round-tripped through the gateway. |
|---|
| Sandbox isolation | Built in Servers run as hardened containers: --security-opt no-new-privileges, CPU and memory caps, and secrets kept out of the server environment (block-secrets defaults on; secrets resolve through the Docker Desktop store). |
|---|
Findings
- Catalog reference detail: the working catalog ref is mcp/docker-mcp-catalog:latest; the docker/mcp-catalog ref returns UNAUTHORIZED.
Tested 2026-11-15 · Server 4.80.0 (self-hosted compose stack), Python SDK 5.2.0 · Full compose stack (web, worker, Postgres, ClickHouse, Redis, object storage) on a workstation, with throwaway project keys.
| Approval gates | Not claimed Not applicable — Langfuse is an observability platform with no execution surface to gate. |
|---|
| Restart recovery | Not claimed Langfuse runs no agent work, so the criterion does not apply as defined. What the battery verified instead is under Evidence: the stored trail survives restarts. |
|---|
| Evidence trail | Built in End to end: traces enter via OTEL, persist in ClickHouse (events_core), and survive web and worker restarts — verified twice, including a trace ingested before a restart. Scores are first-class, timestamped, trace-linked verdict records. Caveat: v4 runs in events_only write mode — the legacy ingestion endpoint refuses trace-create and legacy REST trace read-back is unavailable; the trail is queried via the events tables. |
|---|
| Kill switch | Not claimed Not applicable — no execution surface to stop. |
|---|
| Policy controls | Not claimed Not applicable as an execution control. Ingest authentication was observed: the OTEL endpoint validates project keys (authCheck validKey). |
|---|
| Sandbox isolation | Not claimed Not applicable — Langfuse executes no code or tools. |
|---|
Tested 2026-11-15 · CLI and SDK 4.8.5 · Cloud free plan, dev environment; tasks ran against a project created for this battery. Every run cost $0.0000; no payment card exists on the account.
| Approval gates | Built in wait.forToken held a run for ~2.5 minutes; completing the token externally fired the effect exactly once, and pre-wait steps were not re-executed. Note: wait.forEvent does not exist in SDK 4.8.5 (the installed API surface was inspected directly). |
|---|
| Restart recovery | Not claimed In dev, runs are bound to the dev session: SIGKILLing the session cancelled in-flight runs — executing and suspended alike (“Dev session ended”). Within a live session, waitpoint suspend/resume is durable and exactly-once. Deployed (cloud worker) recovery was not checked: deployments are unavailable to this organization on the free plan. |
|---|
| Evidence trail | Built in Via the CLI: per-run detail (status, timestamps, compute duration excluding wait time, cost) plus full run history and structured in-run logs. The REST run endpoints require the environment secret key — with a personal access token they returned 401/404. |
|---|
| Kill switch | Built in CLI cancel decided a run parked at its waitpoint (“Canceled by user”); completing its token afterwards returned success at the waitpoint level but the run stayed cancelled and the effect never fired. |
|---|
| Policy controls | Not claimed No policy engine. The credential boundary was exercised: missing, fabricated and wrong-class credentials each got 401s, and none of the refused calls executed anything. |
|---|
| Sandbox isolation | Not claimed Dev tasks execute in a local process with the operator machine’s filesystem access (observed directly). Cloud isolation was not checked (deployments unavailable on the free plan). |
|---|
Not in this battery
Two listings were not hands-on tested and keep their “From public docs” marks. E2B's hosted path requires an account step we have not completed (and its self-host instructions are a project of their own); Amazon Bedrock AgentCore is a managed AWS service with no self-serve test path available to us without an AWS account and spend. Upgrading either on paperwork alone would break the rule this page exists to keep: a Verified badge is only shown with published evidence.