Skip to content

Hands-on testing

We ran them. This is what happened.

A listing marked Verified was stood up in a throwaway environment and exercised against thesix criteria: a gated action was held for a decision, the process was killed mid-work, a completed action was replayed under the same key, the stop path was exercised, an out-of-scope call was attempted, and the evidence trail was inspected. These are the observed results — including what failed, and what we could not check, which is recorded as “Not checked” with the reason, never inferred from documentation.

Battery run 2026-11-15 by NextEleven. Environments were throwaway and unprivileged; no vendor was notified in advance and no result was shared with a vendor before publication.

Restate

Tested 2026-11-15 · Server 1.8.1, TypeScript SDK · Self-hosted server and Node endpoint in a throwaway local environment.

Approval gatesBuilt in
An awakeable held the effect until an external resolve arrived; the effect then ran exactly once. The suspension is durable.
Restart recoveryBuilt in
Server SIGKILL mid-work and endpoint SIGKILL mid-work both resumed from the journal with each effect exactly once. Replaying under the same idempotency key returned PreviouslyAccepted — no second execution.
Evidence trailBuilt in
The full ordered journal (input, steps, the approval signal, output) is queryable via `restate sql`. Ordered and inspectable, but not hash-chained.
Kill switchBuilt in
Cancel decided a gated invocation — a later resolve of its awakeable was accepted by the ingress but had no effect. Kill decided in-flight work. The CLI refuses without --yes, a footgun worth knowing.
Policy controlsNot claimed
No policy layer observed; the server ran unauthenticated as-run.
Sandbox isolationNot claimed
Handlers run as host processes.

Temporal

Tested 2026-11-15 · Server 1.32.2 (dev server, file DB), Python SDK · Self-hosted dev server and worker in a throwaway local environment.

Approval gatesDocumented pattern
A workflow waited on a Signal with wait_condition; the effect ran exactly once after approval. There is no built-in approval object — the gate is workflow code on native primitives.
Restart recoveryBuilt in
Worker SIGKILL and server SIGKILL both resumed the workflow exactly-once from event history. A mid-activity kill re-ran that activity (at-least-once, as documented); the downstream effect still ran once.
Evidence trailBuilt in
Complete ordered event history per workflow via CLI and Web UI. Not hash-chained.
Kill switchBuilt in
Cancel and terminate both decide in-flight work; signals sent after a stop are refused.
Policy controlsNot claimed
No policy layer in the open-source server as run.
Sandbox isolationNot claimed
Workers run as host processes.

Findings

  • Duplicate workflow IDs re-execute by default. Starting a second workflow with the same ID after the first completed ran the effect a second time; the REJECT_DUPLICATE reuse policy is opt-in per start request and refused it when set. Verified both ways.

DBOS

Tested 2026-11-15 · SDK 3.5.0, PostgreSQL 16 · Library in a throwaway application backed by a local Postgres.

Approval gatesDocumented pattern
A workflow blocked on DBOS.recv until DBOSClient.send delivered the approval; the effect then ran exactly once.
Restart recoveryBuilt in
Executor SIGKILL mid-workflow: DBOS recovered it automatically (recovery_attempts = 2), completed steps were read back from stored outputs instead of re-executing, and the effect ran once.
Evidence trailBuilt in
The Postgres system schema is the trail: workflow_status plus one operation_outputs row per executed step, in order, queryable with plain SQL. Not hash-chained.
Kill switchBuilt in
Cancel decided both a gated workflow and a sleeping one. A later approval send was recorded in the database but the cancelled workflow never acted on it.
Policy controlsNot claimed
No policy or authorization layer exists in the library as run.
Sandbox isolationNot claimed
In-process library; workflows run in your application process.

Findings

  • Re-enqueueing under the same workflow ID is deduplicated by default — the opposite default from Temporal, verified in the same battery.

Inngest

Tested 2026-11-15 · Dev server 1.53.0 (--persist), SDK 4.26.2 · Self-hosted dev server and application in a throwaway local environment.

Approval gatesDocumented pattern
A function paused in waitForEvent (CEL match on the event payload) until the approval event arrived; the effect then ran exactly once.
Restart recoveryDocumented pattern
Split result: killing the app process mid-run recovered exactly-once (step memoization is server-side), but killing the dev server mid-run lost the in-flight run despite --persist — live queue state is held in memory. Production self-host mode (external Postgres/Redis) was not checked.
Evidence trailBuilt in
Persisted step-level history in the product’s own store: one row per step event with attempts, latency and payloads, plus traces/spans tables. Not hash-chained.
Kill switchDocumented pattern
Cancellation is expressed as a cancelOn event pattern in function configuration.
Policy controlsNot claimed
No policy layer observed as run.
Sandbox isolationNot claimed
Functions run as host processes.

Findings

  • A replayed trigger event under the same key was deduplicated by the function’s declared idempotency — one run, one effect.

LangGraph

Tested 2026-11-15 · Library 1.2.22 with SQLite checkpointer; dev API server 0.4.43 · Library and dev API in a throwaway local environment.

Approval gatesBuilt in
interrupt() is a runtime primitive: the graph parked mid-node, persisted state, and resumed to exactly-once on the resume command — in the library and over the HTTP API. On resume the interrupted node re-executes from its top (documented, observed).
Restart recoveryBuilt in
SIGKILL mid-node: a fresh process resumed the thread from the SQLite checkpointer at node granularity; completed nodes were not re-run and the effect ran once. The dev server’s in-memory runtime is a convenience, not the durable surface.
Evidence trailBuilt in
Full checkpoint history per thread (source, step, next nodes, checkpoint IDs) persisted alongside state snapshots. Not hash-chained.
Kill switchDocumented pattern
The API surface can cancel a run; the library mode has no stop.
Policy controlsNot claimed
No policy layer observed.
Sandbox isolationNot claimed
In-process library.

Findings

  • No submission idempotency: re-invoking a completed thread with the same input starts a new run. A thread_id is a checkpoint key, not an idempotency key.

Microsoft Agent Framework

Tested 2026-11-15 · agent-framework 1.26.0, FileCheckpointStorage · Workflow layer in a throwaway local environment.

Approval gatesBuilt in
request_info parked the run in IDLE_WITH_PENDING_REQUESTS; the driver process exited entirely, and a fresh process resumed from the checkpoint to exactly-once.
Restart recoveryBuilt in
SIGKILL inside an executor: a fresh process resumed from the latest file checkpoint at executor granularity and completed the run.
Evidence trailBuilt in
Checkpoint JSON documents carry an explicit previous_checkpoint_id chain, timestamps, the full message log and executor state — the run history is reconstructible from the storage directory. The chain is structural, not cryptographic.
Kill switchNot claimed
No stop in library mode.
Policy controlsNot checked
A ToolApprovalMiddleware with rules ships in the public API (verified present), but exercising it needs an LLM-client agent run, which this battery did not do.
Sandbox isolationNot claimed
In-process library.

Findings

  • Operator trap, verified: WorkflowBuilder without an explicit name= gets a fresh random UUID name on every build, and checkpoints are stored per workflow name — with defaults, a new process cannot find the previous process’s checkpoints. Setting a stable name makes cross-process resume work.
  • No submission idempotency: re-running with identical input starts a new run.

agentgateway

Tested 2026-11-15 · v1.7.0 (release binary, sha256 verified) · Standalone gateway in front of a local MCP backend, throwaway configuration.

Approval gatesNot claimed
Requests are allowed or denied synchronously; there is no hold-for-approval primitive.
Restart recoveryNot claimed
In-flight requests are not recovered: killing the gateway mid-call cut the client stream and the result was never delivered (work already dispatched to the backend completed anyway). MCP sessions do survive a restart — session state rides in the encrypted session ID.
Evidence trailBuilt in
Structured per-request logs, including denials with the reason (Authorization).
Kill switchDocumented pattern
Killing the gateway stops routing; it does not cancel work already dispatched to a backend.
Policy controlsBuilt in
The strongest policy surface in this battery: CEL rules filtered tools/list per identity, and an unauthorized call was refused before the backend ever saw it. Denials are logged with reasons.
Sandbox isolationNot claimed
The gateway is a proxy; it provides no execution sandbox.

Findings

  • jwtAuth defaults to mode: optional. Configured from the project’s own example, an anonymous caller could list and call any tool whose rule does not reference JWT claims (tools whose rules do reference claims stayed closed). The one-line fix — mode: strict — was verified to close it: anonymous and garbage tokens get 401, a valid token gets through.

NeMo Guardrails

Tested 2026-11-15 · 0.24.1 · Library exercised with the vendor’s own scripted FakeLLMModel test double — call counts are exact by construction.

Approval gatesNot claimed
Rails decide synchronously from model verdicts; there is no hold-for-human state.
Restart recoveryNot claimed
Stateless per call — there is no run state to recover. A filter, not an engine.
Evidence trailNot checked
No structured rail-decision record was captured in this run.
Kill switchNot claimed
No operator stop; rails are synchronous checks.
Policy controlsBuilt in
Input self-check refused a request before generation (exactly 1 LLM call — generation never attempted). Output self-check replaced a reply that leaked a secret with the refusal (3 calls). Clean traffic passed verbatim in both directions.
Sandbox isolationNot claimed
In-process library.

ToolHive

Tested 2026-11-15 · v0.65.0 (release binary, sha256 verified) · Docker Desktop on a workstation; the registry fetch server as the test workload, driven through ToolHive’s proxy with raw JSON-RPC.

Approval gatesNot claimed
No human-approval primitive in the run surface.
Restart recoveryBuilt in
Killing the server container mid-stable-run: the supervisor detected the exit, restarted the full workload (attempt counters, backoff, maxRetries 10), and the workload served real calls again on the same port in under a minute. During the window the egress proxy answers with its own error page, not an MCP error. A kill delivered during initial startup failed its restart attempt — startup-phase kills are handled worse.
Evidence trailDocumented pattern
Structured per-workload logs record exits (with the server’s last logs embedded), restart attempts and auth warnings. Local log files; no queryable audit store observed.
Kill switchBuilt in
thv stop and rm tore workloads down cleanly in every phase (verified by container listing afterwards). A stop issued inside a restart window races the restart.
Policy controlsDocumented pattern
OIDC authentication and authorization policies exist as configuration, but the default run is unauthenticated — see the finding below. The enforcing configuration was not exercised.
Sandbox isolationBuilt in
Every workload is four containers: the MCP server, an ingress proxy, a Squid egress proxy and a DNS sidecar. The server has no direct network path; egress is proxied by construction.

Findings

  • Default posture, from the product’s own log on restart: “no authentication configured: the proxy will accept every request unauthenticated and assign each a synthetic local-user identity with no token or credential check; configure an OIDC provider to enforce authentication per request.”
  • thv list reports “No MCP servers found” transiently during any (re)start — a list call that lands in a startup window looks like an empty registry.

Docker MCP Gateway

Tested 2026-11-15 · Gateway v2.0.1 via Docker Desktop 29.4.3; catalog of 313 servers · Docker Desktop on a workstation; the catalog fetch server in a test profile.

Approval gatesNot claimed
The gateway is allow/deny by configuration; no human-approval primitive observed.
Restart recoveryNot claimed
The gateway is a stateless stdio proxy and servers are per-session containers started with --rm; there is no in-flight run state to recover. Profiles and catalog state persist on disk across sessions.
Evidence trailNot checked
Verbose console logging was observed; no durable, queryable audit record of calls was exercised.
Kill switchDocumented pattern
Ending the gateway process ends the session and its containers.
Policy controlsBuilt in
Per-tool allowlists per profile, enforced at list time and verified in both directions: disabling fetch.fetch left 0 tools listed in a fresh session; re-enabling restored it, and a real call then round-tripped through the gateway.
Sandbox isolationBuilt in
Servers run as hardened containers: --security-opt no-new-privileges, CPU and memory caps, and secrets kept out of the server environment (block-secrets defaults on; secrets resolve through the Docker Desktop store).

Findings

  • Catalog reference detail: the working catalog ref is mcp/docker-mcp-catalog:latest; the docker/mcp-catalog ref returns UNAUTHORIZED.

Langfuse

Tested 2026-11-15 · Server 4.80.0 (self-hosted compose stack), Python SDK 5.2.0 · Full compose stack (web, worker, Postgres, ClickHouse, Redis, object storage) on a workstation, with throwaway project keys.

Approval gatesNot claimed
Not applicable — Langfuse is an observability platform with no execution surface to gate.
Restart recoveryNot claimed
Langfuse runs no agent work, so the criterion does not apply as defined. What the battery verified instead is under Evidence: the stored trail survives restarts.
Evidence trailBuilt in
End to end: traces enter via OTEL, persist in ClickHouse (events_core), and survive web and worker restarts — verified twice, including a trace ingested before a restart. Scores are first-class, timestamped, trace-linked verdict records. Caveat: v4 runs in events_only write mode — the legacy ingestion endpoint refuses trace-create and legacy REST trace read-back is unavailable; the trail is queried via the events tables.
Kill switchNot claimed
Not applicable — no execution surface to stop.
Policy controlsNot claimed
Not applicable as an execution control. Ingest authentication was observed: the OTEL endpoint validates project keys (authCheck validKey).
Sandbox isolationNot claimed
Not applicable — Langfuse executes no code or tools.

Trigger.dev

Tested 2026-11-15 · CLI and SDK 4.8.5 · Cloud free plan, dev environment; tasks ran against a project created for this battery. Every run cost $0.0000; no payment card exists on the account.

Approval gatesBuilt in
wait.forToken held a run for ~2.5 minutes; completing the token externally fired the effect exactly once, and pre-wait steps were not re-executed. Note: wait.forEvent does not exist in SDK 4.8.5 (the installed API surface was inspected directly).
Restart recoveryNot claimed
In dev, runs are bound to the dev session: SIGKILLing the session cancelled in-flight runs — executing and suspended alike (“Dev session ended”). Within a live session, waitpoint suspend/resume is durable and exactly-once. Deployed (cloud worker) recovery was not checked: deployments are unavailable to this organization on the free plan.
Evidence trailBuilt in
Via the CLI: per-run detail (status, timestamps, compute duration excluding wait time, cost) plus full run history and structured in-run logs. The REST run endpoints require the environment secret key — with a personal access token they returned 401/404.
Kill switchBuilt in
CLI cancel decided a run parked at its waitpoint (“Canceled by user”); completing its token afterwards returned success at the waitpoint level but the run stayed cancelled and the effect never fired.
Policy controlsNot claimed
No policy engine. The credential boundary was exercised: missing, fabricated and wrong-class credentials each got 401s, and none of the refused calls executed anything.
Sandbox isolationNot claimed
Dev tasks execute in a local process with the operator machine’s filesystem access (observed directly). Cloud isolation was not checked (deployments unavailable on the free plan).

Not in this battery

Two listings were not hands-on tested and keep their “From public docs” marks. E2B's hosted path requires an account step we have not completed (and its self-host instructions are a project of their own); Amazon Bedrock AgentCore is a managed AWS service with no self-serve test path available to us without an AWS account and spend. Upgrading either on paperwork alone would break the rule this page exists to keep: a Verified badge is only shown with published evidence.