workflow‑llm‑orchestrator

Plan → validate → execute → verify → replan. A live harness for it.

An LLM is asked for a plan, never an action. The runtime decides whether that plan is legal, runs the steps that can run in parallel, checks the results, and repairs whatever was wrong — keeping the work that already succeeded.

Change any part of that pipeline below and run the same eval suite CI runs. Every plan, every step, every verdict and every trace event is inspectable. There is no backend — the whole engine is 38 kB of dependency-free JavaScript running in this tab.

plan→ validate→ execute→ verify→ answer ↩ replan

Leaderboard

Six shipped configurations, each changing exactly one thing about the pipeline, plus anything you build. Press run and the harness executes right here in the page, one task at a time, filling in as it goes.

Variant Tasks passed Plan valid Replans Caught Calls Tokens ms

Build a variant

Every field here maps to a real knob on the orchestrator. Nothing is simulated — the configuration you assemble is handed to the same Orchestrator class the CLI and CI use, running here in the page.

Identity
Model
Tools the planner is shown exactly this list — there are no hidden tools
Verification

Planning & repair
Execution
Stage prompts this is where a wrapper's own behaviour lives

Three LLM calls happen in a run and each takes its own instructions. Swap any of them, rerun the suite, read the delta. Leave one untouched to use the built-in.

Turns the goal into a plan. The single biggest lever on plan quality.

Tasks to run
Your own tasks test the workflow you actually care about

The five tasks above measure this engine. These measure yours. Give a goal and how a correct answer is recognised — grading is a deterministic check, never a model, because an eval that grades itself with an LLM is measuring the grader too.

Your own files seeded into the workspace before every task

Anything here is written into the sandbox before each run, so your tasks have something to read. The workspace is wiped and re-seeded between tasks, so no run can contaminate the next.

Playground

One goal, one run, full detail. No grading here — there is no expected answer for a goal you just typed, so what you get is the plan, the steps, the verdicts and the trace.

What the harness measures

Five tasks, each probing something specific. Grading is deterministic code, never a model — an eval that grades itself with an LLM is measuring the grader as much as the system.

Fixtures

The workspace is re-seeded from scratch before every single task, so no run can be contaminated by the one before it. In the browser it is a plain Map; under Node it is a real directory, and the sandbox rules are shared code rather than two implementations. Outbound network access is off here, so http_get is served from these fixtures only.

Two provider modes, two different measurements

mock

A scripted fixture, not a model. Measures the engine: dependency resolution, the validator rejecting bad plans, the verifier catching a wrong answer, replanning recovering, the budget holding. Free, offline, identical every run — which is what makes it something CI can gate on.

openrouter

Measures the model: how often its first plan is valid, how many steps it uses, how often it needs repairing, what it costs. One agentic run tells you almost nothing, so repeat it.

Conflating these two is the standard mistake in agent benchmarks — it reports model variance as though it were system behaviour.