What a harness review actually covers
Agent = model + harness. The loop is only the heartbeat. Reliability lives in everything wrapped around it: tools, context policy, permissions, verification, state, and traces. Review the model–harness pair, not the prompt in isolation. Layers below are ordered by how often production agents fail when a layer is missing.
0. Identity & blast radiusWho is the agent, which credentials it holds, what a compromise can doP0
1. Control loopReason → act → observe → repeat, plus hard stop conditionsP0
2. Action interfaceTool registry, schemas, sandbox, least privilege, side-effect classP0
3. Guardrails & HITLCode-enforced limits, approval gates, input/output policyP0
4. Verify stepDo not trust “I’m done.” Check environment, tests, schema, or a judgeP1
5. Context policyWhat the model sees each turn; pin vs compact vs drop; tool-output capsP1
6. Graph / orchestrationNodes, routers, subagents, handoffs — only if work actually branchesP1
7. State & memorySession, artifacts, resume, rollback, append-only auditP1
8. ObservabilityEvery model call and tool call as a reconstructable trajectoryP2
9. Eval / feedbackFixed eval set, failure clustering, policy generated from real missesP2
Per-layer questions that catch real failures
Loop
- What starts a run? User message, schedule, webhook, another agent?
- Hard caps: max iterations, max tokens, max wall-clock, max $?
- Exit: model says done, verifier passes, user stop, budget, error budget?
- Same-tool/same-args loop detector?
- Retries classified (timeout vs 4xx vs bad tool JSON vs policy deny)?
Tools
- Closed registry or the model can invent tools?
- Each tool: name, schema, side-effect (read / write / irreversible), auth identity
- Arguments validated in code before execution?
- Secrets injected by the harness, never by the model?
- Sandbox / network allowlist / working directory?
Guardrails
- Which rules are code (cannot be talked out of) vs prompt text (can)?
- Pre-tool permission pipeline: allow / ask-human / deny
- Input: injection, untrusted tool output labeled as untrusted
- Output: PII, send-message, payments, deletes
- Worst case if the harness is misconfigured?
Verify
- Ground truth from the environment (tests, query result, HTTP status)?
- Separate verifier from the actor when stakes are high?
- False “done” path: agent claims success after a no-op?
- Retry-from-failure feeds the actual error, not a paraphrase?
Context
- Pinned always: system, task, policy, current plan?
- Compaction: summarize middle, persist large tool output to files?
- Instruction files versioned (AGENTS.md / CLAUDE.md)?
- Tool descriptions costing tokens every turn — are they earned?
Graph
- Is this a while-loop with tools, or a real state machine?
- Which edges are deterministic vs model-routed?
- Cycles bounded? Checkpoint before irreversible nodes?
- Subagent: isolated context, reduced tools, summary-only return?
Rule of thumb: a graph is orchestration. A harness is governance plus verification. If someone shows you only a pretty node diagram, they have not shown you the harness.
How to ask an AI to present the harness
Do not ask “explain the architecture.” That produces a slide. Ask for a machine-readable dump against a fixed schema, then a human narrative of failure modes. Run the prompts against the repo, the n8n/LangGraph export, or the authoring model with the code in context.
Prompt 1 — extraction (use first, always)
Copy this. Force JSON only. Paste the result into Live board.
You are extracting the AGENT HARNESS of the system in context (code, graph, n8n flow, bot config, or your own runtime). Do not describe the business feature. Describe the software around the model.
Return ONLY valid JSON matching this schema. If a field is unknown, use null and add it to "unknowns". Never invent a tool, gate, or limit that is not in the source.
{
"agent_name": "",
"purpose": "",
"runtime": "langgraph | n8n | custom-loop | claude-agent-sdk | openai-agents | crew | other",
"model": { "primary": "", "routed_models": [], "who_chooses": "fixed | harness | model" },
"loop": {
"pattern": "react | plan-act-verify | graph | supervisor-workers | other",
"entry": "",
"max_iterations": null,
"max_tokens": null,
"max_wall_clock_s": null,
"max_cost_usd": null,
"exit_conditions": [],
"loop_detection": "",
"retry_policy": ""
},
"graph": {
"nodes": [{ "id": "", "kind": "model | tool | gate | verify | human | router | deterministic", "does": "", "can_fail_open": false }],
"edges": [{ "from": "", "to": "", "when": "", "deterministic": true }],
"cycles_allowed": false,
"checkpointing": ""
},
"tools": [{
"name": "",
"side_effect": "read | write | irreversible | network | code-exec",
"auth_identity": "",
"sandbox": "",
"schema_validated_in_code": false,
"approval": "never | always | on-risk | policy",
"can_be_invented_by_model": false
}],
"guardrails": {
"code_enforced": [],
"prompt_only": [],
"input_filters": [],
"output_filters": [],
"human_in_the_loop": [],
"untrusted_content_labeled": false
},
"verify": {
"exists": false,
"method": "none | tests | schema | env-ground-truth | llm-judge | human",
"blocks_done_claim": false,
"retry_on_fail": false
},
"context": {
"pinned": [],
"compaction": "",
"tool_output_cap": "",
"instruction_files": [],
"memory": "none | session | cross-session | both"
},
"state": {
"store": "",
"resume": false,
"rollback": false,
"audit_log": "none | partial | append-only"
},
"observability": {
"traces_every_tool": false,
"traces_every_model_call": false,
"cost_latency": false,
"export": ""
},
"identities": [{ "name": "", "scopes": "", "shared_across_agents": false }],
"unknowns": [],
"top_failure_modes": []
}
After the JSON, stop. No markdown.
Prompt 2 — adversarial review
Using only the harness JSON and source you can see, attack the design. For each finding use: - id - layer (loop|tools|guardrails|verify|context|graph|state|identity|observability) - severity (P0|P1|P2) - failure_mode (what goes wrong in production) - exploit_or_drift (how a model, user, or tool output causes it) - evidence (file/node/config) - fix (smallest code-level change, not a prompt tweak unless the hole is prompt-only) Required scenarios: 1) Agent claims success after doing nothing material 2) Unbounded loop / cost runaway 3) Prompt injection via tool output or retrieved doc 4) Write/send/delete without approval 5) Shared credential / over-scoped token 6) Context overflow drops the task or the policy 7) Subagent or tool returns a lie that the parent trusts 8) Resume after crash leaves partial mutation Sort P0 first. If a scenario cannot happen, say why with evidence.
Prompt 3 — draw it, don’t narrate it
Produce three diagrams as mermaid only. A) LOOP: start -> model -> (tool|respond|handoff|stop) including caps and verify. B) GRAPH: nodes and labeled edges. Mark deterministic edges vs model-routed. Mark gates. C) TRUST BOUNDARY: user, untrusted content, model, harness policy, tools, secrets, external systems. Rules: no decoration, every node must exist in the source, annotate max_iterations on the loop back-edge.
Prompt 4 — permission matrix
Build a table of every tool and every side-effecting API the harness can reach. Columns: tool, system touched, identity used, side_effect, pre-check (schema/authz), approval gate, post-check (verify), logged fields, residual risk. Then list actions that are irreversible and confirm each has: checkpoint, approval, and rollback or compensating action. If any irreversible action has none of those, mark P0.
One-shot for a sloppy repo
Scan this repo / flow for the agent harness. Ignore product copy. Output: 1) JSON dump (schema from Prompt 1) 2) List of while/graph loops with their stop conditions 3) List of tools and whether args are validated outside the model 4) Every string that looks like a policy but is only in a prompt 5) Missing: iteration cap, verify, audit log, sandbox, HITL, compaction 6) The three cheapest fixes that raise reliability the most
Dump schema + sample
Paste extracted JSON on the Live board tab. The sample below is a field-service dispatch agent — swap it for the agent under review.
{
"agent_name": "Support Triage Agent",
"purpose": "Read an inbox queue and draft replies for a human to send",
"runtime": "langgraph",
"model": { "primary": "gpt-4.1", "routed_models": ["haiku-class-router"], "who_chooses": "harness" },
"loop": {
"pattern": "plan-act-verify",
"entry": "scheduled every 15m + chat mention",
"max_iterations": 12,
"max_tokens": 120000,
"max_wall_clock_s": 180,
"max_cost_usd": 1.5,
"exit_conditions": ["verifier.pass", "budget", "human_stop"],
"loop_detection": "hash(tool+args) repeats 3x => abort",
"retry_policy": "timeout x2; 4xx no-retry; schema-error repair x1"
},
"graph": {
"nodes": [
{ "id": "ingest", "kind": "deterministic", "does": "pull overdue WOs", "can_fail_open": false },
{ "id": "plan", "kind": "model", "does": "rank and propose actions", "can_fail_open": false },
{ "id": "act", "kind": "tool", "does": "ticket and chat tools", "can_fail_open": false },
{ "id": "gate_write", "kind": "gate", "does": "approve SF updates", "can_fail_open": false },
{ "id": "verify", "kind": "verify", "does": "re-query ticket status", "can_fail_open": false },
{ "id": "human", "kind": "human", "does": "operator confirm", "can_fail_open": false }
],
"edges": [
{ "from": "ingest", "to": "plan", "when": "rows>0", "deterministic": true },
{ "from": "plan", "to": "gate_write", "when": "action is write", "deterministic": false },
{ "from": "gate_write", "to": "act", "when": "approved", "deterministic": true },
{ "from": "act", "to": "verify", "when": "tool returned", "deterministic": true },
{ "from": "verify", "to": "plan", "when": "fail and iters left", "deterministic": true },
{ "from": "verify", "to": "human", "when": "ambiguous", "deterministic": true }
],
"cycles_allowed": true,
"checkpointing": "langgraph thread + postgres"
},
"tools": [
{ "name": "read_ticket", "side_effect": "read", "auth_identity": "helpdesk_ro", "sandbox": "api", "schema_validated_in_code": true, "approval": "never", "can_be_invented_by_model": false },
{ "name": "update_ticket", "side_effect": "write", "auth_identity": "helpdesk_rw", "sandbox": "api", "schema_validated_in_code": true, "approval": "on-risk", "can_be_invented_by_model": false },
{ "name": "post_update", "side_effect": "network", "auth_identity": "chat_bot", "sandbox": "api", "schema_validated_in_code": true, "approval": "always", "can_be_invented_by_model": false },
{ "name": "web_search", "side_effect": "network", "auth_identity": "none", "sandbox": "allowlist", "schema_validated_in_code": true, "approval": "never", "can_be_invented_by_model": false }
],
"guardrails": {
"code_enforced": ["max 12 iters", "reads stay read-only", "no delete tools", "outbound only to the support channel"],
"prompt_only": ["be concise", "do not invent techs"],
"input_filters": ["strip email bodies over 8k"],
"output_filters": ["redact phone outside SF fields"],
"human_in_the_loop": ["owner change", "any outbound post"],
"untrusted_content_labeled": true
},
"verify": {
"exists": true,
"method": "env-ground-truth",
"blocks_done_claim": true,
"retry_on_fail": true
},
"context": {
"pinned": ["policy", "current queue snapshot", "tool schemas"],
"compaction": "summarize steps 1..n-4; keep last 4 raw",
"tool_output_cap": "8k chars then file-offload",
"instruction_files": ["AGENTS.md"],
"memory": "session"
},
"state": {
"store": "postgres checkpoints",
"resume": true,
"rollback": false,
"audit_log": "append-only"
},
"observability": {
"traces_every_tool": true,
"traces_every_model_call": true,
"cost_latency": true,
"export": "otel -> langsmith"
},
"identities": [
{ "name": "helpdesk_ro", "scopes": "ticket read", "shared_across_agents": false },
{ "name": "helpdesk_rw", "scopes": "ticket update", "shared_across_agents": false }
],
"unknowns": ["exact chat scopes"],
"top_failure_modes": ["stale queue snapshot after compact", "owner-change needs human but graph can skip if router misfires"]
}
Live board
Paste Prompt 1 JSON. The other tabs render from this object. Invalid JSON stays on screen with the parse error.
No dump loaded
Loop map
The only loop that matters is the one the runtime actually runs. Caps and verify must sit on the back-edge, not in a prompt.
Graph / roles
Deterministic edges should be the majority on anything irreversible. Model-routed edges need a gate or a verify immediately after.
Guardrail matrix
Prompt-only rules are wishes. Code-enforced rules are the harness. Review the gap between the two.
Scorecard
Scored from the dump. Unknowns count against the layer. This is a review aid, not a certification.
Efficient review walkthrough
Thirty minutes, not a workshop. Use this order so you do not get lost in the graph pretty-print.
0–5 min — inventory
Run Prompt 1 against the repo or the builder. Confirm runtime, model, entrypoint, tool count, whether a graph even exists.
5–10 min — stop conditions
Find the while / graph recursion / n8n loop. Demand numeric caps. If the only stop is “the model decides it’s done,” that’s a P0.
10–16 min — action surface
Every tool: side effect, identity, schema validation in code, approval. Flag god-tokens and model-invented tools.
16–22 min — verify + HITL
What checks the world after a write? What cannot proceed without a human? Try the “claimed done, did nothing” path.
22–26 min — context & injection
Is tool output labeled untrusted? Is policy pinned through compaction? Can retrieved text override system rules?
26–30 min — reconstructability
Could you replay one bad run from traces + audit log + checkpoints? If not, you cannot operate it.
What “good enough” looks like for a first production agent
Minimum viable harness
- Closed tool registry, schemas validated in code
- Iteration + cost + wall-clock caps
- Least-privilege identity, no shared god token
- Append-only tool/model audit log
- Approval on irreversible actions
- A verify step that can veto “done”
Defer until the agent earns it
- Multi-agent supervisor graphs
- Cross-session memory with write-back
- Model-routed tool invention / MCP free-for-all
- Self-modifying prompts or policy
- Unattended sends, payments, deletes
How to talk to the builder
Need the harness dump (JSON schema), not an architecture slide. Specifically: - the loop stop conditions in code - tool permission table - which policies are code vs prompt - the verify step that blocks a false done - how I replay a failed run Until those exist, this is a demo loop, not a harness.