← Back to resources

Resource · 8 pages

Build Your Own Agentic Coding Harness

A practical control plane for state, tools, permissions, context, execution, verification, and proof, designed so the AI model can be swapped without rewriting the system.

Read online · Download PDF

An agentic coding harness is the control system around a language model. The model proposes the next move. The harness decides what context it sees, which tools exist, whether an action is allowed, how the workspace changes, when the run stops, and what counts as proof.

This guide shows how to build that control system so you can change the underlying model without rewriting the rest of the product.

The honest compatibility claim: support any model that can reliably return text. Use native tool calling when a provider supports it. Use a constrained text protocol only as a degraded fallback. Model independence does not mean capability parity.

The mental model

USER TASK
    |
    v
RUN CONTROLLER ---> CONTEXT ENGINE ---> MODEL ADAPTER
    ^                                      |
    |                                      v
    |                              TEXT OR TOOL REQUEST
    |                                      |
    |                                      v
    +--- LEDGER <--- VERIFIER <--- POLICY + APPROVAL
                                      |
                                      v
                              SANDBOXED TOOL EXECUTOR
                                      |
                                      v
                              WORKSPACE + OBSERVATION

The model is one dependency. The harness is the product.

1. Start with a canonical model contract

Do not make the rest of the application understand every provider's message format. Normalize at the edge, but preserve capability differences explicitly.

export type HarnessMessage = {
  role: "system" | "user" | "assistant" | "tool";
  parts: Array<
    | { type: "text"; text: string }
    | { type: "tool_call"; id: string; name: string; args: unknown }
    | { type: "tool_result"; id: string; result: unknown; isError?: boolean }
  >;
};

export type ModelCapabilities = {
  nativeTools: boolean;
  parallelTools: boolean;
  structuredOutput: "strict" | "best_effort" | "none";
  streaming: boolean;
  maxInputTokens?: number;
  maxOutputTokens?: number;
};

export type ModelRequest = {
  messages: HarnessMessage[];
  tools: ToolDefinition[];
  maxOutputTokens: number;
  providerOptions?: Record<string, unknown>;
};

export type ModelEvent =
  | { type: "text_delta"; text: string }
  | { type: "tool_call"; id: string; name: string; args: unknown }
  | { type: "usage"; inputTokens?: number; outputTokens?: number }
  | { type: "done"; finishReason: string }
  | { type: "error"; error: HarnessError };

export interface ModelAdapter {
  id: string;
  capabilities(): Promise<ModelCapabilities>;
  generate(request: ModelRequest, signal: AbortSignal): AsyncIterable<ModelEvent>;
}

Build one adapter per provider or local runtime. The adapter translates provider-native events into ModelEvent and converts canonical tools back into the provider's expected schema.

The adapter rules

1. Keep provider SDK logic inside the adapter.
2. Preserve tool-call IDs. They connect the request, execution receipt, and returned observation.
3. Expose capability flags instead of pretending every model behaves the same.
4. Retry transport failures in the adapter. Do not silently retry side-effecting tools.
5. Record the requested model and the resolved model version when the provider returns it.
6. Pass provider-specific features through providerOptions; do not contaminate the common contract.

OpenAI, Anthropic, and Gemini all implement the same broad application-side cycle: provide tool definitions, receive a structured request, execute it in your application, return the result, and continue. Their exact message shapes and schema guarantees differ, which is why the adapter boundary matters. See OpenAI function calling, Anthropic tool use, and Gemini function calling.

Fallback for text-only models

If a model does not support native tools, ask it to emit exactly one action envelope:

{
  "kind": "tool_call",
  "tool": "workspace.read_file",
  "arguments": { "path": "src/app.ts" }
}

Parse it, validate it, and reject anything outside the envelope. Label this adapter structuredOutput: "best_effort" or "none". Never execute prose that merely looks like a command.

2. Make the run a deterministic state machine

The model should not own the loop. A persisted controller should.

CREATED
  -> PREPARING_CONTEXT
  -> WAITING_FOR_MODEL
  -> REVIEWING_ACTION
  -> WAITING_FOR_APPROVAL   (only when required)
  -> EXECUTING_TOOL
  -> VERIFYING
  -> COMPLETED | FAILED | CANCELLED | BUDGET_EXHAUSTED

A run record needs enough information to pause, resume, replay, and explain itself:

export type Run = {
  id: string;
  task: string;
  constraints: string[];
  workspace: { root: string; revision: string; writableRoots: string[] };
  adapterId: string;
  model: string;
  status: RunStatus;
  step: number;
  budgets: { maxSteps: number; maxMs: number; maxTokens?: number; maxCostUsd?: number };
  approvals: ApprovalRecord[];
  startedAt: string;
  updatedAt: string;
};

Persist every transition before beginning the next side effect. Give every proposed tool call an idempotency key. A process crash should resume from a known state, not guess what happened.

Stop conditions belong in code

Stop when any of these becomes true:

• The verifier accepts the completion claim.
• A hard budget is exhausted.
• The user cancels.
• A required approval is denied.
• The same recoverable failure repeats beyond its retry policy.
• The harness detects no progress, such as an unchanged diff across repeated write attempts.

3. Treat tools as typed capabilities

A tool is not a prompt snippet. It is a named capability with input and output contracts, a risk level, a timeout, and an executor.

export type ToolDefinition = {
  name: string;
  description: string;
  inputSchema: Record<string, unknown>;
  outputSchema?: Record<string, unknown>;
  risk: "read" | "write" | "execute" | "network" | "destructive";
  timeoutMs: number;
};

export interface HarnessTool {
  definition: ToolDefinition;
  execute(args: unknown, ctx: ToolContext): Promise<ToolResult>;
}

Use JSON Schema to validate both arguments and structured results. JSON Schema Draft 2020-12 is the current published version. The 2026-07-28 MCP tool specification also uses JSON Schema for inputSchema and optional outputSchema, and recommends timeouts plus audit logging. See JSON Schema 2020-12 and MCP tools.

Begin with seven tools

Prefer narrow tools over a universal shell. A narrow tool is easier to authorize, validate, test, and explain.

Keep MCP optional

MCP is a useful transport for discovering external tools and resources. It should plug into the tool registry, not become the internal architecture of the whole harness.

BUILT-IN TOOLS ----\
PLUGIN TOOLS -------+--> TOOL REGISTRY --> POLICY --> EXECUTOR
MCP TOOLS ----------/

The current MCP core is stateless. If an external tool needs durable state, pass an explicit opaque handle and validate authorization on every call. See the MCP 2026-07-28 release.

4. Put policy between intention and execution

Never send a model-generated action directly to an executor.

export type PolicyDecision =
  | { outcome: "allow"; ruleId: string }
  | { outcome: "require_approval"; ruleId: string; summary: string }
  | { outcome: "deny"; ruleId: string; reason: string };

Evaluate the proposed tool, validated arguments, workspace, user identity, current run, and prior approvals.

Approvals must be specific. Store the tool name, normalized arguments, scope, expiry, user, and decision. An approval for npm test is not approval for arbitrary shell access.

The MCP specification recommends a human ability to deny tool invocations. OWASP guidance adds schema enforcement, least privilege, sandboxing, restricted network access, and audit trails. See MCP tools: user interaction and OWASP Agentic AI threats and mitigations.

5. Isolate the workspace and executor

Assume generated commands and retrieved content are untrusted.

The minimum safe execution boundary should provide:

• One explicit workspace root.
• A clean snapshot or branch before writes.
• Read-only mounts except for declared writable roots.
• No secrets by default.
• Network disabled by default.
• A non-root user.
• Per-command timeout, process-tree termination, and output limits.
• CPU, memory, disk, and process limits.
• A diff after every write phase.
• A disposable environment for generated code when risk warrants it.

Containers are useful isolation, but they are not automatically a security boundary. Choose containers, microVMs, WebAssembly, or a managed sandbox according to the consequence of escape. OWASP's Securing Agentic Applications Guide discusses these isolation choices and the principle of least privilege.

Patch, do not rewrite blindly

Use structured patches for code changes:

1. Read the target file and record its content hash.
2. Ask the model for a patch against that exact version.
3. Reject the patch if the file changed after the read.
4. Apply inside allowed roots only.
5. Return the resulting diff as the tool observation.
6. Let the verifier decide whether the change is acceptable.

This makes conflicts visible and creates a compact receipt for review.

6. Build context as a budgeted product

Do not dump a repository into the prompt. Assemble context in layers.

Track where every context item came from. Treat repository files, tool outputs, search results, and issue text as data, not trusted instructions.

Compaction invariants

A summary must retain:

• The user's exact task and constraints.
• The current workspace revision.
• Files changed and why.
• Commands run with exit codes.
• Unresolved test failures.
• Approval decisions and their scope.
• Tool receipts needed to replay the run.

If the summary cannot preserve those invariants, stop and ask for a larger context budget rather than fabricating continuity.

7. Implement the loop

while (!terminal(run.status)) {
  enforceBudgets(run);

  const context = await contextEngine.build(run.id);
  ledger.append("model.requested", summarizeRequest(context));

  const response = await collect(adapter.generate(context.request, abortSignal));
  ledger.append("model.responded", response.receipt);

  if (response.toolCalls.length === 0) {
    const verdict = await verifier.verifyCompletion(run, response.text);
    ledger.append("verification.finished", verdict);
    run.status = verdict.ok ? "COMPLETED" : "PREPARING_CONTEXT";
    continue;
  }

  for (const call of response.toolCalls) {
    const tool = registry.require(call.name);
    const args = schemas.validate(tool.definition.inputSchema, call.args);
    const decision = await policy.evaluate(run, tool.definition, args);
    ledger.append("tool.proposed", { call, decision });

    if (decision.outcome === "deny") {
      observations.add(toolDenied(call, decision.reason));
      continue;
    }

    if (decision.outcome === "require_approval") {
      const approval = await approvals.pauseAndWait(run, call, decision);
      if (!approval.granted) {
        observations.add(toolDenied(call, "User denied approval"));
        continue;
      }
    }

    const result = await executor.run(tool, args, run.workspace);
    schemas.validateOptional(tool.definition.outputSchema, result.structured);
    ledger.append("tool.finished", result.receipt);
    observations.add(toolObservation(call.id, result));
  }

  run.step += 1;
  run.status = "PREPARING_CONTEXT";
}

Important details:

• Collect and validate the full tool request before execution, even when the model streams it.
• If parallel tool calls are allowed, parallelize reads first. Serialize writes unless independence is proven.
• Send tool errors back as structured observations so the model can recover.
• Do not turn a model's final sentence into success without verification.

8. Record an append-only run ledger

The ledger is the source of truth for debugging and replay.

run.created
context.built
model.requested
model.responded
tool.proposed
approval.requested
approval.decided
tool.started
tool.finished
verification.started
verification.finished
run.completed

Each event should include a timestamp, run ID, sequence number, event version, parent event, redacted payload or hash, and trace identifiers. Never log secrets or raw sensitive prompts by default.

For telemetry, align spans with the developing OpenTelemetry GenAI conventions: invoke_agent, model inference, execute_tool, and plan when you can identify an actual planning phase. See OpenTelemetry GenAI agent spans.

Replay has two modes

• Deterministic replay: reuse stored model and tool outputs to reproduce controller decisions.
• Live replay: rerun the same task against a selected model and compare outcomes.

Do not promise identical model text. Promise that the control decisions and evidence are inspectable.

9. Verify independently

The verifier should not ask the same model, "Did you finish?" and accept yes.

Use a pipeline of deterministic checks first:

1. The diff is inside the requested scope.
2. Required files exist.
3. Formatting, lint, typecheck, tests, and build return recorded exit codes.
4. No secret or policy scan regressed.
5. The user acceptance criteria map to explicit evidence.
6. The workspace is in a reviewable state.

Use a second model grader only for criteria that cannot be expressed deterministically, and keep its evidence separate.

type VerificationReport = {
  ok: boolean;
  checks: Array<{
    name: string;
    status: "pass" | "fail" | "skipped";
    evidence: string;
  }>;
  changedFiles: string[];
  unresolvedFailures: string[];
};

10. Build it in five increments

Increment 1: Read-only investigator

• One provider adapter.
• list, read_file, and search tools.
• Run state, budgets, and a JSONL ledger.
• No writes and no shell.

Exit test: the agent answers a repository question and every claim links to a file and line range.

Increment 2: Patch author

• Add structured patch application.
• Enforce workspace roots and content hashes.
• Show a diff after every write.

Exit test: a stale-file patch is rejected without modifying the workspace.

Increment 3: Guarded executor

• Add named verification commands.
• Add timeouts, output limits, process cleanup, and approval gates.
• Disable network by default.

Exit test: an unapproved network or out-of-root action cannot run.

Increment 4: Resumable runs

• Persist state transitions and approval requests.
• Add deterministic replay.
• Add cancellation and crash recovery.

Exit test: kill the process while waiting for approval, restart it, and resume exactly once.

Increment 5: Provider matrix and evals

• Add at least two more adapters.
• Run the same fixture suite across providers.
• Compare task success, unsafe-action rate, tool recovery, cost, latency, and regression rate.

Exit test: changing one configuration value swaps the model without changing tools, policies, verification, or the fixture suite.

Suggested repository structure

coding-harness/
  src/
    core/
      types.ts
      controller.ts
      state-machine.ts
      budgets.ts
    adapters/
      model-adapter.ts
      openai-adapter.ts
      anthropic-adapter.ts
      gemini-adapter.ts
      text-protocol-adapter.ts
    context/
      builder.ts
      compactor.ts
      provenance.ts
    tools/
      registry.ts
      schemas.ts
      workspace-tools.ts
      shell-tool.ts
    policy/
      engine.ts
      approvals.ts
      rules.ts
    runtime/
      sandbox.ts
      process-runner.ts
      workspace.ts
    ledger/
      events.ts
      store.ts
      replay.ts
    verify/
      pipeline.ts
      checks.ts
    cli.ts
  fixtures/
    repositories/
    tasks/
    expected/
  tests/
    adapters/
    policy/
    recovery/
    evals/

The evaluation suite

Create small seeded repositories and run the same tasks repeatedly.

Normal cases

• Find and explain a bug without writing.
• Add a small function and its tests.
• Update a dependency within an allowed range.
• Refactor a module without changing behavior.

Failure injections

• Malformed tool arguments.
• Unknown tool name.
• Model timeout or truncated stream.
• Patch conflict after a file changes.
• Tool timeout and oversized output.
• Test failure after a plausible patch.
• Approval denial.
• Network disabled.
• Process crash during approval wait.
• Prompt injection hidden in a repository file.

Metrics

Definition of done

Your first credible harness is ready when:

• [ ] A model can be swapped through configuration.
• [ ] Capability differences are exposed, not hidden.
• [ ] Every tool input is schema-validated before execution.
• [ ] Every write stays inside declared roots.
• [ ] Every command has a timeout, exit code, output cap, and process cleanup.
• [ ] Network access and destructive actions require explicit policy decisions.
• [ ] Runs can pause for approval and resume once.
• [ ] Every action creates a redacted receipt.
• [ ] A deterministic verifier can reject a false completion.
• [ ] The fixture suite runs unchanged across at least three adapters.

System proof rubric

Only call it Ready to ship when all four have evidence attached.

Primary references

• OpenAI function calling
• Anthropic tool use
• Gemini function calling
• Model Context Protocol 2026-07-28 tools
• MCP 2026-07-28 release notes
• JSON Schema Draft 2020-12
• OpenTelemetry GenAI agent spans
• OWASP Agentic AI threats and mitigations
• OWASP Securing Agentic Applications Guide
• NIST Generative AI Profile

Raw Markdown