Agent Lifecycle Kit

Eval-Driven Development

Test the tools your agents call

Wrong tool, made-up args, endless retries—those failures don’t look like stack traces. EDD gives you a red→green→refactor loop for prompts, MCP schemas, and routing, with CI gates you can trust.

Same loop you already trust for code

If an agent can call tools, guessing isn’t a test plan. Start with a failing eval for the behavior you care about, wire the tool contract and prompt until it passes, then tighten until CI holds—and drop production misses back into the suite.

  1. Red

    Write the intent and metrics (tool, schema, judge).

  2. Green

    Register the MCP schema and a minimal prompt; run until it passes.

  3. Refactor

    Tighten wording; block merges below 95% routing accuracy.

EDD closed loop
flowchart LR
  intent[Agent intent] --> edd[Red / green / refactor]
  edd --> ci[CI threshold gate]
  ci --> ship[Ship]
  ship --> prod[Prod traces + shadow judge]
  prod -->|miss| edd

Try it locally

kit eval
# Works offline with the scripted driver — no API key
$ kit eval run --suite evals/edd/architecture_routing.yaml --model scripted

# Fail the PR if routing accuracy drops under 95%
$ kit eval ci --threshold-routing 95 --out out/reports

# Write a markdown failure report for the PR
$ kit eval report --format md --out out/reports

Questions

What is Eval-Driven Development?

EDD is TDD for agents that call tools. You write a failing eval for the tool and arguments you expect, implement the schema and prompt until it passes, then tighten until CI holds.

How is EDD different from eyeballing prompts?

Each case starts from a fresh context. Tools are mocked, so you measure routing and extraction rather than network luck. Asserts cover JSON schema match plus an optional LLM-as-a-judge. CI can block the merge when routing accuracy drops.

How do I run EDD in CI?

Run kit eval ci --threshold-routing 95 --out out/reports. The scripted driver works offline with no API key. Live models can be used when KIT_EVAL_API_KEY or OPENAI_API_KEY is set.

What happens after a production miss?

Turn the miss into a JSONL case and add it to the suite. Shadow evals can sample live traffic so the next failure is a test, not a surprise.

The rest of the kit

EDD is how you prove tool calls. When the job is a product feature, the orchestrator still routes work through specialist roles: grilling, spec, TDD, cross-functional quality, audit, telemetry, and release. Language and framework profiles load on top of that once the stack is known.

Orchestrator routing for a feature
sequenceDiagram
  autonumber
  participant O as orchestrator
  participant G as grilling
  participant S as spec
  participant T as tdd
  participant X as xfn
  participant Sec as security
  participant Arch as arch-drift
  participant Tel as telemetry
  participant R as release

  O->>G: Stress-test idea and decision frontier
  O->>S: BDD spec and acceptance criteria
  O->>T: Inventory catalog and plan test impact
  O->>X: Cross-functional quality matrix
  O->>T: TDD short loop (gear 1 + gear 2)
  O->>X: Green apply-row XFN suites
  O->>Sec: Security and OWASP audit
  O->>Arch: Hexagonal boundaries, no drift
  O->>Tel: Map SLOs to OpenTelemetry
  O->>R: Conventional PR title and handover
  1. Grilling: If the idea is still mushy, interview until the decision frontier is clear.
  2. Spec: Gherkin and acceptance criteria, including cross-functional rows.
  3. TDD: Inventory the behavior catalog, then gear 1 (domain) and gear 2 (thin adapters) in the same loop. EDD lives here when the change is a prompt or tool schema.
  4. XFN: Green the apply rows (accessibility, load, security) or skip with a reason.
  5. Audit: Security and architecture-drift checks, then pre-commit.
  6. Telemetry and release: Map SLOs, update public docs if you touched them, ship with a conventional PR title.

Orchestrator skill · Coding philosophy

The guide

Connecting an LLM to MCP tools, APIs, or terminals turns a chatbot into a decision-maker. Failures rarely look like stack traces. They look like a wrong tool, a hallucinated parameter, or an infinite retry loop.

EDD treats prompts and tool schemas as version-controlled, evaluated contracts:

  1. Red — JSONL cases and YAML metrics assert the tool (and arguments) you expect.
  2. Green — Register the MCP contract and minimal system instructions. Run until asserts pass.
  3. Refactor — Iterate descriptions and constraints without breaking existing cases. Gate merges with kit eval ci --threshold-routing 95.

The full guide loads below. If it does not, read docs/edd.md.

Around the eval loop

What this kit is

Kit is for teams who are tired of shipping agents on good intentions. It centers Eval-Driven Development (EDD), then layers in lifecycle skills, architecture guardrails, and shared IDE rules so the rest of the workflow stays coherent.

Start here

Kit skills and profiles