# Agent DOE Engine

Tests several agent settings in one planned batch and reports which changes are real, which are noise, and the best combination when goals compete.

Status: active

## The Problem

Tuning an agent the usual way fails twice. One-at-a-time testing is slow and blind — it misses settings that only matter in combination (a bigger batch size that only helps once you also add workers). And a single run can fool you: every run carries random variation, so a number that looks better might just be noise.

## What I Built

> Part of the design-of-experiments optimization pattern (see /toolkit/patterns/design-of-experiments) — here: design size is chosen automatically from the count of settings under test (2–3 → full factorial, 4–7 → fractional factorial, 8–11 → Plackett-Burman), and competing agent goals are resolved by whichever of three methods the run calls for — scalarize, desirability, or pareto — rather than a single fixed objective function.

A design-of-experiments engine that varies several settings together in one planned batch, then tells you which changes are real. You list the settings to test and the results you care about — speed, cost, quality, accuracy — and for each result it reports which settings moved it and by how much (including interaction-only effects), whether that movement is a real effect or a fluke, which effects are mathematically tangled in this design, and the single best configuration when goals compete.

## Architecture

agent-doe-engine is a deterministic statistics engine, not a model-calling one. The matrix generation, sampled measurement, effects fitting, and multi-objective selection all run on numpy and stdlib with no LLM in the loop; the host coding agent's LLM (Claude Code, Codex, or whichever plugin host is running the skill) is invoked only at three judgment points around that core — which factors are worth testing, what one atomic change to try next, and whether a kept result looks gamed. There are no vendor API calls inside the plugin itself: the skill instructs the host's own model.

```mermaid
flowchart LR
  A["worktree.py init<br/>dedicated git worktree"] --> B["suggest_factors.py<br/>scan for factor candidates"]
  B --> C["Host LLM<br/>picks candidates to test"]
  C --> D["validate_factors.py<br/>snapshot/mutate/revert/verify"]
  D --> E{"doe.py detect k"}
  E -->|"k=1"| G["loop.py<br/>autoresearch mode"]
  E -->|"k=2-3"| F1["full factorial<br/>4-8 runs"]
  E -->|"k=4-7"| F2["fractional factorial<br/>8 runs"]
  E -->|"k=8-11"| F3["Plackett-Burman<br/>12-run screening"]
  F1 --> H["doe.py generate<br/>numpy ±1-coded matrix"]
  F2 --> H
  F3 --> H
  H --> I["apply factor values"]
  I --> J["metric_runner.py<br/>sampled measurement + guard"]
  J --> L[("results.jsonl")]
  L -->|"next run"| I
  L --> M["doe.py analyze<br/>OLS effects (doe_stats.py)"]
  M --> N["objectives.py select_best<br/>scalarize / desirability / pareto"]
  G --> G1["optimize-runner agent<br/>hypothesize -> edit -> commit"]
  G1 --> G2["metric_runner.py + loop.py --score"]
  G2 --> G3{"aggregate improved<br/>& guard ok?"}
  G3 -->|"keep"| G1
  G3 -->|"discard"| G4["git revert"] --> G1
  N --> O["overfitting-reviewer agent<br/>read-only, adversarial"]
  G3 -->|"converged / budget exhausted"| O
  O --> P["loop.py --archive"]
  P --> Q["worktree.py cleanup"]
```

**Models — which, where, why:**

| Stage | Model / tier | Where it runs | Why there |
|---|---|---|---|
| Factor candidate ranking | Host LLM (whatever coding agent runs the skill) | Phase 0.2, before any run | `suggest_factors.py` is a deterministic scan; picking which candidates are worth burning a run budget on is judgment, not math |
| Hypothesis generation (autoresearch) | Host LLM, Sonnet-tier per `agents/optimize-runner.md` | one call per loop iteration | Fires up to `budget` times per run, so it's pinned to a fixed code-tier model rather than a slower thinking tier |
| Overfitting review | Host LLM, Sonnet-tier per `agents/overfitting-reviewer.md` | once per run, Phase 3, read-only | Adversarial pattern match over kept diffs (safety removal, metric-gaming, scope violations) needs judgment; the agent has no edit tools |
| DOE generation, effects fitting, selection, measurement, guard | **none — deterministic** | `scripts/doe.py`, `doe_stats.py`, `objectives.py`, `metric_runner.py` | numpy OLS fitting and closed-form scoring math; "the metric is the only judge," per the skill — no model call decides a winner |

Models: none inside the plugin's own code path — the three LLM touch points above are the host coding agent's model, called by the skill's instructions, not by an SDK call in this codebase.

**Tools & infra — which, why:**

- **numpy (>=1.24)** — the only runtime dependency. Full factorial enumeration, fractional-factorial generators, and the Plackett-Burman 12-run Paley construction are numpy-only and were verified equivalent to pyDOE3 1.6.2 up to row/column permutation; the same array ops drive the OLS effects fit in `doe.py analyze`.
- **Git worktrees (`scripts/worktree.py`, stdlib `subprocess` + `argparse`, zero deps)** — every run gets its own `<repo>-agent-doe-engine-<slug>` worktree on branch `agent-doe-engine/<slug>` so the apply/measure/revert cycle across many DOE runs never dirties the user's primary checkout.
- **Filesystem JSON/JSONL, no database** — state lives under `.agent-doe-engine/optimize/` in the *consumer* project: `factors.json`, `doe.json`, `results.jsonl`, `effects.json`, `objectives.json`, plus `experiment.json` / `results.tsv` for autoresearch and an `experiments/` archive. A legacy `.multi-goal/` path from before the project's rename is migrated automatically on first use.
- **subprocess shell execution (`metric_runner.py`)** — runs the caller's own metric/guard commands, extracts a numeric result via a regex cascade (labeled values → percentages → `time(1)` output → trailing number → any number), and aggregates repeated samples (last/min/max/mean/median/p95) with configurable warmups to smooth noisy metrics.
- **pytest (dev only)** — the test suite (`test_doe.py`, `test_doe_stats.py`, `test_objectives.py`, `test_validate_factors.py`, `test_worktree.py`, `test_manifest.py`) exercises the design math and the validators.
- **Dual plugin manifests** — `.claude-plugin/plugin.json` (Claude Code) and `.codex-plugin/plugin.json` (Codex) point at the same skill/commands/agents directories, so the deterministic scripts and skill instructions are host-neutral.
- **Distribution** — npm package `@tyroneross/agent-doe-engine` (metadata-only `package.json`; no JS/TS source, ships the Python scripts, skill, commands, and agents as files) alongside a standard Python install via `pyproject.toml` and `uv`.

## How it works

1. **Isolate.** `worktree.py init` creates or reuses a dedicated git worktree (`<repo>-agent-doe-engine-<slug>` on branch `agent-doe-engine/<slug>`) so the run never touches the user's primary checkout.
2. **Scan.** `suggest_factors.py` walks the target repo for numeric/config knobs worth tuning, ranking candidates and flagging ones a research step could improve on (`--research-levels`).
3. **Pick.** The host LLM reviews the candidate list and selects which factors to test — the one reasoning step before any run spends budget; the user confirms.
4. **Validate.** `validate_factors.py` proves each accepted candidate is actually adjustable via a snapshot → mutate → re-read → revert → verify cycle; only `adjustable` candidates enter `factors.json` (dead constants, duplicate definitions, and failed mutations are rejected with a reason).
5. **Design.** `doe.py detect <k>` routes by factor count — `k=1` → autoresearch; `k=2–3` → full factorial (4–8 runs); `k=4–7` → fractional factorial (8 runs, some effects aliased); `k=8–11` → Plackett-Burman (12-run screening) — then `doe.py generate` builds the ±1-coded matrix with numpy.
6. **Run the matrix.** For each row: apply the factor values, measure every named objective with `metric_runner.py` (sampled, aggregated by last/min/max/mean/median/p95), run the guard command (must exit 0), append the result to `results.jsonl`, then revert — every run starts from the same baseline.
7. **Fit effects.** `doe.py analyze` fits an OLS model (intercept + main effects + two-way interactions, via `doe_stats.py`) per objective, ranks factors by effect size with p-values, confidence intervals, and a low-power warning, and reports which effects are aliased (tangled) in the chosen design.
8. **Select.** `objectives.py select_best` scores every run by the configured method — `scalarize` (weighted min-max-normalized sum), `desirability` (Derringer-Suich weighted geometric mean; Derringer & Suich, *Journal of Quality Technology*, 1980), or `pareto` (non-dominated front, always computed regardless of selection) — and returns `best_factors`, the concrete winning configuration.
9. **Or, for one factor: autoresearch.** `loop.py` drives a greedy loop instead of a matrix: the `optimize-runner` agent (Sonnet-tier) hypothesizes one atomic change, commits it, measures every objective, scores the aggregate improvement ratio against the recorded baseline (`loop.py --score`), and keeps the change or `git revert`s it based on whether the aggregate improved and the guard passed. Convergence: 5 consecutive discards, a regressing trend, or budget exhaustion.
10. **Review.** Phase 3 dispatches the `overfitting-reviewer` agent (read-only, Sonnet-tier) against every kept commit, checking for removed safety checks, fragile shortcuts, metric-gaming, scope violations, and Goodhart risk on any kept result driven by an `unvalidated` objective — reports `strong_checkpoint` / `guidance` findings, never a hard blocker.
11. **Archive and clean up.** The run is archived (`loop.py --archive`) and the worktree removed (`worktree.py cleanup`), keeping the branch by default so the user can review, merge, or cherry-pick before it's deleted.

## Real Effect vs Fluke

Run the same setting twice and the result won't be identical — that spread is the noise. A real effect is a change bigger than the noise; a fluke fits inside it. Every effect gets a p-value and a confidence interval, with a low-power warning when there aren't enough runs to tell the two apart. The point: you don't ship a change that was never real.

## The Designs It Uses

It picks the smallest design that still answers the question:

| Settings | Design | What it does |
|---|---|---|
| 2–3 | Full factorial (4–8 runs) | Tests every combination — most accurate, nothing tangled |
| 4–7 | Fractional factorial (8 runs) | A carefully chosen subset; reports which effects are tangled |
| 8–11 | Plackett-Burman (12 runs) | Screening — finds the few settings that matter out of many |

For a single setting it falls back to a "try a change, measure, keep it if better" loop — cheaper to set up, but blind to interactions.

## Competing Goals

When you care about numbers that fight each other (faster, but cheaper, but more accurate), it offers three ways to choose: **scalarize** (best weighted blend), **desirability** (every goal must clear a minimum bar), and **pareto** (show all the best trade-offs before committing).

## Worked Example

Using the two-setting case named in "The Problem" — batch size and worker count, the pair called out because a bigger batch size only helps once workers are also added:

**Factors**: `batch_size` (low / high), `workers` (low / high) — 2 settings under test.

**Matrix**: 2 settings falls in the 2–3 tier from the table above, so the engine selects a full factorial: 2² = 4 runs, every combination covered.

| Run | batch_size | workers |
|---|---|---|
| 1 | low | low |
| 2 | high | low |
| 3 | low | high |
| 4 | high | high |

**Effect analysis**: ⚠️ no benchmark yet. "Real Effect vs Fluke" defines the method applied to this matrix — a p-value and confidence interval per effect, a low-power warning when four runs aren't enough to separate signal from noise, and explicit reporting of the interaction term (whether the batch-size effect actually depends on the workers setting) — but no measured p-values, confidence intervals, or effect sizes from an actual run are published in this write-up.

## Tech stack

Python + numpy (only runtime dependency) for the DOE matrix math and OLS effects fitting; stdlib `subprocess`/`argparse` for the git-worktree isolation layer and sampled metric/guard execution — no other packages. pytest for the dev test suite. Dual plugin manifests (`.claude-plugin/plugin.json`, `.codex-plugin/plugin.json`) make the same skill/scripts host-neutral across Claude Code and Codex. Distributed as the npm package `@tyroneross/agent-doe-engine` (metadata-only — no JS/TS source) alongside a standard `pyproject.toml`/`uv` Python install. State is JSON/JSONL on disk under `.agent-doe-engine/optimize/` in the consumer project — no database. Apache-2.0.

## Links

- [HTML project page](https://rosslabs.ai/projects/agent-doe-engine/)
- [Source code](https://github.com/tyroneross/agent-doe-engine)