A factorial design tests several settings together, so an effect that only appears in combination — a bigger batch that helps once you also add workers — shows up instead of hiding. Agent DOE Engine and Multi-Goal cap a batch at 11 factors, p-values and confidence intervals hold only when the design supplies a defensible error estimate, and a screening result stays a candidate until a confirmation run repeats it.
Design of experiments tests several settings together so interactions can be measured. Start by defining the objective, factor levels, required resolution, run budget, and error plan. Agent DOE Engine and Multi-Goal use two-level factorial, fractional factorial, or Plackett-Burman designs as local defaults. P-values and confidence intervals are valid only when the design supplies a defensible error estimate. Screening results require confirmation. Prompt & Model Benchmark Lab uses a separate calibration method and is not a factorial design.
Technical reference: NIST design selection, two-level full factorial designs, and Plackett-Burman screening.
Four steps, one paragraph each — about a two-minute read. The "each step, for real" section below expands every one of these into the actual mechanism, with real numbers from each project's own write-up.
Before any run happens, you list what to test and what you care about: the settings that might change the outcome (factors), and the results you're actually judging — speed, cost, quality, accuracy (objectives). Agent DOE Engine and Multi-Goal both cap this at up to 11 factors; past that the design gets too big to run in one batch.
See the mechanism →Agent DOE Engine and Multi-Goal use two-level factors and choose a local starting design by factor count. A defensible design also depends on the experiment's objective, required resolution, interaction risk, replication plan, and available runs.
See the mechanism →The matrix is a plan, not a result — every row still has to execute. Each row is one combination of settings, run and measured on every objective, so the batch produces one full set of numbers per objective per configuration in a single planned pass instead of a string of one-off trials.
See the mechanism →Repeated results vary. Estimate each effect with uncertainty only when the design has a defensible error estimate. Saturated screening designs should surface directional signals for confirmation instead of automatic p-values.
See the mechanism →The paragraphs above are the map. These open into the actual designs, thresholds, and numbers, drawn from each project's own write-up.
A factor is anything you can set before a run — batch size, worker count, temperature, model choice. An objective is what you measure after — latency, cost, an accuracy score. Both are fixed before any run starts, because the next step depends on planning every combination in advance instead of reacting to one change at a time.
One-at-a-time testing is the default most people reach for: change one setting, measure, keep it if it's better. Per Agent DOE Engine's own write-up, it's slow and blind to interactions — a bigger batch size that only helps once you also add workers never shows up, because it's never tested in combination.
Multi-Goal was extracted and extended from build-loop's single-metric optimize skill, generalizing the same factor/objective framing from tuning one number to tuning several that can fight each other — latency dropping while bundle size grows, or test coverage climbing while build time balloons.
With two-level factors, a full factorial evaluates every combination. Two factors require 4 treatment combinations and three require 8. Replication adds runs and supplies an estimate of experimental error.
The local tools use an 8-run fractional factorial for 4-7 factors. Fractional designs trade runs for aliasing, so the output must state the design resolution and which effects are confounded.
The local tools use a 12-run Plackett-Burman design to screen 8-11 factors. A saturated 12-run design has no residual degrees of freedom. Main effects can also be confounded with two-factor interactions, so apparent winners need confirmation runs or a stated error model.
For a single factor there's no matrix to build. Both engines fall back to a one-at-a-time loop — cheaper to set up, but blind to interactions by construction. Multi-Goal's write-up describes a related fallback: when a full design is too expensive, it drops to a greedy single-factor autoresearch loop rather than skip measurement entirely.
Agent DOE Engine varies several settings together in one planned batch. The matrix estimates effects. Replication, repeated measurements, historical error, or an explicit model assumption is still required to estimate noise.
Multi-Goal runs the same matrix, then "fits per-objective effects" — each objective gets its own model of which factors moved it, computed from the same set of runs. Nothing about the run step is objective-specific; one batch feeds every objective's analysis at once.
Prompt & Model Benchmark Lab's Run primitive — Cases × Candidates, where a Candidate is a model + prompt + params combination scored against every test Case — is structurally the same shape: one planned batch, not a string of one-off comparisons. It doesn't use factorial-design math to build that batch, so it's covered in the "where this shows up" table below rather than folded into this mechanism section as a literal DOE run.
This is the step that keeps a plausible-looking number from shipping as fact. Per Agent DOE Engine's own write-up, the point is direct: "you don't ship a change that was never real." Effect size and uncertainty are reported together — a big effect with a wide confidence interval is not the same claim as a big effect with a tight one, even when the point estimate looks identical.
A p-value and confidence interval depend on the fitted model and its error estimate. If the design is saturated or underpowered, report that limitation directly. Use replication or confirmation runs before treating a screening signal as a durable effect.
When objectives compete — faster but more expensive, more accurate but slower — a real effect on one objective still has to be weighed against the others before anything gets called a winner. Both engines offer three resolution strategies: scalarize (best weighted blend across objectives), desirability (Derringer-Suich — every objective must clear its own target and tolerance), and Pareto (surface the full set of non-dominated trade-offs and choose from those, rather than collapsing to one number).
Same four steps, one row each, against the three projects that run this discipline. A dash means the project's own write-up doesn't describe that step in those terms — not confirmed absent, just not stated that way.
Agent DOE Engine and Multi-Goal are close siblings: Multi-Goal was extracted and extended
from build-loop's single-metric optimize skill, generalizing the same
factorial-design math from tuning one agent setting to tuning any set of competing software
metrics. Both pick a full factorial, fractional factorial, or Plackett-Burman design by
factor count, and both resolve competing objectives with the same three strategies —
scalarize, desirability, Pareto.
Prompt & Model Benchmark Lab doesn't build a factorial matrix or fit an effect model — its write-up describes no factor sweep at all. What it shares with the other two is the underlying question: how much evidence supports this score. Its Cases × Candidates run plans one batch, but calibration confidence is not a p-value and does not estimate a factorial effect. The two methods should remain separate in both the output and the interpretation.