Presient SE Harness Pack

What it is

A pilot that runs Qwen3-Coder on bug-fixing tasks with and without a lightweight harness, on a pinned model route, and scores every run. Before more money was spent, the result was audited, and the honest reading turned out narrower than the headline number.

What it does

• Pinned engine, 100 runs per condition: the model fixed the bugs about 98% of the time with or without the harness (98% vs 99%).
• The harness cut out-of-scope edits from 15 of 100 runs to 6 of 100 (p = 0.019).
• A pre-spend audit found the +10-point composite pass rate was being read as a bug-fixing gain, corrected the claim, and fixed four scoring defects.
• An earlier 48-run pilot on an unpinned route was archived rather than pooled, because it ran on a different engine.

The problem

A coding model can fix a bug and still edit files it was not asked to touch. This pack is a set of hooks and rules for a coding agent, plus a test setup that compares the same model with and without the harness. The recorded result: the model fixed the bugs about 98% of the time with or without the harness; the harness cut edits outside the task from 15 of 100 runs to 6.

Design decisions

• Compare two conditions, bare and harnessed, on the same fixtures and the same model, with the router pinned. This isolates the harness, but results hold only for the model and tasks tested.
• Use hooks: a destructive-command gate, and checks that stop the agent from finishing while edits are unverified. The tradeoff is more turns and time per run.
• Score scope separately from test passing. Decomposing the score showed that the lift was about staying in scope, not about fixing bugs, and the claim was narrowed to match.
• Write the next experiment down before running it, including its stop and cost rules. The tradeoff is slower iteration.

How it works

1. FixturesSmall tasks built from real repositories, each green except for one target bug.
2. RouterA LiteLLM router sends the agent's requests to a pinned model provider and logs the model actually served.
3. ConditionsThe bare condition has no extras, and the harnessed condition adds hooks and project rules.
4. Matrix runnerRuns each fixture in each condition in isolation and keeps the transcript, diff and git status as evidence.
5. ScorerScores target fixed, other tests unbroken and no edits outside scope as separate checks.

Security: enforced, simulated and known gaps

• Enforced in code: a hook blocks destructive commands in the harnessed condition, and the runner uses an isolation wrapper so the host's own settings do not leak into either condition.
• Known gap: the results are for one model and a small set of tasks, and the repository is not public.

What broke and how it was fixed

What broke

The check that other tests still pass could never fail on four of the fixtures, because their browser tests exit with code 0 even when tests fail. Evidence was also deleted after scoring, so scope failures could not be traced.

How it was found

A pre-spend audit of the first result found it before more credits were spent. The audit decomposed the composite score and tested each check.

The fix

The runner now keeps the diff, git status and files touched for every run, reads each test file's own fail lines, and removes tests that fail in this environment even with the correct fix.

What guards it now

tests/test_verify.sh and the runner and scorer tests cover it, and each fixture was shown failing before the fix and passing after.

Reference

4c97bbc

Where it runs

Runs on the owner's own server as command-line scripts and a local router. It is not hosted and not public.

Who did what

The owner conceived the product, architecture, specs and acceptance tests and directed the work. Coding agents (Claude, Codex, Gemini) did much of the implementation and review, and the pre-spend audit commit is co-authored by Claude.

Limits and honest status

• Five fixtures and one model; bug fixing was already saturated, so the study cannot show a bug-fixing gain.
• A cheaper quantization of the same model showed no scope effect; both comparisons are underpowered.
• Local project, no public repository.

Connected to

Presient Skill EvalsSoftware Factory