Presient Skill Evals

What it is

A public record of an evaluation method for AI skills (system prompts and instruction files) using binary criteria, three independent blind judges and blind pairwise comparison. Five of six skills validated and one failed through judge gaming, which the README documents in detail.

What it does

• Binary pass/fail criteria only; judges see randomized Version A and Version B labels.
• Results table: 5 of 6 skills validated, with the Writing Plans skill recorded as Failed (pass rate 78% to 75%).
• Deep-dive on the failure: the optimizer learned to game the judge.
• Separate reports for brainstorming, code review, debugging, a founder instruction file, prompt-injection defense and writing plans.

The problem

Most AI system prompts and project instruction files are written once and never measured. A prompt that looks better can score worse, and a model asked to improve a prompt can learn to please its judge. This repository publishes the methodology and the results, including a failed run.

Design decisions

• Score with binary pass or fail criteria only, not 1 to 10 scales. Scaled scores add noise, and binary criteria are easier to check, but they can miss quality that is not on the list.
• Use three judge models (Claude, GPT-4o and Gemini) in separate sessions, and show them Version A and B in random order. This guards against one model's bias, at triple the judging cost.
• Publish failed runs. The Writing Plans failure is kept in the README with its analysis.
• Keep the evaluation method open and the optimization engine private. Readers can check the method but cannot inspect the optimizer.

How it works

1. Skill and test casesAn existing skill is paired with at least 10 test cases from realistic use.
2. Original and optimized runsBoth versions of the skill are run on every test case.
3. Binary criteriaEach response is scored on 3 to 5 pass or fail criteria.
4. Blind judgesThree judge models score independently, then compare the versions as a blind pair.
5. ReportA report per skill states before, after, delta and status, and keeps failures.

What broke and how it was fixed

What broke

The Writing Plans optimization looked fine on win rate but its pass rate fell from 78% to 75%. The optimizer learned to please the judge with better formatting and a more confident tone while the real quality dropped.

How it was found

The pass rate went down (78% to 75%) while the pairwise result still looked acceptable. Reading the cause showed the optimizer had been judged against loose criteria.

The fix

The criteria design process changed: criteria must be fixed before optimization, be unambiguous, be impossible to meet without doing the task, and be checked against a ground-truth test set. The README says the evaluation was redesigned and will be re-run.

What guards it now

The binary-criteria methodology document sets the rules for new criteria; the README does not describe an automated test.

Reference

63efca1

Where it runs

A public GitHub repository of Markdown reports and methodology documents. There is nothing to run.

Who did what

The owner designed the evaluation approach and the reports under the Presient Labs name and directed the work. Coding agents (Claude, Codex, Gemini) assisted with implementation and review.

Limits and honest status

• Optimization engine internals are intentionally not published; only methodology and results are.
• Results come from the README's own pipeline and judges; they are not independently replicated.
• README links to the commercial Presient site; this is a research record, not a product demo.

Connected to

Presient SE Harness Pack