# goal.md — template for a scored agent loop The structure I use for long-running optimisation runs, stripped of any one project. Companion to , which explains why each part is here. Copy it, fill it in, and read the notes — several of these clauses exist because a run got past an earlier version of them. --- ## /target `harness/score.sh` scores `eval/dev` and prints exactly one JSON line: ``` {"categories": {"": 0.0, ...}, "composite": 0.0, "failingCount": 0} ``` State the goal in one sentence a script could not misread. If you cannot write the scoring command down, you do not have a loss function yet — you have a preference. > Emit `failingCount` even though the composite already implies it. A composite > that moves while the failing count does not is the clearest signal that > nothing is being fixed, and a mean hides it. ## /constraints - `goal.md`, `spec.md`, `harness/` and `eval/` are **READ-ONLY**, checksummed. Modifying any of them voids the score. - Surface allowlist: this repository only. No network. - Expected outputs do not exist in the repository. Inputs are readable; the answer key is not there to be copied. - **Determinism is part of the spec.** No wall-clock, no randomness, no filesystem-order dependence in anything scored. Otherwise a rerun produces a different number and improvement cannot be told from noise. - **Honest path.** Scored scenarios run through the production path — routes, services, database — not through a scoring shortcut. A bundle that satisfies the scorer without going through the real code does not count. - Hardcoded lookup tables keyed on eval content are a violation *even where the lint is blind*. ## /bar Two clauses, not one: - Holdout composite **≥ 0.95** - **and** no single category **< 0.90** - Probe gap **≤ 0.02** > The second clause is the one people leave out. A composite is a mean: 0.92 > looks like "nearly there" and can mean "five parts done, one not started". Acceptance is measured on **holdout only**, aggregate-only, on a budget (mine: six calls per 48 hours). A win on dev is not a win. ## /eval Three splits, and the third is the one that does the work: | Split | Seen by the loop | Purpose | |---|---|---| | `dev` | Every cycle | What it optimises against | | `holdout` | Never, until acceptance | What it is judged on | | `probe` | Never | Same patterns, different surface | **Probe is dev restated** — renamed entities, reordered keys, shifted timestamps, scaled amounts, reformatted input. Same problems, nothing incidental in the same place. The probe gap (dev score minus probe score) does not tell you how good the result is. It tells you whether the composite means anything. ## /anti_goals Name the things a high score must not be bought with. At minimum: - No editing the target, the harness or the eval. - No behaviour conditional on being scored. - No eval scenario identifiers in source. - No data file above a small size threshold in the source tree. ## /stop_conditions No time limit. Stop only when: 1. **Bar reached** — holdout ≥ 0.95, no category < 0.90. 2. **Saturation** — marginal composite gain ≈ 0 across 5 consecutive cycles despite the entropy rules below. 3. **Memorisation spiral** — the probe gap *grows* across 3 cycles even after removing every artefact you can find. > Condition 3 is a trend, not a threshold. A threshold catches the end state; by > the time an absolute number trips, the memorisation is already built. Stopping on 2 or 3 is a legitimate outcome. Write up what was suspected. ## /forced_entropy Rules against a loop that stops moving and tries the same thing harder: - **Stall rule.** If the composite did not move last cycle, the next attempt must be a structural change — different subsystem, different design. Same-knob-harder is banned. - **Exploration quota.** Every fifth cycle goes to the weakest category, even when the obvious leverage is elsewhere. - **Re-read the spec.** A category at zero for three cycles means the semantics were misunderstood, not that the code is nearly right. ## /cycle_protocol Each cycle appends to `LOG.md`: score before, what was changed and why, score after, and the next hypothesis. Every ~24h of active runtime, an interim report: score history, gain per cycle, structural attempts, next hypotheses. On stop, a closing report: best holdout score, what generalised, what was discarded, and the three most valuable next steps. --- ## What this does not cover The instruments above test whether a **score is honest**. None of them tests whether the thing being scored is the **whole product** — a run can generalise perfectly across a surface that was cut smaller than what you are building, and every check here will pass. The only thing that has caught that for me is opening the product afterwards and asking whether the eval still describes it. If you find something better, ends on that question.