@@ -2,6 +2,75 @@ |
| 2 | 2 | |
| 3 | 3 | ## Unreleased |
| 4 | 4 | |
| 5 | +### Sprint 30 — `multi_turn_coherence_decay` probe |
| 6 | + |
| 7 | +Closes the P2 "multi_turn_coherence_decay probe" backlog item. Sway |
| 8 | +had zero coverage of multi-turn behavior before this — every |
| 9 | +shipped adherence probe was single-turn. Adapters that pass |
| 10 | +`delta_kl` cleanly frequently degrade by turn 2 or 3 of real |
| 11 | +dialogue, where the model's own previous responses enter the |
| 12 | +context window and create compounding drift. The new probe is the |
| 13 | +first that catches that failure mode. |
| 14 | + |
| 15 | +**New probe (`kind: multi_turn_coherence_decay`, category: adherence).** |
| 16 | + |
| 17 | +For each prompt the probe greedy-generates ft's turn-1 response, |
| 18 | +then rolls a multi-turn synthetic dialogue with cycled generic |
| 19 | +follow-ups (`Continue.`, `Tell me more.`, …). At each turn 2..N |
| 20 | +both views see the same ft-grounded chat history; the probe |
| 21 | +computes `KL(base || ft)` at the next-token position. The |
| 22 | +per-turn KL series is fit to `kl = a · exp(-b · turn)` and |
| 23 | +the probe reports `half_life_turns = ln(2) / b`. |
| 24 | + |
| 25 | +Verdict ladder: |
| 26 | +- `ok` — clean exponential decay; PASS when `half_life_turns ≥ |
| 27 | + assert_half_life_turns` (default 2.0). |
| 28 | +- `stable` — KL stayed within 0.1% relative spread across turns; |
| 29 | + half-life is formally infinite; clipped to `max_turns × 10` |
| 30 | + and rendered with a "held coherence" message; always PASS. |
| 31 | +- `non_monotonic` — KL grew turn-over-turn (atypical but |
| 32 | + possible); WARN with the curve in evidence. |
| 33 | +- `degenerate` — KL ≈ 0 at every turn; FAIL with "probable no-op |
| 34 | + adapter" diagnosis. |
| 35 | + |
| 36 | +**No null calibration.** Mirrors `prompt_collapse`'s rationale: a |
| 37 | +null adapter has no signal to decay, so the null distribution of |
| 38 | +half-lives is meaningless. Fixed-threshold verdicts are the |
| 39 | +published path. |
| 40 | + |
| 41 | +**Chat-template requirement.** Multi-turn dialogue requires the |
| 42 | +base's tokenizer to carry a `chat_template`. Bases without one |
| 43 | +SKIP gracefully with a clear message. The probe consults |
| 44 | +`ctx.backend._tokenizer.chat_template` — same backdoor |
| 45 | +`prompt_collapse` uses for the same reason (avoids broadening the |
| 46 | +public scoring contract for one probe's needs). |
| 47 | + |
| 48 | +Reports surface a tiny unicode sparkline of per-turn KL in the |
| 49 | +verdict message so terminal-only readers see curve shape without |
| 50 | +opening the JSON. |
| 51 | + |
| 52 | +**Implementation:** |
| 53 | +- `probes/multi_turn_coherence.py` — spec, probe, curve fit, |
| 54 | + verdict mapping, sparkline. |
| 55 | +- `probes/__init__.py` — registers the new probe. |
| 56 | + |
| 57 | +**Test surface:** |
| 58 | +- `tests/unit/test_probe_multi_turn_coherence.py` — 22 unit tests |
| 59 | + covering skip paths, end-to-end with planted-distribution |
| 60 | + sequences (decreasing / flat / growing curves), fit math (clean |
| 61 | + exp / stable / growing / zero / partial-zero), verdict mapping, |
| 62 | + sparkline rendering, and chat-template detection. |
| 63 | +- `tests/integration/test_probe_multi_turn_coherence.py` — |
| 64 | + slow+online HF smoke on a tiny SmolLM2-135M LoRA across 3 |
| 65 | + dialogue turns. Exercises the full chat-template + turn-loop + |
| 66 | + curve-fit path on a real backend. |
| 67 | + |
| 68 | +**README** gains a "Multi-turn coherence" section between |
| 69 | +`Tool-use fidelity` and `Reproducing a sway run` with a worked |
| 70 | +YAML example. The probe table at "Why it exists" picks up the |
| 71 | +new entry under Adherence (plus `tool_use_fidelity` under |
| 72 | +Attribution — the table missed S27's addition). |
| 73 | + |
| 5 | 74 | ### Sprint 28 — `tenseleyflow/sway-action` GitHub Action |
| 6 | 75 | |
| 7 | 76 | Closes the P2 "GitHub Action" discoverability item from Audit 03 |