@@ -0,0 +1,158 @@ |
| | 1 | +# Testing guide (contributor-facing) |
| | 2 | + |
| | 3 | +Everything you need to run the test suite locally and understand what each |
| | 4 | +layer does. |
| | 5 | + |
| | 6 | +## Layers |
| | 7 | + |
| | 8 | +``` |
| | 9 | +tests/ |
| | 10 | + test_smoke.py package + CLI boot |
| | 11 | + unit/ fast, in-process, no network |
| | 12 | + integration/ crosses 2+ modules (e.g. parser + store) |
| | 13 | + e2e/ full CLI against tmp stores |
| | 14 | + fixtures/ factories + mocks (see below) |
| | 15 | + golden/ checked-in JSON goldens per (name, torch_version) |
| | 16 | +``` |
| | 17 | + |
| | 18 | +## Markers |
| | 19 | + |
| | 20 | +| marker | meaning | default | |
| | 21 | +|---|---|---| |
| | 22 | +| (none) | fast unit, <1s each | run | |
| | 23 | +| `slow` | expensive; may load the tiny model | **skipped** | |
| | 24 | +| `gpu` | requires CUDA | skipped on CPU/MPS | |
| | 25 | +| `online` | touches the network (HF Hub) | skipped offline | |
| | 26 | + |
| | 27 | +`pyproject.toml` sets `addopts = ["-m", "not slow and not gpu and not online"]` |
| | 28 | +so the default `uv run pytest` is always the fast, local subset. |
| | 29 | + |
| | 30 | +## Running |
| | 31 | + |
| | 32 | +``` |
| | 33 | +uv run pytest # fast subset, default |
| | 34 | +uv run pytest -m slow # tiny-model and long-running paths |
| | 35 | +uv run pytest -m "slow and online" # tiny-model download + inference |
| | 36 | +uv run pytest --update-goldens # regenerate goldens (see below) |
| | 37 | +uv run pytest -v path/to/test_file.py # single-file verbose |
| | 38 | +``` |
| | 39 | + |
| | 40 | +## Fixtures |
| | 41 | + |
| | 42 | +### `tests/fixtures/dlm_factory.py` |
| | 43 | + |
| | 44 | +Builds synthetic `.dlm` text. Stable shape matching Sprint 03's parser. |
| | 45 | + |
| | 46 | +```python |
| | 47 | +from tests.fixtures.dlm_factory import make_dlm, prose, instruction, preference |
| | 48 | + |
| | 49 | +text = make_dlm( |
| | 50 | + sections=[ |
| | 51 | + prose("# intro\n\nbody\n"), |
| | 52 | + instruction(("Q1?", "A1."), ("Q2?", "A2.")), |
| | 53 | + preference(("prompt", "good", "bad")), |
| | 54 | + ], |
| | 55 | + base_model="smollm2-135m", |
| | 56 | + dlm_id="01HZ...", # omit for a fresh ULID |
| | 57 | + training_overrides={"lora_r": 16}, |
| | 58 | +) |
| | 59 | +``` |
| | 60 | + |
| | 61 | +### `tests/fixtures/hardware_mocks.py` |
| | 62 | + |
| | 63 | +Context managers for backend simulation without real hardware. |
| | 64 | + |
| | 65 | +```python |
| | 66 | +from tests.fixtures.hardware_mocks import force_cuda, force_mps, force_cpu |
| | 67 | + |
| | 68 | +with force_cuda(sm=(8, 9), vram_gb=24.0): |
| | 69 | + # torch.cuda.is_available() is True, capability (8, 9), mem 24GB |
| | 70 | + ... |
| | 71 | + |
| | 72 | +with force_mps(): |
| | 73 | + # MPS is available; CUDA is not |
| | 74 | + ... |
| | 75 | +``` |
| | 76 | + |
| | 77 | +Nesting works — the inner context is restored on exit. |
| | 78 | + |
| | 79 | +### `tests/fixtures/tiny_model.py` |
| | 80 | + |
| | 81 | +SmolLM2-135M-Instruct as a session-scoped fixture. Download is gated behind |
| | 82 | +`@pytest.mark.online`; the session-scoped `tiny_model_dir` fixture returns the |
| | 83 | +cached path. |
| | 84 | + |
| | 85 | +```python |
| | 86 | +import pytest |
| | 87 | + |
| | 88 | +@pytest.mark.online |
| | 89 | +@pytest.mark.slow |
| | 90 | +def test_something(tiny_model_dir): |
| | 91 | + # tiny_model_dir is a pathlib.Path to the cached model |
| | 92 | + ... |
| | 93 | +``` |
| | 94 | + |
| | 95 | +The revision is pinned via `DLM_TINY_MODEL_REVISION` (defaulting to `main` |
| | 96 | +until Sprint 06's base-model registry owns the SHA). |
| | 97 | + |
| | 98 | +### `tests/fixtures/golden.py` |
| | 99 | + |
| | 100 | +```python |
| | 101 | +from tests.fixtures.golden import assert_golden |
| | 102 | + |
| | 103 | +def test_loss_curve(): |
| | 104 | + values = compute_loss_curve() |
| | 105 | + assert_golden({"loss": values}, name="loss-curve-v1") |
| | 106 | +``` |
| | 107 | + |
| | 108 | +Goldens live at `tests/golden/<name>.torch-<version>.json`. Bumping torch |
| | 109 | +creates a new key; the old one stays until deliberately removed. |
| | 110 | + |
| | 111 | +## Regenerating goldens |
| | 112 | + |
| | 113 | +``` |
| | 114 | +uv run pytest --update-goldens |
| | 115 | +``` |
| | 116 | + |
| | 117 | +This flips `assert_golden` into write mode. Review the diff before |
| | 118 | +committing: |
| | 119 | + |
| | 120 | +``` |
| | 121 | +git diff tests/golden/ |
| | 122 | +``` |
| | 123 | + |
| | 124 | +A two-person review is mandatory for golden changes — they're determinism |
| | 125 | +contracts. See Sprint 15's `scripts/regen-determinism-golden.py` for the |
| | 126 | +heavier regeneration workflow once that lands. |
| | 127 | + |
| | 128 | +## CI layout |
| | 129 | + |
| | 130 | +Three GitHub Actions jobs: |
| | 131 | + |
| | 132 | +1. **lint / typecheck / test** — ubuntu-latest + macos-latest matrix. |
| | 133 | + Runs ruff, ruff format --check, mypy, default pytest selection. |
| | 134 | +2. **no-network sandbox** — ubuntu-latest. Blocks egress via iptables, |
| | 135 | + then runs the local-only CLI surfaces (`dlm --version`, `--help`, |
| | 136 | + and later `init`/`doctor`/`show`). Asserts the "no telemetry, ever" |
| | 137 | + promise. |
| | 138 | +3. **slow tests (hf-cache)** — ubuntu-latest. Restores HF cache keyed |
| | 139 | + on `(pyproject.toml hash, TINY_MODEL_REVISION)`, pre-warms the tiny |
| | 140 | + model, then runs `pytest -m slow`. |
| | 141 | + |
| | 142 | +## Offline-first autouse |
| | 143 | + |
| | 144 | +`tests/conftest.py` sets `HF_HUB_OFFLINE=1` + `TRANSFORMERS_OFFLINE=1` + |
| | 145 | +`HF_DATASETS_OFFLINE=1` via an autouse fixture. The `tiny_model_dir` |
| | 146 | +fixture temporarily clears these for its scope when an online test opts |
| | 147 | +in. This means a test that *accidentally* touches HF without the fixture |
| | 148 | +will fail fast instead of downloading silently. |
| | 149 | + |
| | 150 | +## Common pitfalls |
| | 151 | + |
| | 152 | +- **Importing torch in test collection is slow** (~5s). Fixtures that |
| | 153 | + need it import lazily inside functions. |
| | 154 | +- **Hardware mocks don't simulate actual CUDA computation.** They only |
| | 155 | + toggle `is_available`-shaped attributes. Tests that need a real GPU use |
| | 156 | + the `gpu` marker. |
| | 157 | +- **Golden drift on torch bumps is expected.** Regeneration is the fix; |
| | 158 | + review the old vs new checksum side-by-side before approval. |
