@@ -1,6 +1,6 @@ |
| 1 | 1 | # bencch |
| 2 | 2 | |
| 3 | | -Compiler bench for `armfortas`. |
| 3 | +Structured compiler runner and reporting bench for `armfortas`. |
| 4 | 4 | |
| 5 | 5 | This repo holds: |
| 6 | 6 | |
@@ -12,6 +12,19 @@ This repo holds: |
| 12 | 12 | |
| 13 | 13 | ## Current Setup |
| 14 | 14 | |
| 15 | +`bencch` is no longer aiming to be the primary product story for a generic |
| 16 | +compiler bench. Its role is clearer now: |
| 17 | + |
| 18 | +- the root armfortas harness is the fast armfortas-first runner and the default |
| 19 | + home for new source-directed testing ideas |
| 20 | +- `bencch` is the structured matrix/reporting/differential runner around that |
| 21 | + same testing language |
| 22 | + |
| 23 | +Source comments in shared fixtures are the canonical leaf-assertion language. |
| 24 | +`bencch` should consume those directives where supported and explain |
| 25 | +unsupported directives clearly, rather than inventing a separate assertion |
| 26 | +dialect. |
| 27 | + |
| 15 | 28 | Today `bencch` is wired to a surrounding `armfortas` checkout. The practical way |
| 16 | 29 | to use it is from the `armfortas` workspace root. CLI-side compiler and tool |
| 17 | 30 | paths are overridable now; linked capture still comes from the surrounding |
@@ -125,8 +138,20 @@ Common things the runner understands: |
| 125 | 138 | - per-opt status like `xfail when O1, O2 because "reason"` |
| 126 | 139 | - consistency checks like `cli_obj_vs_system_as` and `capture_run_reproducible` |
| 127 | 140 | |
| 141 | +The suite DSL is for orchestration: |
| 142 | + |
| 143 | +- opts |
| 144 | +- compilers and references |
| 145 | +- graph composition |
| 146 | +- capability policy |
| 147 | +- reporting and bundles |
| 148 | + |
| 149 | +Leaf assertions should come from shared source directives whenever possible. |
| 150 | + |
| 128 | 151 | ## Notes |
| 129 | 152 | |
| 130 | 153 | - `.docs/` is local and gitignored. |
| 131 | | -- The runner is currently strongest on stage capture, differential behavior, |
| 132 | | - and consistency work around reproducibility and cross-path mismatches. |
| 154 | +- The runner is currently strongest on matrices, differential behavior, |
| 155 | + capability-aware execution, reports, bundles, and graph orchestration. |
| 156 | +- The shared-language reset and follow-through testing roadmap live in the |
| 157 | + parent `armfortas` repo under `.docs/testing/`. |
