bencch Public

Watch 0 Fork 0 Star 0

markdown · 4371 bytes Raw Blame History

bencch

Structured compiler runner and reporting bench for armfortas.

This repo holds:

bench-core/ — bench-owned compiler-facing types
bench/ — the afs-tests runner
suites/ — authored bench suites
fixtures/ — reusable fixture programs
reports/ — failure and consistency bundles

Current Setup

bencch is no longer aiming to be the primary product story for a generic compiler bench. Its role is clearer now:

the root armfortas harness is the fast armfortas-first runner and the default home for new source-directed testing ideas
bencch is the structured matrix/reporting/differential runner around that same testing language

Source comments in shared fixtures are the canonical leaf-assertion language. bencch should consume those directives where supported and explain unsupported directives clearly, rather than inventing a separate assertion dialect.

Today bencch is wired to a surrounding armfortas checkout. The practical way to use it is from the armfortas workspace root. CLI-side compiler and tool paths are overridable now; linked capture still comes from the surrounding workspace. That linked compiler surface is currently isolated in bench/src/compiler.rs, and the bench-owned compiler-facing types now live in bench-core/.

cargo run -p afs-tests -- list
cargo run -p afs-tests -- run --suite frontend

Standalone compiler adapters are not finished yet.

Usage

List suites:

cargo run -p afs-tests -- list

Run one suite family:

cargo run -p afs-tests -- run --suite consistency/runtime

Run against an explicit compiler binary:

cargo run -p afs-tests -- run --suite consistency/runtime-control-flow --armfortas-bin ./target/debug/armfortas

Run differential checks with explicit reference compiler paths:

cargo run -p afs-tests -- run --suite differential/runtime-control-flow --gfortran-bin /opt/homebrew/bin/gfortran --flang-bin /opt/homebrew/bin/flang-new

Run one case with full stage capture:

cargo run -p afs-tests -- run --suite frontend --case stage_walk --all --verbose

Run consistency coverage:

cargo run -p afs-tests -- run --suite consistency --all

Run differential coverage:

cargo run -p afs-tests -- run --suite differential

Reports are written under bencch/reports/.

Environment overrides work too:

BENCCH_ARMFORTAS_BIN=./target/debug/armfortas cargo run -p afs-tests -- run --suite consistency/object

Suite Format

Suites are plain text files under suites/.

suite "consistency/runtime"

case "mixed_types_cli_run_reproducible"
source "../../fixtures/runtime/mixed_types.f90"
opts => all
armfortas => run
repeat => 3
consistency => cli_run_reproducible
expect run.stdout check-comments
expect run.exit_code equals 0
end

Graph cases use entry plus ordered file lines:

suite "modules/runtime-graphs"

case "module_chain_runtime"
entry "../../fixtures/modules/module_chain/main.f90"
file "../../fixtures/modules/module_chain/math_seed.f90"
file "../../fixtures/modules/module_chain/math_values.f90"
file "../../fixtures/modules/module_chain/main.f90"
opts => O0, O1, O2
armfortas => run
expect run.stdout check-comments
end

Today the armfortas adapter materializes graph cases into one generated source in declared file order before capture/compile. The authored files still stay in the failure bundle.

Common things the runner understands:

stage capture like armfortas => tokens, ir, asm, obj, run
opt matrices like opts => O0, O1, O2
references like differential => gfortran, flang-new
expected failures like xfail "reason"
per-opt status like xfail when O1, O2 because "reason"
consistency checks like cli_obj_vs_system_as and capture_run_reproducible

The suite DSL is for orchestration:

opts
compilers and references
graph composition
capability policy
reporting and bundles

Leaf assertions should come from shared source directives whenever possible.

Notes

.docs/ is local and gitignored.
The runner is currently strongest on matrices, differential behavior, capability-aware execution, reports, bundles, and graph orchestration.
The shared-language reset and follow-through testing roadmap live in the parent armfortas repo under .docs/testing/.

View source

  
        1
        # bencch
      
        2
        
        3
        Structured compiler runner and reporting bench for `armfortas`.
      
        4
        
        5
        This repo holds:
      
        6
        
        7
        - `bench-core/` — bench-owned compiler-facing types
      
        8
        - `bench/` — the `afs-tests` runner
      
        9
        - `suites/` — authored bench suites
      
        10
        - `fixtures/` — reusable fixture programs
      
        11
        - `reports/` — failure and consistency bundles
      
        12
        
        13
        ## Current Setup
      
        14
        
        15
        `bencch` is no longer aiming to be the primary product story for a generic
      
        16
        compiler bench. Its role is clearer now:
      
        17
        
        18
        - the root armfortas harness is the fast armfortas-first runner and the default
      
        19
          home for new source-directed testing ideas
      
        20
        - `bencch` is the structured matrix/reporting/differential runner around that
      
        21
          same testing language
      
        22
        
        23
        Source comments in shared fixtures are the canonical leaf-assertion language.
      
        24
        `bencch` should consume those directives where supported and explain
      
        25
        unsupported directives clearly, rather than inventing a separate assertion
      
        26
        dialect.
      
        27
        
        28
        Today `bencch` is wired to a surrounding `armfortas` checkout. The practical way
      
        29
        to use it is from the `armfortas` workspace root. CLI-side compiler and tool
      
        30
        paths are overridable now; linked capture still comes from the surrounding
      
        31
        workspace. That linked compiler surface is currently isolated in
      
        32
        `bench/src/compiler.rs`, and the bench-owned compiler-facing types now live in
      
        33
        `bench-core/`.
      
        34
        
        35
        ```bash
      
        36
        cargo run -p afs-tests -- list
      
        37
        cargo run -p afs-tests -- run --suite frontend
      
        38
        ```
      
        39
        
        40
        Standalone compiler adapters are not finished yet.
      
        41
        
        42
        ## Usage
      
        43
        
        44
        List suites:
      
        45
        
        46
        ```bash
      
        47
        cargo run -p afs-tests -- list
      
        48
        ```
      
        49
        
        50
        Run one suite family:
      
        51
        
        52
        ```bash
      
        53
        cargo run -p afs-tests -- run --suite consistency/runtime
      
        54
        ```
      
        55
        
        56
        Run against an explicit compiler binary:
      
        57
        
        58
        ```bash
      
        59
        cargo run -p afs-tests -- run --suite consistency/runtime-control-flow --armfortas-bin ./target/debug/armfortas
      
        60
        ```
      
        61
        
        62
        Run differential checks with explicit reference compiler paths:
      
        63
        
        64
        ```bash
      
        65
        cargo run -p afs-tests -- run --suite differential/runtime-control-flow --gfortran-bin /opt/homebrew/bin/gfortran --flang-bin /opt/homebrew/bin/flang-new
      
        66
        ```
      
        67
        
        68
        Run one case with full stage capture:
      
        69
        
        70
        ```bash
      
        71
        cargo run -p afs-tests -- run --suite frontend --case stage_walk --all --verbose
      
        72
        ```
      
        73
        
        74
        Run consistency coverage:
      
        75
        
        76
        ```bash
      
        77
        cargo run -p afs-tests -- run --suite consistency --all
      
        78
        ```
      
        79
        
        80
        Run differential coverage:
      
        81
        
        82
        ```bash
      
        83
        cargo run -p afs-tests -- run --suite differential
      
        84
        ```
      
        85
        
        86
        Reports are written under `bencch/reports/`.
      
        87
        
        88
        Environment overrides work too:
      
        89
        
        90
        ```bash
      
        91
        BENCCH_ARMFORTAS_BIN=./target/debug/armfortas cargo run -p afs-tests -- run --suite consistency/object
      
        92
        ```
      
        93
        
        94
        ## Suite Format
      
        95
        
        96
        Suites are plain text files under `suites/`.
      
        97
        
        98
        ```text
      
        99
        suite "consistency/runtime"
      
        100
        
        101
        case "mixed_types_cli_run_reproducible"
      
        102
        source "../../fixtures/runtime/mixed_types.f90"
      
        103
        opts => all
      
        104
        armfortas => run
      
        105
        repeat => 3
      
        106
        consistency => cli_run_reproducible
      
        107
        expect run.stdout check-comments
      
        108
        expect run.exit_code equals 0
      
        109
        end
      
        110
        ```
      
        111
        
        112
        Graph cases use `entry` plus ordered `file` lines:
      
        113
        
        114
        ```text
      
        115
        suite "modules/runtime-graphs"
      
        116
        
        117
        case "module_chain_runtime"
      
        118
        entry "../../fixtures/modules/module_chain/main.f90"
      
        119
        file "../../fixtures/modules/module_chain/math_seed.f90"
      
        120
        file "../../fixtures/modules/module_chain/math_values.f90"
      
        121
        file "../../fixtures/modules/module_chain/main.f90"
      
        122
        opts => O0, O1, O2
      
        123
        armfortas => run
      
        124
        expect run.stdout check-comments
      
        125
        end
      
        126
        ```
      
        127
        
        128
        Today the armfortas adapter materializes graph cases into one generated source
      
        129
        in declared file order before capture/compile. The authored files still stay in
      
        130
        the failure bundle.
      
        131
        
        132
        Common things the runner understands:
      
        133
        
        134
        - stage capture like `armfortas => tokens, ir, asm, obj, run`
      
        135
        - opt matrices like `opts => O0, O1, O2`
      
        136
        - references like `differential => gfortran, flang-new`
      
        137
        - expected failures like `xfail "reason"`
      
        138
        - per-opt status like `xfail when O1, O2 because "reason"`
      
        139
        - consistency checks like `cli_obj_vs_system_as` and `capture_run_reproducible`
      
        140
        
        141
        The suite DSL is for orchestration:
      
        142
        
        143
        - opts
      
        144
        - compilers and references
      
        145
        - graph composition
      
        146
        - capability policy
      
        147
        - reporting and bundles
      
        148
        
        149
        Leaf assertions should come from shared source directives whenever possible.
      
        150
        
        151
        ## Notes
      
        152
        
        153
        - `.docs/` is local and gitignored.
      
        154
        - The runner is currently strongest on matrices, differential behavior,
      
        155
          capability-aware execution, reports, bundles, and graph orchestration.
      
        156
        - The shared-language reset and follow-through testing roadmap live in the
      
        157
          parent `armfortas` repo under `.docs/testing/`.

1	# bencch
2
3	Structured compiler runner and reporting bench for `armfortas`.
4
5	This repo holds:
6
7	- `bench-core/` — bench-owned compiler-facing types
8	- `bench/` — the `afs-tests` runner
9	- `suites/` — authored bench suites
10	- `fixtures/` — reusable fixture programs
11	- `reports/` — failure and consistency bundles
12
13	## Current Setup
14
15	`bencch` is no longer aiming to be the primary product story for a generic
16	compiler bench. Its role is clearer now:
17
18	- the root armfortas harness is the fast armfortas-first runner and the default
19	home for new source-directed testing ideas
20	- `bencch` is the structured matrix/reporting/differential runner around that
21	same testing language
22
23	Source comments in shared fixtures are the canonical leaf-assertion language.
24	`bencch` should consume those directives where supported and explain
25	unsupported directives clearly, rather than inventing a separate assertion
26	dialect.
27
28	Today `bencch` is wired to a surrounding `armfortas` checkout. The practical way
29	to use it is from the `armfortas` workspace root. CLI-side compiler and tool
30	paths are overridable now; linked capture still comes from the surrounding
31	workspace. That linked compiler surface is currently isolated in
32	`bench/src/compiler.rs`, and the bench-owned compiler-facing types now live in
33	`bench-core/`.
34
35	```bash
36	cargo run -p afs-tests -- list
37	cargo run -p afs-tests -- run --suite frontend
38	```
39
40	Standalone compiler adapters are not finished yet.
41
42	## Usage
43
44	List suites:
45
46	```bash
47	cargo run -p afs-tests -- list
48	```
49
50	Run one suite family:
51
52	```bash
53	cargo run -p afs-tests -- run --suite consistency/runtime
54	```
55
56	Run against an explicit compiler binary:
57
58	```bash
59	cargo run -p afs-tests -- run --suite consistency/runtime-control-flow --armfortas-bin ./target/debug/armfortas
60	```
61
62	Run differential checks with explicit reference compiler paths:
63
64	```bash
65	cargo run -p afs-tests -- run --suite differential/runtime-control-flow --gfortran-bin /opt/homebrew/bin/gfortran --flang-bin /opt/homebrew/bin/flang-new
66	```
67
68	Run one case with full stage capture:
69
70	```bash
71	cargo run -p afs-tests -- run --suite frontend --case stage_walk --all --verbose
72	```
73
74	Run consistency coverage:
75
76	```bash
77	cargo run -p afs-tests -- run --suite consistency --all
78	```
79
80	Run differential coverage:
81
82	```bash
83	cargo run -p afs-tests -- run --suite differential
84	```
85
86	Reports are written under `bencch/reports/`.
87
88	Environment overrides work too:
89
90	```bash
91	BENCCH_ARMFORTAS_BIN=./target/debug/armfortas cargo run -p afs-tests -- run --suite consistency/object
92	```
93
94	## Suite Format
95
96	Suites are plain text files under `suites/`.
97
98	```text
99	suite "consistency/runtime"
100
101	case "mixed_types_cli_run_reproducible"
102	source "../../fixtures/runtime/mixed_types.f90"
103	opts => all
104	armfortas => run
105	repeat => 3
106	consistency => cli_run_reproducible
107	expect run.stdout check-comments
108	expect run.exit_code equals 0
109	end
110	```
111
112	Graph cases use `entry` plus ordered `file` lines:
113
114	```text
115	suite "modules/runtime-graphs"
116
117	case "module_chain_runtime"
118	entry "../../fixtures/modules/module_chain/main.f90"
119	file "../../fixtures/modules/module_chain/math_seed.f90"
120	file "../../fixtures/modules/module_chain/math_values.f90"
121	file "../../fixtures/modules/module_chain/main.f90"
122	opts => O0, O1, O2
123	armfortas => run
124	expect run.stdout check-comments
125	end
126	```
127
128	Today the armfortas adapter materializes graph cases into one generated source
129	in declared file order before capture/compile. The authored files still stay in
130	the failure bundle.
131
132	Common things the runner understands:
133
134	- stage capture like `armfortas => tokens, ir, asm, obj, run`
135	- opt matrices like `opts => O0, O1, O2`
136	- references like `differential => gfortran, flang-new`
137	- expected failures like `xfail "reason"`
138	- per-opt status like `xfail when O1, O2 because "reason"`
139	- consistency checks like `cli_obj_vs_system_as` and `capture_run_reproducible`
140
141	The suite DSL is for orchestration:
142
143	- opts
144	- compilers and references
145	- graph composition
146	- capability policy
147	- reporting and bundles
148
149	Leaf assertions should come from shared source directives whenever possible.
150
151	## Notes
152
153	- `.docs/` is local and gitignored.
154	- The runner is currently strongest on matrices, differential behavior,
155	capability-aware execution, reports, bundles, and graph orchestration.
156	- The shared-language reset and follow-through testing roadmap live in the
157	parent `armfortas` repo under `.docs/testing/`.