@@ -1,14 +1,29 @@ |
| 1 | | -# bensch — Overview |
| 1 | +# bensch — Project Overview |
| 2 | 2 | |
| 3 | | -bensch is a POSIX shell testing framework extracted from the [fortsh](https://github.com/FortranGoingOnForty/fortsh) project's test infrastructure. It provides interactive PTY testing, POSIX compliance verification, and builtin command validation for any shell binary. |
| 3 | +## What |
| 4 | + |
| 5 | +bensch is a POSIX shell testing framework extracted from the [fortsh](https://github.com/FortranGoingOnForty/fortsh) project. It tests any shell binary for POSIX compliance, interactive feature correctness, and builtin command compatibility. |
| 6 | + |
| 7 | +## Why |
| 8 | + |
| 9 | +The fortsh project accumulated ~1014 interactive PTY tests and ~3800 POSIX compliance assertions as part of its CI. 93% of these tests are shell-agnostic — they test POSIX behavior and readline-compatible features, not fortsh-specific functionality. Extracting them into a standalone framework lets other shell projects benefit from the same rigor. |
| 4 | 10 | |
| 5 | 11 | ## Origin |
| 6 | 12 | |
| 7 | | -The framework was developed over 2025-2026 as part of fortsh's CI, growing to 1014 interactive PTY tests and ~3800 POSIX compliance assertions across three platforms (x86 Linux, ARM64 Linux, macOS ARM64). When it became clear that 93% of the tests were shell-agnostic, the framework was extracted into its own project. |
| 13 | +The test infrastructure was developed over 2025-2026 during fortsh's push to achieve full cross-platform interactive parity (x86 Linux, ARM64 Linux, macOS ARM64). The extraction happened in April 2026 after fortsh v1.6.0 achieved 1014/1014 interactive tests passing on all three platforms. |
| 14 | + |
| 15 | +## Architecture Decisions |
| 16 | + |
| 17 | +**Fresh repo, not filter-branch.** fortsh's git history is about a Fortran compiler project. Carrying 1700+ commits of compiler work into a test framework repo would be noise. Each file was brought over as its own commit, preserving provenance in commit messages. |
| 18 | + |
| 19 | +**Shell profiles over runtime detection.** Rather than auto-detecting shell capabilities at runtime (fragile, slow), bensch uses YAML profiles that declaratively describe what each shell supports. This is explicit, testable, and easy to extend. |
| 20 | + |
| 21 | +**Comparison against reference shell.** POSIX tests don't check against a POSIX specification document — they compare against bash (or any reference shell). This is pragmatic: bash IS the de facto standard, and discrepancies between your shell and bash are the actionable findings. |
| 22 | + |
| 23 | +**Separate framework from test suites.** The Python PTY engine (`framework/`) is fully generic. The test specifications (`suites/`) are organized by portability level (POSIX-generic, bash-extended, fortsh-specific). Adding tests for a new shell means adding a profile and optionally a new suite directory. |
| 24 | + |
| 25 | +## Maintenance |
| 8 | 26 | |
| 9 | | -## Architecture |
| 27 | +bensch and fortsh will co-evolve. Bugs found through testing fortsh will be fixed in bensch's framework code. New test cases written for fortsh's interactive features will be added to bensch if they're shell-agnostic. |
| 10 | 28 | |
| 11 | | -- **framework/** — Python PTY engine (pexpect), YAML test runner, output matchers |
| 12 | | -- **suites/** — Test specifications organized by category (POSIX, interactive, builtins) |
| 13 | | -- **profiles/** — Shell capability descriptors (prompt pattern, readline support, rc disable method) |
| 14 | | -- **bensch** — Entry point that wires a shell binary to the appropriate test suites |
| 29 | +The sync is manual but low-friction: both repos share the same test specification format (YAML) and the same Python framework API. A diff between `fortsh/tests/interactive/` and `bensch/framework/` + `bensch/suites/interactive/` will show divergence. |
