@@ -2,6 +2,92 @@ |
| 2 | 2 | |
| 3 | 3 | A POSIX shell testing framework. Pass any shell binary, get a compliance report. |
| 4 | 4 | |
| 5 | +## Quick Start |
| 6 | + |
| 7 | +```bash |
| 8 | +# Test bash |
| 9 | +./bensch --shell /bin/bash --suite posix |
| 10 | + |
| 11 | +# Test a custom shell with interactive PTY tests |
| 12 | +./bensch --shell /path/to/myshell --suite interactive |
| 13 | + |
| 14 | +# Full test run with profile auto-detection |
| 15 | +./bensch --shell /bin/zsh --suite all |
| 16 | + |
| 17 | +# See what's available |
| 18 | +./bensch --list-suites |
| 19 | +./bensch --list-profiles |
| 5 | 20 | ``` |
| 6 | | -bensch --shell /bin/bash --suite posix,interactive |
| 21 | + |
| 22 | +## What It Tests |
| 23 | + |
| 24 | +bensch runs three categories of tests against any shell binary: |
| 25 | + |
| 26 | +| Suite | Tests | What It Checks | |
| 27 | +|-------|-------|----------------| |
| 28 | +| `posix` | ~3,800 | POSIX compliance: expansion, quoting, redirection, control flow, builtins, heredocs, job control | |
| 29 | +| `interactive` | ~1,014 | PTY behavior: line editing, history, tab completion, vi mode, signals, prompts | |
| 30 | +| `builtins` | ~650 | Individual builtin commands compared against a reference shell (bash) | |
| 31 | + |
| 32 | +## How It Works |
| 33 | + |
| 34 | +**POSIX tests** run each command in both the shell under test and a reference shell (default: bash), then compare outputs. This tells you exactly where your shell diverges from established behavior. |
| 35 | + |
| 36 | +**Interactive tests** spawn the shell in a pseudo-terminal via pexpect and drive it with keystrokes — typing commands, pressing Tab, Ctrl+R, arrow keys, Ctrl+C — then verify the terminal output matches expectations. |
| 37 | + |
| 38 | +**Builtin tests** exercise individual commands (`cd`, `export`, `test`, `trap`, etc.) and compare output and exit codes against the reference shell. |
| 39 | + |
| 40 | +## Shell Profiles |
| 41 | + |
| 42 | +Each shell has a profile (`profiles/*.yaml`) describing its capabilities: |
| 43 | + |
| 44 | +```yaml |
| 45 | +# profiles/bash.yaml |
| 46 | +name: bash |
| 47 | +prompt_pattern: '\$ ' |
| 48 | +prompt_set_command: "PS1='$ '" |
| 49 | +mode_reset_command: "set -o emacs" |
| 50 | +capabilities: |
| 51 | + readline: true |
| 52 | + vi_mode: true |
| 53 | + job_control: true |
| 54 | + command_completion: true |
| 7 | 55 | ``` |
| 56 | + |
| 57 | +Profiles control prompt detection, session reset, rc-file disabling, and which test suites are applicable. Shells without readline (like dash) automatically skip interactive editing tests. |
| 58 | + |
| 59 | +Available profiles: `bash`, `zsh`, `dash`, `ksh`, `fortsh`, `generic` |
| 60 | + |
| 61 | +## Requirements |
| 62 | + |
| 63 | +- Python 3.8+ with pip (for interactive tests) |
| 64 | +- bash (as reference shell for comparison tests) |
| 65 | +- A shell binary to test |
| 66 | + |
| 67 | +Dependencies are installed automatically into a local `.venv` on first run. |
| 68 | + |
| 69 | +## Project Structure |
| 70 | + |
| 71 | +``` |
| 72 | +bensch/ |
| 73 | +├── bensch # Entry point script |
| 74 | +├── framework/ # Core PTY engine and test runner |
| 75 | +│ ├── shell_pty.py # pexpect-based PTY wrapper |
| 76 | +│ ├── runner.py # YAML test runner with session management |
| 77 | +│ ├── profile.py # Shell profile loader |
| 78 | +│ └── utils/ # Key definitions, output matchers |
| 79 | +├── suites/ # Test specifications |
| 80 | +│ ├── interactive/ # YAML PTY tests (posix, editing, history, ...) |
| 81 | +│ ├── posix/ # Shell-script POSIX compliance tests |
| 82 | +│ └── builtins/ # Builtin command tests (portable, extended) |
| 83 | +├── profiles/ # Shell capability profiles |
| 84 | +└── docs/ # Documentation |
| 85 | +``` |
| 86 | + |
| 87 | +## Origin |
| 88 | + |
| 89 | +bensch was extracted from the [fortsh](https://github.com/FortranGoingOnForty/fortsh) project's test infrastructure, which achieved 1014/1014 interactive test parity across x86 Linux, ARM64 Linux, and macOS ARM64. The framework is 93% shell-agnostic by design. |
| 90 | + |
| 91 | +## License |
| 92 | + |
| 93 | +GPLv3 |
