tenseleyflow/bensch / be8ebe0

Browse files

write documentation: YAML spec format, shell profiles, writing tests

docs/yaml-spec-format.md — complete reference for test YAML format
docs/shell-profiles.md — profile system, creating profiles, capabilities
docs/writing-tests.md — guidelines for portable shell-agnostic tests
Authored by espadonne
SHA
be8ebe063a4b949d6dfc5b7bb0d09e26a5136b07
Parents
4e0a449
Tree
b46b1d4

4 changed files

StatusFile+-
D docs/.gitkeep 0 0
A docs/shell-profiles.md 80 0
A docs/writing-tests.md 109 0
A docs/yaml-spec-format.md 75 0
docs/.gitkeepdeleted
docs/shell-profiles.mdadded
@@ -0,0 +1,80 @@
1
+# Shell Profiles
2
+
3
+Profiles tell bensch how to interact with a specific shell. They live in `profiles/` as YAML files.
4
+
5
+## Profile Fields
6
+
7
+```yaml
8
+name: bash                              # Internal identifier
9
+display_name: "GNU Bash"                # Human-readable name
10
+
11
+prompt_pattern: '\$ '                   # Regex to detect the shell prompt
12
+prompt_set_command: "PS1='$ '"          # Command to set a known prompt
13
+mode_reset_command: "set -o emacs"      # Reset to known editing mode
14
+
15
+rc_disable:                             # How to skip rc file loading
16
+  flags: ["--norc", "--noprofile"]      # CLI flags
17
+  env:                                  # Environment variables
18
+    BASH_ENV: ""
19
+
20
+test_mode_env:                          # Extra env for cleaner test output
21
+  FORTSH_MINIMAL_ECHO: "1"             # (shell-specific, only if needed)
22
+
23
+history_disable:                        # Prevent history file interference
24
+  env:
25
+    HISTFILE: /dev/null
26
+
27
+capabilities:                           # What features the shell supports
28
+  readline: true
29
+  vi_mode: true
30
+  job_control: true
31
+  command_completion: true
32
+  programmable_completion: true
33
+  coproc: true
34
+  arrays: true
35
+  associative_arrays: true
36
+
37
+suites:                                 # Suite selection overrides
38
+  skip:                                 # Suites to skip for this shell
39
+    - "interactive/editing"
40
+  extra:                                # Extra suites to include
41
+    - "builtins/extended/fortsh"
42
+```
43
+
44
+## Creating a Profile for a New Shell
45
+
46
+1. Copy `profiles/generic.yaml` to `profiles/myshell.yaml`
47
+2. Set the `prompt_pattern` to match your shell's default prompt
48
+3. Set `prompt_set_command` to a command that forces a known prompt
49
+4. Set `mode_reset_command` if the shell supports emacs/vi modes
50
+5. Configure `rc_disable` with the appropriate flags or env vars
51
+6. Set `capabilities` truthfully — tests will be skipped for missing capabilities
52
+7. Add any `test_mode_env` your shell supports for cleaner test output
53
+
54
+## Auto-Detection
55
+
56
+When `--profile` is not specified, bensch detects the profile from the binary name:
57
+
58
+| Binary | Profile |
59
+|--------|---------|
60
+| `bash*` | bash |
61
+| `zsh*` | zsh |
62
+| `dash*` | dash |
63
+| `ksh*`, `mksh*`, `pdksh*` | ksh |
64
+| `fortsh*` | fortsh |
65
+| anything else | generic |
66
+
67
+## Capability Flags
68
+
69
+Capabilities determine which test suites run:
70
+
71
+| Capability | Required For |
72
+|------------|-------------|
73
+| `readline` | Line editing, history navigation tests |
74
+| `vi_mode` | Vi editing mode tests |
75
+| `job_control` | Signal and background job tests |
76
+| `command_completion` | Tab completion tests |
77
+| `programmable_completion` | `complete`/`compgen` builtin tests |
78
+| `coproc` | Coprocess tests |
79
+| `arrays` | Indexed array tests |
80
+| `associative_arrays` | Associative array tests |
docs/writing-tests.mdadded
@@ -0,0 +1,109 @@
1
+# Writing Portable Shell Tests
2
+
3
+Guidelines for writing tests that work across different shells.
4
+
5
+## POSIX Script Tests
6
+
7
+The POSIX tests use a comparison pattern: run the same command in both the reference shell and the shell under test, then compare outputs.
8
+
9
+```sh
10
+# In your test script, source the shared harness
11
+. "$SCRIPT_DIR/../test_harness.sh"
12
+
13
+section "100. MY TESTS"
14
+
15
+# Compare output against reference shell
16
+compare_output "echo works" 'echo hello world'
17
+
18
+# Compare exit code only
19
+compare_exit "false returns 1" 'false'
20
+
21
+# Compare both output and exit code
22
+compare_both "variable expansion" 'X=hello; echo $X'
23
+```
24
+
25
+### Normalization
26
+
27
+Error messages differ between shells (`bash: foo: not found` vs `sh: foo: not found`). The `normalize_shell_name` function strips shell-specific prefixes before comparison:
28
+
29
+```sh
30
+# This handles any shell name prefix automatically
31
+normalize_shell_name "zsh: no such file" → "SHELL: no such file"
32
+```
33
+
34
+### RC File Disabling
35
+
36
+Tests should run in a clean environment. Use `$RC_DISABLE_ENV` (set by bensch from the shell profile) to suppress rc file loading:
37
+
38
+```sh
39
+output=$(env $RC_DISABLE_ENV "$SHELL_BIN" -c "$command" 2>&1)
40
+```
41
+
42
+## Interactive YAML Tests
43
+
44
+### Keep Tests Shell-Agnostic
45
+
46
+- Don't test shell-specific prompt formats (use `expect_output` for command output, not prompt text)
47
+- Don't test `$0` or shell name output
48
+- Use POSIX builtins and syntax
49
+- Don't assume specific error message wording
50
+
51
+### Good Test Patterns
52
+
53
+```yaml
54
+# Good: tests POSIX behavior
55
+- name: "variable expansion"
56
+  steps:
57
+    - send_line: "X=hello; echo $X"
58
+  expect_output: "hello"
59
+
60
+# Good: tests interactive feature generically
61
+- name: "up arrow recalls history"
62
+  steps:
63
+    - send_line: "echo recall_me"
64
+    - send_key: "Up"
65
+    - send_key: "Enter"
66
+  expect_output: "recall_me"
67
+```
68
+
69
+### Patterns to Avoid
70
+
71
+```yaml
72
+# Bad: shell-specific prompt
73
+- name: "prompt shows"
74
+  expect_output: "fortsh>"  # Only works for fortsh
75
+
76
+# Bad: shell-specific builtin
77
+- name: "abbreviation"
78
+  steps:
79
+    - send_line: "abbr g=git"  # fortsh-only feature
80
+```
81
+
82
+### Fresh Sessions
83
+
84
+For tests that modify shell state (set -o vi, PS1 changes), use `fresh_session: true` to get a clean PTY:
85
+
86
+```yaml
87
+- name: "vi mode works"
88
+  fresh_session: true
89
+  steps:
90
+    - send_line: "set -o vi"
91
+    - send: "echo test"
92
+    - send_key: "Escape"
93
+    # ...
94
+```
95
+
96
+### Custom Environment
97
+
98
+Pass environment variables to the test session:
99
+
100
+```yaml
101
+- name: "variable completion"
102
+  env:
103
+    MYVAR: "testvalue"
104
+  steps:
105
+    - send: "echo $MYV"
106
+    - send_key: "Tab"
107
+    - send_key: "Enter"
108
+  expect_output: "testvalue"
109
+```
docs/yaml-spec-format.mdadded
@@ -0,0 +1,75 @@
1
+# YAML Test Specification Format
2
+
3
+bensch interactive tests are defined in YAML files. Each file contains a list of tests with steps and expected output.
4
+
5
+## Structure
6
+
7
+```yaml
8
+metadata:
9
+  category: "History"
10
+  description: "Tests for command history navigation"
11
+
12
+tests:
13
+  - name: "Up arrow recalls previous command"
14
+    steps:
15
+      - send_line: "echo hello"
16
+      - send_key: "Up"
17
+      - send_key: "Enter"
18
+    expect_output: "hello"
19
+    match_type: "contains"
20
+```
21
+
22
+## Step Types
23
+
24
+| Step | Description | Example |
25
+|------|-------------|---------|
26
+| `send_line` | Send text + Enter | `send_line: "echo hello"` |
27
+| `send` | Send text without Enter | `send: "partial"` |
28
+| `send_key` | Send a named key | `send_key: "Up"` |
29
+| `send_keys` | Send multiple keys | `send_keys: ["C-a", "C-k"]` |
30
+| `wait` | Sleep for seconds | `wait: 0.3` |
31
+| `wait_for_prompt` | Wait for shell prompt | `wait_for_prompt: true` |
32
+| `expect` | Wait for pattern in output | `expect: "pattern"` |
33
+| `resize` | Change terminal size | `resize: {rows: 24, cols: 80}` |
34
+
35
+## Key Names
36
+
37
+Standard terminal keys are available via name:
38
+
39
+- **Arrow keys**: `Up`, `Down`, `Left`, `Right`
40
+- **Control keys**: `C-a` through `C-z`, `C-c`, `C-d`, `C-r`, `C-l`
41
+- **Alt keys**: `M-f`, `M-b`, `M-d`, `M-.`
42
+- **Special**: `Enter`, `Tab`, `Escape`, `Backspace`, `Delete`
43
+- **Function keys**: `F1` through `F12`
44
+- **Navigation**: `Home`, `End`, `PageUp`, `PageDown`
45
+
46
+## Match Types
47
+
48
+| Type | Description |
49
+|------|-------------|
50
+| `contains` | Output contains the expected string (default) |
51
+| `exact` | Output exactly matches |
52
+| `regex` | Output matches regex pattern |
53
+
54
+## Test Options
55
+
56
+| Option | Description | Default |
57
+|--------|-------------|---------|
58
+| `fresh_session` | Force a new PTY session | `false` |
59
+| `env` | Extra environment variables | `{}` |
60
+| `rc_file` | RC file path | `/dev/null` |
61
+| `expect_not` | Output must NOT contain this | — |
62
+
63
+## Example: Tab Completion Test
64
+
65
+```yaml
66
+- name: "Tab completes filename"
67
+  steps:
68
+    - send_line: "cd /tmp"
69
+    - send_line: "touch testfile.txt"
70
+    - send: "ls testf"
71
+    - send_key: "Tab"
72
+    - send_key: "Enter"
73
+  expect_output: "testfile.txt"
74
+  match_type: "contains"
75
+```