bensch Public

Watch 0 Fork 0 Star 0
markdown · 2607 bytes Raw Blame History

Writing Portable Shell Tests

Guidelines for writing tests that work across different shells.

POSIX Script Tests

The POSIX tests use a comparison pattern: run the same command in both the reference shell and the shell under test, then compare outputs.

# In your test script, source the shared harness
. "$SCRIPT_DIR/../test_harness.sh"

section "100. MY TESTS"

# Compare output against reference shell
compare_output "echo works" 'echo hello world'

# Compare exit code only
compare_exit "false returns 1" 'false'

# Compare both output and exit code
compare_both "variable expansion" 'X=hello; echo $X'

Normalization

Error messages differ between shells (bash: foo: not found vs sh: foo: not found). The normalize_shell_name function strips shell-specific prefixes before comparison:

# This handles any shell name prefix automatically
normalize_shell_name "zsh: no such file" → "SHELL: no such file"

RC File Disabling

Tests should run in a clean environment. Use $RC_DISABLE_ENV (set by bensch from the shell profile) to suppress rc file loading:

output=$(env $RC_DISABLE_ENV "$SHELL_BIN" -c "$command" 2>&1)

Interactive YAML Tests

Keep Tests Shell-Agnostic

  • Don't test shell-specific prompt formats (use expect_output for command output, not prompt text)
  • Don't test $0 or shell name output
  • Use POSIX builtins and syntax
  • Don't assume specific error message wording

Good Test Patterns

# Good: tests POSIX behavior
- name: "variable expansion"
  steps:
    - send_line: "X=hello; echo $X"
  expect_output: "hello"

# Good: tests interactive feature generically
- name: "up arrow recalls history"
  steps:
    - send_line: "echo recall_me"
    - send_key: "Up"
    - send_key: "Enter"
  expect_output: "recall_me"

Patterns to Avoid

# Bad: shell-specific prompt
- name: "prompt shows"
  expect_output: "fortsh>"  # Only works for fortsh

# Bad: shell-specific builtin
- name: "abbreviation"
  steps:
    - send_line: "abbr g=git"  # fortsh-only feature

Fresh Sessions

For tests that modify shell state (set -o vi, PS1 changes), use fresh_session: true to get a clean PTY:

- name: "vi mode works"
  fresh_session: true
  steps:
    - send_line: "set -o vi"
    - send: "echo test"
    - send_key: "Escape"
    # ...

Custom Environment

Pass environment variables to the test session:

- name: "variable completion"
  env:
    MYVAR: "testvalue"
  steps:
    - send: "echo $MYV"
    - send_key: "Tab"
    - send_key: "Enter"
  expect_output: "testvalue"
View source
1 # Writing Portable Shell Tests
2
3 Guidelines for writing tests that work across different shells.
4
5 ## POSIX Script Tests
6
7 The POSIX tests use a comparison pattern: run the same command in both the reference shell and the shell under test, then compare outputs.
8
9 ```sh
10 # In your test script, source the shared harness
11 . "$SCRIPT_DIR/../test_harness.sh"
12
13 section "100. MY TESTS"
14
15 # Compare output against reference shell
16 compare_output "echo works" 'echo hello world'
17
18 # Compare exit code only
19 compare_exit "false returns 1" 'false'
20
21 # Compare both output and exit code
22 compare_both "variable expansion" 'X=hello; echo $X'
23 ```
24
25 ### Normalization
26
27 Error messages differ between shells (`bash: foo: not found` vs `sh: foo: not found`). The `normalize_shell_name` function strips shell-specific prefixes before comparison:
28
29 ```sh
30 # This handles any shell name prefix automatically
31 normalize_shell_name "zsh: no such file" → "SHELL: no such file"
32 ```
33
34 ### RC File Disabling
35
36 Tests should run in a clean environment. Use `$RC_DISABLE_ENV` (set by bensch from the shell profile) to suppress rc file loading:
37
38 ```sh
39 output=$(env $RC_DISABLE_ENV "$SHELL_BIN" -c "$command" 2>&1)
40 ```
41
42 ## Interactive YAML Tests
43
44 ### Keep Tests Shell-Agnostic
45
46 - Don't test shell-specific prompt formats (use `expect_output` for command output, not prompt text)
47 - Don't test `$0` or shell name output
48 - Use POSIX builtins and syntax
49 - Don't assume specific error message wording
50
51 ### Good Test Patterns
52
53 ```yaml
54 # Good: tests POSIX behavior
55 - name: "variable expansion"
56 steps:
57 - send_line: "X=hello; echo $X"
58 expect_output: "hello"
59
60 # Good: tests interactive feature generically
61 - name: "up arrow recalls history"
62 steps:
63 - send_line: "echo recall_me"
64 - send_key: "Up"
65 - send_key: "Enter"
66 expect_output: "recall_me"
67 ```
68
69 ### Patterns to Avoid
70
71 ```yaml
72 # Bad: shell-specific prompt
73 - name: "prompt shows"
74 expect_output: "fortsh>" # Only works for fortsh
75
76 # Bad: shell-specific builtin
77 - name: "abbreviation"
78 steps:
79 - send_line: "abbr g=git" # fortsh-only feature
80 ```
81
82 ### Fresh Sessions
83
84 For tests that modify shell state (set -o vi, PS1 changes), use `fresh_session: true` to get a clean PTY:
85
86 ```yaml
87 - name: "vi mode works"
88 fresh_session: true
89 steps:
90 - send_line: "set -o vi"
91 - send: "echo test"
92 - send_key: "Escape"
93 # ...
94 ```
95
96 ### Custom Environment
97
98 Pass environment variables to the test session:
99
100 ```yaml
101 - name: "variable completion"
102 env:
103 MYVAR: "testvalue"
104 steps:
105 - send: "echo $MYV"
106 - send_key: "Tab"
107 - send_key: "Enter"
108 expect_output: "testvalue"
109 ```