@@ -0,0 +1,230 @@ |
| 1 | +# SSH deploy notes |
| 2 | + |
| 3 | +S07 ships the SSH-key data layer + the `AuthorizedKeysCommand` (AKC) integration. To turn it into a working git-over-SSH endpoint, sshd needs a small amount of operator setup. This doc captures what that looks like and the security knobs that matter. The actual git protocol handler lands in S13. |
| 4 | + |
| 5 | +## Architecture |
| 6 | + |
| 7 | +``` |
| 8 | +ssh client |
| 9 | + │ TCP/22 |
| 10 | + ▼ |
| 11 | +sshd (system) ──► AuthorizedKeysCommand: shithubd ssh-authkeys <fingerprint> |
| 12 | + ─ stdout: a single authorized_keys line, OR empty |
| 13 | + ─ exit 0 in both cases (failing closed on any error) |
| 14 | + └──► forced command per the line: |
| 15 | + shithubd ssh-shell <user_id> |
| 16 | + (S13 dispatcher; S07 placeholder logs and exits non-zero) |
| 17 | +``` |
| 18 | + |
| 19 | +Why AKC instead of a static `~/.ssh/authorized_keys`: |
| 20 | + |
| 21 | +- The static-file approach requires regenerating a file on every key change and has weak consistency guarantees with replicas. |
| 22 | +- AKC is sshd's purpose-built mechanism for dynamic lookups; failure semantics are well-understood. |
| 23 | +- The forced command + restrictive options strip every interactive affordance, so a hijacked key can only invoke `shithubd ssh-shell` — never get a real shell. |
| 24 | + |
| 25 | +## Linux user setup |
| 26 | + |
| 27 | +A dedicated low-privilege system user runs the AKC binary. It owns no files, has no shell, and never logs in interactively. |
| 28 | + |
| 29 | +```sh |
| 30 | +sudo useradd --system --no-create-home --shell /usr/sbin/nologin shithub-ssh |
| 31 | +``` |
| 32 | + |
| 33 | +The shithub binary lives somewhere both root and `shithub-ssh` can read: |
| 34 | + |
| 35 | +```sh |
| 36 | +sudo install -m 0755 ./bin/shithubd /usr/local/bin/shithubd |
| 37 | +``` |
| 38 | + |
| 39 | +Configuration (`SHITHUB_DATABASE_URL`, etc.) lives in a 0600-mode file readable only by `shithub-ssh`: |
| 40 | + |
| 41 | +```sh |
| 42 | +sudo install -d -m 0750 -o shithub-ssh -g shithub-ssh /etc/shithub |
| 43 | +sudoedit /etc/shithub/ssh.env # 0600, owned by shithub-ssh |
| 44 | +``` |
| 45 | + |
| 46 | +Recommended `/etc/shithub/ssh.env`: |
| 47 | + |
| 48 | +```sh |
| 49 | +SHITHUB_DATABASE_URL=postgres://shithub_ssh:****@127.0.0.1:5432/shithub?sslmode=disable |
| 50 | +``` |
| 51 | + |
| 52 | +## sshd configuration |
| 53 | + |
| 54 | +``` |
| 55 | +# /etc/ssh/sshd_config.d/shithub.conf |
| 56 | + |
| 57 | +# Run AKC for ALL connections that match the standard auth path. |
| 58 | +AuthorizedKeysCommand /usr/local/bin/shithubd ssh-authkeys %f |
| 59 | +AuthorizedKeysCommandUser shithub-ssh |
| 60 | + |
| 61 | +# Disable password auth and host-based auth — only key-based. |
| 62 | +PasswordAuthentication no |
| 63 | +PubkeyAuthentication yes |
| 64 | + |
| 65 | +# AKC only sees the connecting client's pubkey FINGERPRINT (%f), not |
| 66 | +# the full key. shithubd looks it up by fingerprint and emits the matching |
| 67 | +# stored key + forced-command line, which sshd then uses to authenticate. |
| 68 | + |
| 69 | +# Mitigate connection floods. Real numbers depend on traffic profile — |
| 70 | +# these are reasonable defaults. |
| 71 | +MaxStartups 30:30:100 |
| 72 | +LoginGraceTime 20s |
| 73 | + |
| 74 | +# Standard hardening. Already on most modern distros; spelled out here for |
| 75 | +# review: |
| 76 | +PermitRootLogin no |
| 77 | +PermitEmptyPasswords no |
| 78 | +ChallengeResponseAuthentication no |
| 79 | +UsePAM no # we own auth end-to-end via AKC |
| 80 | +ClientAliveInterval 60 |
| 81 | +ClientAliveCountMax 3 |
| 82 | +``` |
| 83 | + |
| 84 | +Wrapping the AKC invocation through `EnvironmentFile` to load |
| 85 | +`SHITHUB_DATABASE_URL` is the cleanest way to keep secrets out of `argv`: |
| 86 | +the simplest path is a 2-line wrapper script in `/usr/local/bin/`: |
| 87 | + |
| 88 | +```sh |
| 89 | +#!/bin/sh |
| 90 | +# /usr/local/bin/shithub-ssh-authkeys |
| 91 | +set -e |
| 92 | +. /etc/shithub/ssh.env |
| 93 | +exec /usr/local/bin/shithubd ssh-authkeys "$1" |
| 94 | +``` |
| 95 | + |
| 96 | +Then `AuthorizedKeysCommand /usr/local/bin/shithub-ssh-authkeys %f`. The |
| 97 | +wrapper is owned by root and 0755 so sshd can verify the path's ownership |
| 98 | +chain (sshd refuses to invoke an AKC whose path or any parent is |
| 99 | +group/world writable). |
| 100 | + |
| 101 | +Validate before reloading: |
| 102 | + |
| 103 | +```sh |
| 104 | +sudo sshd -t # config syntax check |
| 105 | +sudo systemctl reload sshd # apply |
| 106 | +``` |
| 107 | + |
| 108 | +## Postgres role separation |
| 109 | + |
| 110 | +The AKC binary runs under a low-privilege role with the minimum surface |
| 111 | +needed to look up keys and update last-used columns. Everything else |
| 112 | +(insert / delete / etc.) goes through the web binary's normal role. |
| 113 | + |
| 114 | +```sql |
| 115 | +CREATE ROLE shithub_ssh LOGIN PASSWORD '****'; |
| 116 | +GRANT CONNECT ON DATABASE shithub TO shithub_ssh; |
| 117 | +GRANT USAGE ON SCHEMA public TO shithub_ssh; |
| 118 | +GRANT SELECT ON user_ssh_keys TO shithub_ssh; |
| 119 | +GRANT UPDATE (last_used_at, last_used_ip) ON user_ssh_keys TO shithub_ssh; |
| 120 | +``` |
| 121 | + |
| 122 | +Read-only would be cleaner but precludes the last-used update. The |
| 123 | +column-level UPDATE grant is the smallest privilege that still serves the |
| 124 | +operational need. |
| 125 | + |
| 126 | +## AKC behavior contract |
| 127 | + |
| 128 | +The AKC subcommand has three outcomes: |
| 129 | + |
| 130 | +| Input | Output | Exit | |
| 131 | +|---|---|---| |
| 132 | +| Known fingerprint | `command="..." options... <algo> <b64>` (single line) | 0 | |
| 133 | +| Unknown fingerprint | empty stdout | 0 | |
| 134 | +| Any error (DB down, panic, malformed input) | empty stdout | 0 | |
| 135 | + |
| 136 | +**sshd reads stdout content as the auth answer; non-zero exit is a |
| 137 | +configuration error, not a deny.** Failing closed (return empty on |
| 138 | +unrecoverable conditions) is therefore the correct posture: better to |
| 139 | +deny a legitimate connection than to authorize the wrong user. |
| 140 | + |
| 141 | +The forced-command line emitted on a hit: |
| 142 | + |
| 143 | +``` |
| 144 | +command="/usr/local/bin/shithubd ssh-shell <user_id>",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty <algo> <b64> |
| 145 | +``` |
| 146 | + |
| 147 | +The option set strips every interactive affordance — defense in depth on |
| 148 | +top of `shithubd ssh-shell`'s own command parsing. |
| 149 | + |
| 150 | +## Latency |
| 151 | + |
| 152 | +AKC runs on every SSH connection. Targets: |
| 153 | + |
| 154 | +- p99 < 100 ms with a warm DB on the production droplet. |
| 155 | +- Per-process pool sized small (`MaxConns: 4`) — sshd spawns a fresh |
| 156 | + process per connection, so the pool's lifetime is the connection's. |
| 157 | +- Connect timeout capped at 750 ms so a flaky DB doesn't stall sshd. |
| 158 | + |
| 159 | +`shithub_http_*` metrics are emitted by the web binary, not by AKC. Add a |
| 160 | +synthetic check that calls `shithubd ssh-authkeys <test-fp>` periodically |
| 161 | +and alerts if it exceeds the SLO. |
| 162 | + |
| 163 | +## Last-used tracking |
| 164 | + |
| 165 | +The AKC subcommand updates `user_ssh_keys.last_used_at` and |
| 166 | +`last_used_ip` after a successful lookup. It runs in a fire-and-forget |
| 167 | +goroutine with a 500 ms timeout; any error is silently dropped. This |
| 168 | +keeps the hot path fast at the cost of occasional missed updates under |
| 169 | +DB pressure — acceptable for a UI-only display field. |
| 170 | + |
| 171 | +If a future sprint needs strict last-used accuracy (anomaly detection, |
| 172 | +forensic timeline), promote the path to a small append-only log + worker |
| 173 | +roll-up. Don't make AKC's success contingent on the update. |
| 174 | + |
| 175 | +## Operational notes |
| 176 | + |
| 177 | +- **Deleting a key while a session is active:** sshd doesn't re-auth |
| 178 | + mid-session. Existing sessions persist until disconnect. Document this |
| 179 | + in user-facing security help; it's expected SSH behavior. |
| 180 | +- **Fingerprint canonicalization:** the codebase uses |
| 181 | + `SHA256:<base64-no-padding>`. Both `ssh.FingerprintSHA256` and the |
| 182 | + `ssh-keygen -E sha256 -lf <pubfile>` output produce this exact format. |
| 183 | + When matching by hand, copy from the fingerprint shown in the user's |
| 184 | + SSH-keys settings page, NOT from `ssh-keygen -lf` (which uses MD5 |
| 185 | + unless `-E sha256` is passed). |
| 186 | +- **Fail2ban / connection-rate limiting:** lives at the OS layer. Document |
| 187 | + desired thresholds for ops in the S37 deploy bundle. AKC alone does |
| 188 | + not throttle — that's sshd's `MaxStartups` and the firewall's job. |
| 189 | + |
| 190 | +## Smoke test |
| 191 | + |
| 192 | +```sh |
| 193 | +# 1. Bring up dev Postgres + apply migrations. |
| 194 | +make dev-db && make build && SHITHUB_DATABASE_URL=... ./bin/shithubd migrate up |
| 195 | + |
| 196 | +# 2. Add a key (via the UI or a direct INSERT for testing). |
| 197 | + |
| 198 | +# 3. Invoke the AKC subcommand with a known fingerprint: |
| 199 | +SHITHUB_DATABASE_URL=... ./bin/shithubd ssh-authkeys "SHA256:<...>" |
| 200 | +# Expect: a single authorized_keys line on stdout, exit 0. |
| 201 | + |
| 202 | +# 4. Invoke with an unknown fingerprint: |
| 203 | +SHITHUB_DATABASE_URL=... ./bin/shithubd ssh-authkeys "SHA256:not-a-real-fingerprint-xxxx" |
| 204 | +# Expect: empty stdout, exit 0. |
| 205 | + |
| 206 | +# 5. Connect via SSH (placeholder shell): |
| 207 | +ssh -p 22 git@<host> |
| 208 | +# Expect: stderr line "shithubd ssh-shell: user_id=N original_command=..." |
| 209 | +# Exit non-zero with "git over SSH not enabled yet" — replaced fully in S13. |
| 210 | +``` |
| 211 | + |
| 212 | +## Pitfalls / risks |
| 213 | + |
| 214 | +- **Wrong-user authorization** is the catastrophic bug. Audit the |
| 215 | + unknown-fingerprint and DB-error paths in any future change to |
| 216 | + `cmd/shithubd/ssh.go`. |
| 217 | +- **Stray newlines / unescaped options** in the emitted line break sshd |
| 218 | + parsing in subtle ways. The codebase emits a strict template; don't |
| 219 | + introduce dynamic options without escaping. |
| 220 | +- **Group/world-writable paths in the AKC chain** make sshd refuse to |
| 221 | + invoke AKC. The wrapper script and binary must be 0755, parents 0755. |
| 222 | +- **Postgres role drift:** if `shithub_ssh` ever gains broader privileges, |
| 223 | + a compromise of the AKC binary becomes a much bigger event. Audit |
| 224 | + grants on every deploy. |
| 225 | + |
| 226 | +## Related docs |
| 227 | + |
| 228 | +- `docs/internal/auth.md` — email/password auth (S05). |
| 229 | +- `docs/internal/2fa.md` — TOTP + recovery codes (S06). |
| 230 | +- `docs/internal/observability.md` — slog redaction. |