`11f0fda`

S14: docs/internal/{hooks,worker}.md

Authored by mfwolffe <wolffemf@dukes.jmu.edu> 6 days ago

SHA: 11f0fdaf85bfc8192a65ff31a0305145d37476f8
Parents: aa9ab45
Tree: e17b934

2 changed files

Status	File	+	-
A	`docs/internal/hooks.md`	105	0
A	`docs/internal/worker.md`	96	0

docs/internal/hooks.mdadded

++# Git hooks
++
++Bare-repo git hooks are how the push pipeline (S14) gets called.
++shithub installs **pre-receive** (gates) and **post-receive** (enqueue)
++on every repo. Both are tiny shell shims that exec into the same
++`shithubd` binary the web/SSH/worker layers run.
++
++## Why shell shims and not direct binary symlinks
++
++git invokes hooks with stdin piped, a particular cwd, and a controlled
++env. The shim normalizes the entry point: `exec /path/to/shithubd hook
++<name>`. Hooks aren't symlinked because (a) macOS git treats some
++symlinks oddly under hardened runtime, and (b) the shim explicitly
++re-exec's so signals and exit codes propagate cleanly.
++
++The shim is regenerated on every `hooks.Install` call — there's no
++manual editing path, by design. If the binary moves (deploy upgrade,
++versioned install path), run `shithubd hooks reinstall --all` to point
++every repo at the new location.
++
++## Hook execution flow
++
++```
++git push  ──▶  receive-pack
++              │
++              ▼
++         pre-receive  (one process per push)
++              │ stdin: "<old> <new> <ref>" lines
++              │ env:   SHITHUB_USER_ID / _USERNAME / _REPO_ID / _REPO_FULL_NAME
++              │        SHITHUB_PROTOCOL / _REMOTE_IP / _REQUEST_ID
++              ▼
++         exec shithubd hook pre-receive
++              │ exit 0 → accept
++              │ exit ≠ 0 → reject; stderr is shown to the pusher
++              ▼
++       (push proceeds)
++              │
++              ▼
++         post-receive  (one process per push)
++              │ stdin: same shape
++              ▼
++         exec shithubd hook post-receive
++              │ INSERT push_events row per ref
++              │ INSERT job (push:process)
++              │ NOTIFY shithub_jobs
++              │ exit 0
++              ▼
++       worker picks it up via LISTEN
++```
++
++## pre-receive contract
++
++Implements the **minimum gates** described in S14. The full branch
++protection engine is S20.
++
++Re-checks the following from the DB even though the env carries them
++(env can be stale on a long-lived SSH session):
++
++* User not suspended (`users.suspended_at IS NULL`).
++* Repo not archived (`repos.is_archived = false`).
++* Repo not soft-deleted (`repos.deleted_at IS NULL`).
++
++Failures emit a `shithub: ...` line on stderr that git surfaces directly
++to the pusher's terminal. Latency budget: <100ms p99.
++
++## post-receive contract
++
++* Reads stdin, ignores empty/malformed lines.
++* For each `<old> <new> <ref>` line: INSERT a `push_events` row, then
++  enqueue a `push:process` job carrying the new event id.
++* Issues a single `NOTIFY shithub_jobs` per push (workers wake on the
++  next tx commit).
++* Exits 0 even on internal errors. The push has already landed; the
++  pipeline is async — partial enqueue failures surface in the worker
++  logs and a backstop reconciler (post-MVP) can re-process orphaned
++  events.
++
++## Installation
++
++* On repo create (`internal/repos/create.go::Create`): runs
++  `hooks.Install` after `RepoFS.InitBare` succeeds. The S11 plumbing
++  initial commit does *not* fire hooks — that's correct, the contract
++  is hooks fire on user-driven pushes only.
++* On deploy: `shithubd hooks reinstall --all` walks every active repo
++  via the DB and re-installs. Single repos: `--repo owner/name`.
++* The binary path baked into the shim is `os.Executable()` of the
++  running shithubd at install time, resolved through `filepath.Abs`.
++  Test fixtures that don't exercise hooks pass `ShithubdPath: ""` to
++  `repos.Create.Deps` to skip installation.
++
++## Failure modes worth knowing
++
++* **`SHITHUB_*` env not set** (e.g. someone manually triggered a push
++  via a path that bypassed S12/S13): pre-receive returns the "missing
++  context" error. post-receive logs a warning and exits 0 — the push
++  still lands.
++* **DB unreachable from hook**: pre-receive returns "server error";
++  the user sees a generic message, the push aborts. post-receive logs
++  and exits 0; backstop reconciler will pick up the unprocessed push
++  on next opportunity.
++* **Binary path drift between install and execution**: the shim's hard-
++  coded path no longer exists. git will report "hook execution failed"
++  and abort the push. Operators recover with `hooks reinstall`.
++* **Stale shim from previous shithubd version**: `Install` is
++  idempotent and overwrites; re-running on a deploy is the right move.

docs/internal/worker.mdadded

++# Worker
++
++The worker pool drains the Postgres-backed job queue introduced in S14.
++One binary serves the web API, the SSH dispatcher, the hooks, and the
++worker — `shithubd worker` boots a long-running process whose only job
++is to dequeue and run.
++
++## Architecture
++
++```
++post-receive  ──INSERT push_event + INSERT job + NOTIFY──▶  Postgres jobs
++                                                                  │
++                                                                  ▼
++                                                         shithubd worker
++                                                  ┌──────────────────────┐
++                                                  │ N goroutines          │
++                                                  │ ClaimJob (FOR UPDATE  │
++                                                  │   SKIP LOCKED)        │
++                                                  │ → handler             │
++                                                  │ → MarkCompleted /     │
++                                                  │   Reschedule /        │
++                                                  │   MarkFailed          │
++                                                  └──────────────────────┘
++```
++
++The pool also runs a dedicated LISTEN goroutine that holds one Postgres
++connection on `LISTEN shithub_jobs`. A NOTIFY from any process (typically
++the post-receive hook) wakes idle workers in <1s without polling. The
++backstop poll (every 5s by default) covers dropped notifications.
++
++## Job kinds shipped in S14
++
++| Kind                   | Trigger                              | Idempotent on            |
++| ---------------------- | ------------------------------------ | ------------------------ |
++| `push:process`         | post-receive hook per ref            | `push_events.processed_at` |
++| `repo:size_recalc`     | enqueued by `push:process`           | overwrite-last-wins        |
++| `jobs:purge_completed` | future cron / manual ad-hoc          | always safe to re-run      |
++
++Adding a new kind: write the handler in `internal/worker/jobs/<kind>.go`,
++add the `Kind` constant to `internal/worker/types.go`, register it in
++`cmd/shithubd/worker.go`. The dispatch loop is generic — no other code
++needs to change.
++
++## Failure handling
++
++* **Transient errors** (handler returns a non-nil non-poison error):
++  reschedule with `Backoff(attempts) ± 20% jitter`, where `Backoff` is
++  `30s * 2^(attempts-1)` capped at 1h.
++* **Poison errors** (handler returns an error wrapping `worker.ErrPoison`):
++  jump to `MarkJobFailed` immediately. Use this for malformed payloads or
++  references to vanished rows.
++* **Panics**: caught by `safeRun` and treated as a transient error so a
++  buggy handler can't take down a worker goroutine.
++* **Stuck locks**: `ClaimJob` ignores rows whose `locked_at` is older
++  than 5 minutes — a worker that died mid-job releases its rows on the
++  next claim cycle.
++* **Max attempts**: `jobs.max_attempts` (default 5). Past the limit the
++  row is moved to `failed_at`. Failed rows surface in the S34 admin
++  panel for poison-job triage.
++
++## Concurrency contract
++
++* `FOR UPDATE SKIP LOCKED` on the inner SELECT means N concurrent workers
++  on the same kind can claim N distinct rows in one round. Each row is
++  processed exactly once across the cluster.
++* Workers hold a row's lock only while their handler runs. The lock is
++  released atomically by `MarkJobCompleted` / `RescheduleJob` /
++  `MarkJobFailed`.
++* Job handlers MUST be idempotent. A worker process killed mid-handler
++  leaves the row locked until the 5-minute stale-lock window passes,
++  at which point another worker may pick it up and re-run. Guard the
++  side-effects (e.g. `processed_at IS NULL` checks).
++
++## Metrics
++
++All exported through the standard `/metrics` registry:
++
++* `shithub_worker_jobs_processed_total{kind,outcome}` — outcome ∈
++  `ok | retry | failed | poison`.
++* `shithub_worker_job_duration_seconds{kind}` — handler latency
++  histogram.
++* `shithub_worker_in_flight{kind}` — gauge of currently-running handlers.
++
++## Operational notes
++
++* Default pool size is 4. Override via `--workers <n>` on the CLI or
++  `SHITHUB_WORKERS=<n>` in the environment.
++* Graceful shutdown: SIGINT/SIGTERM cancels the root context. Workers
++  stop pulling new jobs and let in-flight handlers finish (bounded by
++  `JobTimeout`, default 5 minutes). The LISTEN goroutine drops its
++  conn cleanly.
++* The pool size on the Postgres connection is sized to `Workers + 2`
++  (one for LISTEN, one slack for enqueues during shutdown).
++* `LISTEN/NOTIFY` semantics: when the hook calls `pg_notify` inside a
++  transaction, the notification is *only* delivered after commit. So
++  failed inserts never wake workers for nothing.