Worker

The worker pool drains the Postgres-backed job queue introduced in S14. One binary serves the web API, the SSH dispatcher, the hooks, and the worker — shithubd worker boots a long-running process whose only job is to dequeue and run.

Architecture

post-receive  ──INSERT push_event + INSERT job + NOTIFY──▶  Postgres jobs
                                                                  │
                                                                  ▼
                                                         shithubd worker
                                                  ┌──────────────────────┐
                                                  │ N goroutines          │
                                                  │ ClaimJob (FOR UPDATE  │
                                                  │   SKIP LOCKED)        │
                                                  │ → handler             │
                                                  │ → MarkCompleted /     │
                                                  │   Reschedule /        │
                                                  │   MarkFailed          │
                                                  └──────────────────────┘

The pool also runs a dedicated LISTEN goroutine that holds one Postgres connection on LISTEN shithub_jobs. A NOTIFY from any process (typically the post-receive hook) wakes idle workers in <1s without polling. The backstop poll (every 5s by default) covers dropped notifications.

Job kinds shipped in S14

Adding a new kind: write the handler in internal/worker/jobs/<kind>.go, add the Kind constant to internal/worker/types.go, register it in cmd/shithubd/worker.go. The dispatch loop is generic — no other code needs to change.

Failure handling

• Transient errors (handler returns a non-nil non-poison error): reschedule with Backoff(attempts) ± 20% jitter, where Backoff is 30s * 2^(attempts-1) capped at 1h.
• Poison errors (handler returns an error wrapping worker.ErrPoison): jump to MarkJobFailed immediately. Use this for malformed payloads or references to vanished rows.
• Panics: caught by safeRun and treated as a transient error so a buggy handler can't take down a worker goroutine.
• Stuck locks: ClaimJob ignores rows whose locked_at is older than 5 minutes — a worker that died mid-job releases its rows on the next claim cycle.
• Max attempts: jobs.max_attempts (default 5). Past the limit the row is moved to failed_at. Failed rows surface in the S34 admin panel for poison-job triage.

Concurrency contract

• FOR UPDATE SKIP LOCKED on the inner SELECT means N concurrent workers on the same kind can claim N distinct rows in one round. Each row is processed exactly once across the cluster.
• Workers hold a row's lock only while their handler runs. The lock is released atomically by MarkJobCompleted / RescheduleJob / MarkJobFailed.
• Job handlers MUST be idempotent. A worker process killed mid-handler leaves the row locked until the 5-minute stale-lock window passes, at which point another worker may pick it up and re-run. Guard the side-effects (e.g. processed_at IS NULL checks).

Metrics

All exported through the standard /metrics registry:

• shithub_worker_jobs_processed_total{kind,outcome} — outcome ∈ ok | retry | failed | poison.
• shithub_worker_job_duration_seconds{kind} — handler latency histogram.
• shithub_worker_in_flight{kind} — gauge of currently-running handlers.

Operational notes

• Default pool size is 4. Override via --workers <n> on the CLI or SHITHUB_WORKERS=<n> in the environment.
• Graceful shutdown: SIGINT/SIGTERM cancels the root context. Workers stop pulling new jobs and let in-flight handlers finish (bounded by JobTimeout, default 5 minutes). The LISTEN goroutine drops its conn cleanly.
• The pool size on the Postgres connection is sized to Workers + 2 (one for LISTEN, one slack for enqueues during shutdown).
• LISTEN/NOTIFY semantics: when the hook calls pg_notify inside a transaction, the notification is only delivered after commit. So failed inserts never wake workers for nothing.