@@ -0,0 +1,188 @@ |
| | 1 | +# Architecture |
| | 2 | + |
| | 3 | +A bird's-eye view of how shithub fits together. For depth on any |
| | 4 | +one subsystem, see the linked per-area doc. |
| | 5 | + |
| | 6 | +## Process model |
| | 7 | + |
| | 8 | +One Go binary, three roles: |
| | 9 | + |
| | 10 | +- `shithubd web` — the HTTP server. Renders the UI, serves |
| | 11 | + `/api/v1/`, handles git over HTTPS. |
| | 12 | +- `shithubd worker` — pulls from the Postgres-backed job queue. |
| | 13 | + Owns webhook delivery, notification fan-out, async fan-out, |
| | 14 | + search reindex. |
| | 15 | +- `shithubd cron` — periodic chores (lifecycle sweeps, queue |
| | 16 | + janitor, webhook purge). Triggered by a systemd timer; one-shot. |
| | 17 | + |
| | 18 | +All three read the same config and the same DB. They differ only |
| | 19 | +in which `cmd/shithubd/<role>.go` they boot into. |
| | 20 | + |
| | 21 | +Plus a small admin/operator CLI surface (`shithubd admin …`, |
| | 22 | +`shithubd migrate …`, `shithubd config …`, `shithubd hook …`) |
| | 23 | +that runs as one-shot subprocesses. |
| | 24 | + |
| | 25 | +## Layered packages |
| | 26 | + |
| | 27 | +``` |
| | 28 | + +------------------------+ |
| | 29 | +HTTP request --> | internal/web/handlers | Per-area handlers (auth, repo, …) |
| | 30 | + +------------------------+ |
| | 31 | + | |
| | 32 | + +------------------------+ |
| | 33 | + | internal/web/render | html/template + helpers |
| | 34 | + | internal/web/middleware| auth, csrf, ratelimit, body cap |
| | 35 | + +------------------------+ |
| | 36 | + | |
| | 37 | + +------------------------+ |
| | 38 | + | internal/<domain> | repos, issues, pulls, orgs, … |
| | 39 | + +------------------------+ |
| | 40 | + | |
| | 41 | + +------------------------+ |
| | 42 | + | internal/auth/policy | the only place "can X do Y" lives |
| | 43 | + +------------------------+ |
| | 44 | + | |
| | 45 | + +------------------------+ |
| | 46 | + | internal/<domain>/sqlc | sqlc-generated DB code |
| | 47 | + +------------------------+ |
| | 48 | + | |
| | 49 | + Postgres |
| | 50 | +``` |
| | 51 | + |
| | 52 | +Cross-cutting: |
| | 53 | + |
| | 54 | +- `internal/markdown` — sole renderer for user-authored markdown. |
| | 55 | + Lint-enforced: nothing outside this package may import goldmark |
| | 56 | + or bluemonday. |
| | 57 | +- `internal/security/{ssrf,openredirect}` — outbound HTTP defense |
| | 58 | + + redirect validation. |
| | 59 | +- `internal/cache/lru` — generic LRU + singleflight. |
| | 60 | +- `internal/pagination/keyset` — cursor pagination with |
| | 61 | + HMAC-signed cursors. |
| | 62 | +- `internal/webhook` — outbound webhook delivery + signing. |
| | 63 | + |
| | 64 | +## Request lifecycle (web) |
| | 65 | + |
| | 66 | +1. Caddy terminates TLS, forwards to `127.0.0.1:8080`. |
| | 67 | +2. The chi router matches the route. Middleware stack on every |
| | 68 | + request: |
| | 69 | + - Request ID + access log |
| | 70 | + - `OptionalUser` (resolves session cookie or anon) |
| | 71 | + - CSRF (nosurf-derived) — enabled by default; `/api/v1/` and |
| | 72 | + git transports are explicit exemptions |
| | 73 | + - Rate limit (per-IP for anon, per-user for logged-in on hot |
| | 74 | + paths) |
| | 75 | +3. Auth-gated routes have `RequireUser` or `RequireSiteAdmin` |
| | 76 | + middleware in their group. |
| | 77 | +4. The handler resolves the domain object, calls `policy.Can(...)` |
| | 78 | + to authorize the action, executes, renders. |
| | 79 | +5. Templates render through `internal/web/render`. Markdown bodies |
| | 80 | + pass through `internal/markdown` for sanitization. |
| | 81 | + |
| | 82 | +## Request lifecycle (API) |
| | 83 | + |
| | 84 | +`/api/v1/*` is under a CSRF-exempt group with PAT auth: |
| | 85 | + |
| | 86 | +1. `MaxBodySize` cap (256 KiB). |
| | 87 | +2. `PATAuthMiddleware` — resolves token to user; 401 on missing, |
| | 88 | + 403 on insufficient scope. |
| | 89 | +3. Per-route `RequireScope` checks the specific scope. |
| | 90 | +4. Handler runs. |
| | 91 | + |
| | 92 | +## Data flow: a push |
| | 93 | + |
| | 94 | +``` |
| | 95 | + git client |
| | 96 | + | |
| | 97 | + v git-receive-pack POST |
| | 98 | + web service ──> shithubd hook pre-receive ──> Postgres (auth check) |
| | 99 | + | | |
| | 100 | + | <── decision (accept/reject) |
| | 101 | + v | |
| | 102 | + accept push, v |
| | 103 | + write to /data/repos INSERT push_events (shithub_hook role) |
| | 104 | + | |
| | 105 | + v |
| | 106 | + INSERT jobs (search reindex, notify, ...) |
| | 107 | + | |
| | 108 | + v |
| | 109 | + worker picks up job (FOR UPDATE SKIP LOCKED) |
| | 110 | + | |
| | 111 | + v |
| | 112 | + fan-out (webhooks, notifications, search) |
| | 113 | +``` |
| | 114 | + |
| | 115 | +The `shithub_hook` role has minimum-needed grants |
| | 116 | +(`docs/internal/db-roles.md`). The web role can do everything |
| | 117 | +the runtime needs but is intentionally separate from the migration |
| | 118 | +role used at deploy time. |
| | 119 | + |
| | 120 | +## Deployment topology |
| | 121 | + |
| | 122 | +``` |
| | 123 | + +----------------------------+ |
| | 124 | + public ---> | Caddy (TLS, rate limits) | :443 |
| | 125 | + +----------------------------+ |
| | 126 | + | 127.0.0.1:8080 |
| | 127 | + +----------------------------+ |
| | 128 | + | shithubd web (systemd) | |
| | 129 | + | shithubd worker (systemd) | |
| | 130 | + | shithubd cron (timer) | |
| | 131 | + +----------------------------+ |
| | 132 | + | |
| | 133 | + +-----------+ +-----------------+ |
| | 134 | + | Postgres | | Spaces (S3) | |
| | 135 | + +-----------+ | - WAL archive | |
| | 136 | + | - daily dumps | |
| | 137 | + | - LFS / blobs | |
| | 138 | + +-----------------+ |
| | 139 | + \ / |
| | 140 | + \ WireGuard mesh (10.50.0.0/24) |
| | 141 | + \________ ________/ |
| | 142 | + | |
| | 143 | + +----------------+ |
| | 144 | + | Monitoring | |
| | 145 | + | Prom/Loki/AM | |
| | 146 | + | Grafana | |
| | 147 | + +----------------+ |
| | 148 | +``` |
| | 149 | + |
| | 150 | +Monitoring is on the WireGuard mesh; metrics ports never face the |
| | 151 | +public internet. See [deploy.md](./deploy.md) for the full |
| | 152 | +operator guide. |
| | 153 | + |
| | 154 | +## Observability |
| | 155 | + |
| | 156 | +Three independent channels: |
| | 157 | + |
| | 158 | +- **Structured logs** (`internal/infra/log`) → stdout → journald |
| | 159 | + → promtail → Loki. |
| | 160 | +- **Metrics** (Prometheus) at `/metrics`, basic-auth gated in |
| | 161 | + prod. |
| | 162 | +- **Tracing** (OTel HTTP) optional, sample-rate controlled. |
| | 163 | +- **Error reporting** (Sentry-protocol DSN, GlitchTip-compatible). |
| | 164 | + |
| | 165 | +See [observability.md](./observability.md). |
| | 166 | + |
| | 167 | +## What's deliberately not here |
| | 168 | + |
| | 169 | +- **No microservices.** One binary per role; everything else is |
| | 170 | + premature. |
| | 171 | +- **No background ETL pipeline.** Work that doesn't fit a job-row |
| | 172 | + shape doesn't exist yet; if it does, add it as a new job kind |
| | 173 | + rather than a new daemon. |
| | 174 | +- **No message bus.** Postgres LISTEN/NOTIFY suffices for the |
| | 175 | + intra-process signaling we need. |
| | 176 | +- **No Redis.** The LRU + singleflight cache is in-process; if a |
| | 177 | + shared cache becomes necessary, the seam is `internal/cache` |
| | 178 | + and we'd swap in a Redis-backed implementation behind the same |
| | 179 | + interface. |
| | 180 | + |
| | 181 | +## Where to read next |
| | 182 | + |
| | 183 | +- New to the code: [code-tab.md](./code-tab.md) walks how a single |
| | 184 | + request flows end-to-end on the most-traveled page. |
| | 185 | +- New to ops: [deploy.md](./deploy.md), then |
| | 186 | + [runbooks/](./runbooks/). |
| | 187 | +- New to security: [threat-model.md](./threat-model.md) + |
| | 188 | + [security-checklist.md](./security-checklist.md). |
