@@ -0,0 +1,163 @@ |
| | 1 | +# Day-one operator runbook |
| | 2 | + |
| | 3 | +This is what the on-call (the project author for v1) reads on |
| | 4 | +launch day and the day after. Pair with the cutover checklist |
| | 5 | +(`deploy/cutover/checklist.md`) for the actual cutover steps. |
| | 6 | + |
| | 7 | +## "It's been an hour" |
| | 8 | + |
| | 9 | +Open three tabs and walk this list: |
| | 10 | + |
| | 11 | +1. **Grafana → shithubd-overview dashboard.** |
| | 12 | + - `Web up` and `Worker up` should both be 1. |
| | 13 | + - p95 latency on top routes should sit near baseline (S36 |
| | 14 | + numbers, doubled is the soft ceiling — see |
| | 15 | + `docs/internal/capacity.md`). |
| | 16 | + - DB calls/sec should NOT be 10× baseline. If it is, an N+1 |
| | 17 | + regression slipped through; check `journalctl -u shithubd-web |
| | 18 | + --grep 'high-query-count'` (the QueryCounter middleware |
| | 19 | + emits warning lines). |
| | 20 | + |
| | 21 | +2. **Loki → `{service="shithubd"} |~ "panic|ERROR"`.** |
| | 22 | + - Zero panics is the bar. |
| | 23 | + - One or two ERROR lines from genuinely broken third-party |
| | 24 | + subscribers is normal (we auto-disable after 50; this is |
| | 25 | + fine). |
| | 26 | + - A flood of ERROR lines from the same handler is a bug — |
| | 27 | + bisect. |
| | 28 | + |
| | 29 | +3. **Status page on docs.shithub.example/status.html.** |
| | 30 | + - Confirm "All systems normal." with a current timestamp. |
| | 31 | + - If you've had any blip, even a 30-second one, log it under |
| | 32 | + "Recent incidents" with what happened. Trust comes from |
| | 33 | + transparency, not from the page being empty. |
| | 34 | + |
| | 35 | +If all three look clean, take a breath. Then post the announcement |
| | 36 | +if you haven't. |
| | 37 | + |
| | 38 | +## "It's been a day" |
| | 39 | + |
| | 40 | +24h post-cutover checklist: |
| | 41 | + |
| | 42 | +1. **Backups.** The first daily logical backup should have run. |
| | 43 | + ```sh |
| | 44 | + ssh db |
| | 45 | + sudo -u postgres rclone --config /root/.config/rclone/rclone.conf \ |
| | 46 | + lsf spaces-prod:shithub-backups/daily/$(date -u +%Y/%m/%d)/ |
| | 47 | + ``` |
| | 48 | + Should list one `.dump` file. If empty, see |
| | 49 | + `docs/internal/runbooks/backups.md#missed-backup`. |
| | 50 | + |
| | 51 | +2. **WAL archive.** `pg_stat_archiver.failed_count` should be 0 |
| | 52 | + and `last_archived_time` should be within minutes of now. |
| | 53 | + ```sh |
| | 54 | + ssh db |
| | 55 | + sudo -u postgres psql -d shithub -c "SELECT * FROM pg_stat_archiver;" |
| | 56 | + ``` |
| | 57 | + |
| | 58 | +3. **Mirror push to GitHub.** The mirror job runs hourly. Confirm: |
| | 59 | + ```sh |
| | 60 | + git ls-remote https://github.com/tenseleyFlow/shithub.git trunk |
| | 61 | + git ls-remote https://shithub.example/shithub/shithub.git trunk |
| | 62 | + ``` |
| | 63 | + Both should report the same SHA (within an hour of the latest |
| | 64 | + push to shithub). |
| | 65 | + |
| | 66 | +4. **Signups + abuse.** Check `/admin/users` for the past 24h. |
| | 67 | + - Anything obviously bot-shaped (timing, email pattern, |
| | 68 | + username pattern)? The per-/24 throttle should have caught |
| | 69 | + bursts; a long tail of single-account spam needs manual |
| | 70 | + review. |
| | 71 | + - Suspended accounts: log who and why for the retro. |
| | 72 | + |
| | 73 | +5. **Issue tracker.** Open |
| | 74 | + `https://shithub.example/shithub/shithub/issues`. Triage every |
| | 75 | + new issue: |
| | 76 | + - **P0** — site down / data loss / security. Fix immediately. |
| | 77 | + - **P1** — broken core flow (signup, push, merge). Fix this |
| | 78 | + week. |
| | 79 | + - **P2** — minor bug / UX. Fix in next sprint. |
| | 80 | + - **P3** — feature ask / WONTFIX / accept-with-rationale. |
| | 81 | + - Reply on every issue within 24h, even if just "tracked, P2." |
| | 82 | + |
| | 83 | +6. **Retro.** Update |
| | 84 | + `docs/internal/retro/v1.0.0.md` — fill the "Numbers" table, |
| | 85 | + the "What surprised us" section, the top-3 issues table. |
| | 86 | + |
| | 87 | +## "First incident?" |
| | 88 | + |
| | 89 | +The general procedure is in |
| | 90 | +`docs/internal/runbooks/incidents.md`. Day-one specifics: |
| | 91 | + |
| | 92 | +1. **Status page first.** Within 5 min of detecting an incident, |
| | 93 | + update `docs/public/status.md` to "Investigating: <one line>". |
| | 94 | + Push, sync to docs bucket. The fact that you're aware is |
| | 95 | + half the battle. |
| | 96 | +2. **Identify the right runbook.** Most v1 incidents will be one |
| | 97 | + of: |
| | 98 | + - `incidents.md#shithubd-down` — process crashed. |
| | 99 | + - `incidents.md#postgres-down` — DB unreachable. |
| | 100 | + - `incidents.md#archive-failing` — WAL archiver broken. |
| | 101 | + - `incidents.md#job-backlog` — worker queue runaway. |
| | 102 | + - `incidents.md#webhook-delivery-failing` — outbound delivery |
| | 103 | + issues. |
| | 104 | +3. **Mitigate before fix.** If it's the announcement spike |
| | 105 | + tipping the rate-limit floor, you can briefly raise the floor |
| | 106 | + (config edit + restart) rather than untangling the abuse |
| | 107 | + signal. The runbook + the rate-limit-tuning notes are both at |
| | 108 | + hand. |
| | 109 | +4. **Status page on resolution.** "Resolved at HH:MM UTC. <One |
| | 110 | + sentence on cause.> <One sentence on prevention if relevant.>" |
| | 111 | + |
| | 112 | +## Communication posture |
| | 113 | + |
| | 114 | +- **Honest > fast.** If you don't know the cause, "investigating" |
| | 115 | + is a fine status. Speculation in public is a debt you pay |
| | 116 | + later. |
| | 117 | +- **No timelines under stress.** "We'll have an update by 18:00 |
| | 118 | + UTC" is fine. "We'll have a fix by 18:00 UTC" is a trap. |
| | 119 | +- **Engage the reporter.** When a user files a serious bug, |
| | 120 | + thank them on the issue. Even if the fix is days out, the |
| | 121 | + acknowledgement is the part they remember. |
| | 122 | + |
| | 123 | +## Rate-limit tuning |
| | 124 | + |
| | 125 | +If legitimate signups are getting throttled (CIDR-shared |
| | 126 | +networks during the announcement spike), the steps are: |
| | 127 | + |
| | 128 | +```sh |
| | 129 | +# 1. Edit the production inventory's rate-limit vars. |
| | 130 | +$EDITOR deploy/ansible/inventory/production |
| | 131 | +# signup_per_ip_per_hour: 5 → 10 |
| | 132 | +# signup_per_24_per_hour: 20 → 50 |
| | 133 | + |
| | 134 | +# 2. Apply only the app role (no DB / edge churn). |
| | 135 | +make deploy ANSIBLE_INVENTORY=production ANSIBLE_TAGS=app |
| | 136 | +``` |
| | 137 | + |
| | 138 | +The web service restart picks up new env. Document the change in |
| | 139 | +the retro; tighten back to defaults once the spike passes. |
| | 140 | + |
| | 141 | +## "Drop the GitHub mirror?" decision |
| | 142 | + |
| | 143 | +Default plan: **keep the mirror for 90 days post-launch**, then |
| | 144 | +re-evaluate. The mirror exists as a recovery surface; if the |
| | 145 | +self-hosted instance has had no extended outage in 90 days, |
| | 146 | +drop the mirror push job and mark the GitHub repo as archived. |
| | 147 | + |
| | 148 | +The decision and date land in the next retro after the 90-day |
| | 149 | +mark. |
| | 150 | + |
| | 151 | +## When to escalate (to whom?) |
| | 152 | + |
| | 153 | +Solo on-call for v1. There is no second-tier. Escalation paths: |
| | 154 | + |
| | 155 | +- **DB destroyed beyond restore.** Restore from cross-region DR |
| | 156 | + bucket per `runbooks/restore.md`. |
| | 157 | +- **Account compromise on the operator's email.** Revoke the |
| | 158 | + compromised email's session epoch via SQL; trigger a |
| | 159 | + "rotate-secrets" pass. |
| | 160 | +- **Legal request.** Acknowledge receipt; do not respond to the |
| | 161 | + substance same-day. |
| | 162 | +- **Mass-abuse incident.** Read-only mode, then triage. The |
| | 163 | + read-only-mode runbook is the lever. |
