Read-only mode
shithub does not have a one-flag "read-only" toggle. When you need one — disaster mid-recovery, surprise database failover, controlled maintenance — these are the levers:
The fastest "stop writes" lever
# At the edge: serve every write method as 503 with a banner.
ssh edge
sudo caddy reload --config /etc/caddy/Caddyfile.read-only
The repo ships a Caddyfile.read-only snippet that responds 503
to POST/PUT/PATCH/DELETE/OPTIONS while letting GET
and HEAD through. git-receive-pack is also blocked; clones
still work.
Reverse with caddy reload --config /etc/caddy/Caddyfile.
User experience:
- Reads work normally.
- Writes return a 503 with an HTML page explaining "shithub is in read-only maintenance — try again later."
- Pushes fail with a git-side error.
This is the easiest reversal; nothing in the app changes.
Stop the worker
If you're worried about background writes (job processing mutating state), stop the worker too — see drain-workers.md.
Set the DB to read-only
Heavy hammer. ALTER SYSTEM SET default_transaction_read_only = on; SELECT pg_reload_conf(); makes every new connection
read-only. The web service will blow up on its first write
attempt — usually ERROR: cannot execute INSERT in a read-only transaction.
This is only useful if you suspect a bug in the read-only mode above is letting writes leak through. Otherwise skip it.
To revert: ALTER SYSTEM SET default_transaction_read_only = off; SELECT pg_reload_conf();.
During a real DB failover
If you've failed over to a standby that's read-write and need the app to talk to it:
- Update the
db.urlinventory variable to point at the new primary. ANSIBLE_TAGS=app make deploy ANSIBLE_INVENTORY=production— web/worker pick up the new connection string.
There's no replication tooling shipped today; if you have a standby, you set it up out of band.
What read-only mode does NOT do
- Stop the auth-audit log from writing — login attempts still hit the DB.
- Block the
shithubd hookinvocations from writingpush_events(because the AKC + git transport still let the TCP connection through). To stop that, also stop the web service or block port 22. - Block sign-ups — those are POSTs and are blocked at the edge, but the in-app banner under read-only mode is the only signal to a logged-in user.
Communicating it
In-app banner on every page during read-only mode is the right
default. Today this is via a feature flag (web.banner.text
config key); set it before flipping the Caddyfile so users see
the banner on the last successful render.
View source
| 1 | # Read-only mode |
| 2 | |
| 3 | shithub does not have a one-flag "read-only" toggle. When you |
| 4 | need one — disaster mid-recovery, surprise database failover, |
| 5 | controlled maintenance — these are the levers: |
| 6 | |
| 7 | ## The fastest "stop writes" lever |
| 8 | |
| 9 | ```sh |
| 10 | # At the edge: serve every write method as 503 with a banner. |
| 11 | ssh edge |
| 12 | sudo caddy reload --config /etc/caddy/Caddyfile.read-only |
| 13 | ``` |
| 14 | |
| 15 | The repo ships a `Caddyfile.read-only` snippet that responds 503 |
| 16 | to `POST`/`PUT`/`PATCH`/`DELETE`/`OPTIONS` while letting `GET` |
| 17 | and `HEAD` through. `git-receive-pack` is also blocked; clones |
| 18 | still work. |
| 19 | |
| 20 | Reverse with `caddy reload --config /etc/caddy/Caddyfile`. |
| 21 | |
| 22 | User experience: |
| 23 | |
| 24 | - Reads work normally. |
| 25 | - Writes return a 503 with an HTML page explaining "shithub is |
| 26 | in read-only maintenance — try again later." |
| 27 | - Pushes fail with a git-side error. |
| 28 | |
| 29 | This is the easiest reversal; nothing in the app changes. |
| 30 | |
| 31 | ## Stop the worker |
| 32 | |
| 33 | If you're worried about *background* writes (job processing |
| 34 | mutating state), stop the worker too — see |
| 35 | [drain-workers.md](./drain-workers.md). |
| 36 | |
| 37 | ## Set the DB to read-only |
| 38 | |
| 39 | Heavy hammer. `ALTER SYSTEM SET default_transaction_read_only = |
| 40 | on; SELECT pg_reload_conf();` makes every new connection |
| 41 | read-only. The web service will blow up on its first write |
| 42 | attempt — usually `ERROR: cannot execute INSERT in a read-only |
| 43 | transaction`. |
| 44 | |
| 45 | This is only useful if you suspect a bug in the read-only mode |
| 46 | above is letting writes leak through. Otherwise skip it. |
| 47 | |
| 48 | To revert: `ALTER SYSTEM SET default_transaction_read_only = off; |
| 49 | SELECT pg_reload_conf();`. |
| 50 | |
| 51 | ## During a real DB failover |
| 52 | |
| 53 | If you've failed over to a standby that's read-write and need |
| 54 | the app to talk to it: |
| 55 | |
| 56 | 1. Update the `db.url` inventory variable to point at the new |
| 57 | primary. |
| 58 | 2. `ANSIBLE_TAGS=app make deploy ANSIBLE_INVENTORY=production` — |
| 59 | web/worker pick up the new connection string. |
| 60 | |
| 61 | There's no replication tooling shipped today; if you have a |
| 62 | standby, you set it up out of band. |
| 63 | |
| 64 | ## What read-only mode does NOT do |
| 65 | |
| 66 | - Stop the auth-audit log from writing — login attempts still |
| 67 | hit the DB. |
| 68 | - Block the `shithubd hook` invocations from writing |
| 69 | `push_events` (because the AKC + git transport still let the |
| 70 | TCP connection through). To stop that, also stop the web |
| 71 | service or block port 22. |
| 72 | - Block sign-ups — those are POSTs and are blocked at the edge, |
| 73 | but the in-app banner under read-only mode is the only signal |
| 74 | to a logged-in user. |
| 75 | |
| 76 | ## Communicating it |
| 77 | |
| 78 | In-app banner on every page during read-only mode is the right |
| 79 | default. Today this is via a feature flag (`web.banner.text` |
| 80 | config key); set it before flipping the Caddyfile so users see |
| 81 | the banner on the last successful render. |