Rotate secrets

Quarterly cadence; sooner if compromise is suspected. The secret classes:

Session signing key

The session key signs the cookie that authenticates a logged-in session. Rotating it logs every user out because every existing cookie's MAC stops verifying.

1. Generate a new key:

   openssl rand -base64 32
   

2. Update the inventory variable session_key. Keep the old key in a comment for one rotation cycle so you can revert.
3. make deploy ANSIBLE_INVENTORY=production ANSIBLE_TAGS=app.
4. Verify: sign in to your own account with a fresh browser; the cookie set after sign-in is signed by the new key.

User-visible impact: every user is signed out. Notify in-band before doing this if avoidable; do it without notice if the old key may be compromised.

TOTP AEAD key

The TOTP AEAD key encrypts every user's TOTP shared secret at rest in the database. Rotating this key requires a re-encryption migration — without it, every 2FA enrollment becomes unreadable.

The procedure is:

1. Add the new key to web.env as auth.totp_key_b64_next alongside the existing auth.totp_key_b64.
2. Restart web (the package supports a "current + next" pair: it reads with current, falls back to next, writes with current).
3. Run the re-encryption job: shithubd admin re-encrypt-totp --to-key=auth.totp_key_b64_next (operator-only). This decrypts each row with the old key and re-encrypts with the new.
4. Promote auth.totp_key_b64_next to auth.totp_key_b64 (drop the suffix), remove the old key.
5. Restart web.

Do not skip step 3. Failing to re-encrypt before retiring the old key locks every 2FA-enabled user out of their account; recovery codes are the only path back in, and not everyone has them saved.

DB password

Rotate by adding a new password and removing the old, without downtime.

1. As postgres:

   ALTER ROLE shithub WITH PASSWORD '<new>';
   

2. Update web.env and worker.env db_password.
3. make deploy ANSIBLE_INVENTORY=production ANSIBLE_TAGS=app. The web/worker units will restart and reconnect with the new password.

If you suspect the old password was leaked, do steps 1–3 in sequence within minutes — between (1) and (3) the running web process still has its open connections (which authenticated under the old password) but new connections will use the new.

Object store credentials

1. In the Spaces dashboard, generate a new access key with the same scope as the old.
2. Update inventory s3_access_key_id and s3_secret_access_key.
3. make deploy ANSIBLE_INVENTORY=production ANSIBLE_TAGS=app.
4. Verify: trigger a webhook delivery (which writes a body snapshot) and confirm it lands in the bucket.
5. Once confirmed, revoke the old key in the Spaces dashboard.

Do not revoke the old key first; the running process will lose access mid-flight.

Webhook AEAD key

The webhook secret AEAD key encrypts every webhook's secret at rest. Rotation is two-step like TOTP:

1. Add webhook.aead_key_next alongside webhook.aead_key.
2. Run shithubd admin re-encrypt-webhooks --to-key=webhook. aead_key_next.
3. Promote and restart.

Failing to re-encrypt before retiring the old key disables every webhook (the auto-disable logic kicks in on first decrypt failure).

Operator SSH keys

Standard procedure: add the new key to every host's ~operator/.ssh/authorized_keys, log in with the new key to confirm, remove the old. Ansible's authorized_key module makes this idempotent; the base role will pick up changes if the inventory's operator_ssh_keys list is the source of truth.

Audit

Every rotation is logged in the host's journal (the deploy run's output) and, for DB rotations, in pg_stat_activity history if your retention allows. There's no centralized rotation log; if you want one, capture each rotation in your team's incident channel with date + class + reason.