Backing up the production .env

/etc/shithub/web.env (and /etc/shithub/worker.env, when split) holds every load-bearing secret on the droplet:

• DB password (SHITHUB_DATABASE_URL)
• Session signing key, TOTP AEAD key, webhook AEAD key
• Spaces access keys (object store)
• Postmark / SMTP creds
• SSH-git host key seed (when configured)

The DB content is captured by backups.md — those dumps and WAL segments rebuild the data. web.env is what re-binds a fresh droplet to that data: without the same secrets, the existing DB rows can't be decrypted (TOTP/webhook payloads), sessions can't be signed, and S3 buckets can't be reached.

This doc covers backing up web.env itself.

The mental model

Treat web.env like a master password. The DB backup chain is fully automated and tested; the env file is operator-managed because it has different lifecycle:

• Mostly stable: changes only when secrets rotate (see rotate-secrets.md) or new config keys land.
• Tiny: a few KB.
• Maximum sensitivity: leaking it is a "rotate everything" incident. So it shouldn't ride alongside the DB dumps in Spaces — different blast radius.

That puts it in a different storage tier than DB dumps. Pick one of the three options below.

Option A (recommended) — password manager

Keep an encrypted copy in your operator password manager (1Password, Bitwarden, KeePassXC, …) as a "Secure Note" or file attachment.

When to update:

• After every secret rotation per rotate-secrets.md
• After any change to web.env for new config keys
• Right after a fresh droplet provision (initial baseline)

How:

ssh root@shithub.sh 'cat /etc/shithub/web.env'
# Paste output into a Secure Note titled e.g. "shithub-prod web.env (YYYY-MM-DD)"
# Tag with the rotation date so you can identify the active copy.

Pros: zero new infrastructure, auditable access (PM logs), already in your daily-use security tooling. Cons: manual step that's easy to skip after a rotation — set a calendar reminder for the quarterly rotation cadence.

Option B — encrypted blob alongside DB backups

Include an encrypted copy of web.env in the daily backup chain. Encrypt with age or gpg so even a compromised Spaces key can't decrypt it; the recipient key is held only by operators (in their PM).

Sketch (NOT yet wired up — would be a follow-up if we go this route):

# In shithub-backup-daily, after the pg_dump succeeds:
age -r "$(cat /etc/shithub/env-backup.recipients)" \
    -o "$LOCAL_DIR/web.env.${STAMP}.age" \
    /etc/shithub/web.env
rclone copyto "$LOCAL_DIR/web.env.${STAMP}.age" \
       "$BUCKET/env/$(date -u +%Y/%m/%d)/web.env.${STAMP}.age"

Pros: automatic, captures rotations as they happen. Cons: adds a dep (age), a new key-mgmt surface (the recipient key), and a failure mode (if the recipient key is lost, the backups become useless).

Option C — DO snapshot

DO droplet snapshots include /etc/shithub/web.env by virtue of including the whole filesystem. This is "free" coverage but the caveats are real:

• Point-in-time only: a snapshot taken before a secret rotation has the OLD secret. Restoring a stale snapshot desynchronizes you from any DB rows encrypted with the new key (TOTP, webhook payloads).
• Snapshots are scheduled to be deleted under DO's free policy (4 retained, oldest pruned).
• Snapshot restore replaces the droplet, including the block volume's previous state (depending on snapshot type).

Suitable as a belt-and-suspenders on top of A or B. Not sufficient as the only backup of the env file.

Restore procedure

If the live web.env is lost (droplet replaced, file deleted, permissions wedged):

1. Stop the running services that need it:

   systemctl stop shithubd-web shithubd-cron
   

2. Recreate /etc/shithub/web.env with the right ownership and mode (see deploy/ansible/roles/shithubd/tasks/main.yml for canonical perms — currently root:shithub 0640):

   install -o root -g shithub -m 0640 /dev/stdin /etc/shithub/web.env <<'EOF'
   <paste from your Secure Note>
   EOF
   

3. Restart:

   systemctl start shithubd-web shithubd-cron
   curl -fsS http://127.0.0.1:8080/healthz   # 200
   

4. If the secrets in the restored copy are stale (you rotated after the backup), follow rotate-secrets.md to re-apply the current values.

What goes wrong if you skip this

Concrete scenarios where lacking an env backup turns a recoverable incident into a re-key-the-world incident:

How this relates to the rest of the backup story