`ea09767`

docs/actions: document runner pool operations (S41j-4)

Authored by mfwolffe <wolffemf@dukes.jmu.edu> 22 hours ago

SHA: ea09767d0868087944652edb121e94bb528e7508
Parents: 3b6ce7e
Tree: 9bea261

4 changed files

Status	File	+	-
M	`docs/internal/actions-runner-api.md`	19	1
M	`docs/internal/actions-schema.md`	8	1
M	`docs/internal/runbooks/actions-runner.md`	82	2
M	`docs/public/api/actions-runner.md`	5	0

docs/internal/actions-runner-api.mdmodified

  the runner token before it expires, because the runner uses that same token for
  heartbeat authentication.
 +Operators can drain, undrain, rotate, and hard-revoke runners with
 +`shithubd admin runner drain`, `undrain`, `rotate-token`, and `revoke`.
 +Drained runners keep heartbeating and may finish already claimed jobs,
 +but heartbeat claims return 204 until the runner is undrained. Hard
 +revocation sets the runner offline, records `revoked_at`, revokes all
 +registration tokens, and makes job API JWTs minted for that runner
 +invalid. This is the token-compromise boundary: a host with an old config
 +file cannot claim new jobs or update already claimed jobs after
 +revocation lands in Postgres.
++
  `POST /api/v1/runners/heartbeat` accepts:
  ```http
  Request body:
  ```json
 -{"labels":["self-hosted","linux","ubuntu-latest","x64"],"capacity":1}
 +{
 +  "labels": ["self-hosted", "linux", "ubuntu-latest", "x64"],
 +  "capacity": 1,
 +  "host_name": "runner-host-1",
 +  "version": "v0.1.0"
 +}
  ```
  Returns 204 when no matching job is claimable. Returns 200 with
  per-owner/org concurrent job caps are not dispatchable. Approval simply
  sets `workflow_runs.approved_by_user_id`; the next heartbeat can claim the
  same queued jobs, so no duplicate run is created.
 +`host_name` and `version` are optional runner metadata. The server stores
 +trimmed values up to 255 bytes for pool diagnostics and preserves the
 +previous values when old runners omit them.
  The job payload includes `checkout_url`, `checkout_token`, resolved
  `secrets`, and `mask_values`; repo secrets shadow org secrets with the
  same name. The server also stores an encrypted claim-time copy of the mask

docs/internal/actions-schema.mdmodified

  ## SQL schema
 -Actions migrations currently span 0042–0051, 0053, 0057, 0060, and 0064–0066.
 +Actions migrations currently span 0042–0051, 0053, 0057, 0060, and 0064–0067.
  Migration 0052 belongs to the repo source-remotes feature, 0054
  belongs to push event protocol tracking, 0055 belongs to the social
  feed, 0056 belongs to user profile contribution settings, 0058 belongs
  | 0057  | `workflow_job_secret_masks` | Encrypted claim-time log mask snapshots per job               |
  | 0060  | Actions retention indexes   | Narrow cleanup indexes for terminal steps/runs                |
  | 0066  | `actions_*_policies`, `workflow_run_approvals` | Enablement, runner-pool caps, and approval decisions |
 +| 0067  | `workflow_runners` ops state | Host/version metadata, drain state, and hard revocation state |
  A few load-bearing choices, called out so they're easy to spot in a
  later schema diff:
  - S41c: `shithubd admin runner register --name <foo>` issues a
    registration token + writes a row to `workflow_runners`.
 +- S41j: `shithubd admin runner drain|undrain|rotate-token|revoke|cleanup-stale`
 +  gives operators pool controls. Drained runners keep heartbeating and
 +  may finish already claimed jobs but receive no new claims. Revoked
 +  runners are set offline, all registration tokens are revoked, and job
 +  API JWTs from that runner are rejected even if the runner still has an
 +  old config file.
  - S41g: `POST /api/v1/jobs/{id}/cancel` and the repository run-detail
    UI request cancellation. Running jobs flip `cancel_requested`; queued
    jobs are made terminal immediately.

docs/internal/runbooks/actions-runner.mdmodified

    --dns-servers 172.30.0.1
  ```
 +## Pool Operations
++
 +List pool state:
++
 +```sh
 +shithubd admin runner list --output json
 +shithubd admin runner queue --output json
 +```
++
 +`list` includes labels, capacity, active job count, last heartbeat,
 +host name, runner version, drain state, and revoke state. `queue`
 +groups queued jobs by requested `runs-on` label so unsupported labels
 +are visible without querying Postgres.
++
 +Drain a runner before host maintenance:
++
 +```sh
 +shithubd admin runner drain --id 7 --reason 'kernel update'
 +```
++
 +The runner keeps heartbeating and can finish already claimed jobs, but
 +new heartbeat claims return 204 until it is undrained:
++
 +```sh
 +shithubd admin runner undrain --id 7
 +```
++
 +Rotate a registration token after a config-management change:
++
 +```sh
 +shithubd admin runner rotate-token --id 7 --expires-in 24h --output json
 +```
++
 +Update the runner host config with the printed token, restart
 +`shithubd-runner`, then verify heartbeats with `runner list`.
++
 +Mark stale runners offline:
++
 +```sh
 +shithubd admin runner cleanup-stale --older-than 2m
 +```
++
 +Hard-revoke a compromised runner:
++
 +```sh
 +shithubd admin runner revoke --id 7 --reason 'host compromise'
 +```
++
 +Revocation records `revoked_at`, marks the runner offline, revokes every
 +registration token for that runner, and causes existing job API JWTs from
 +that runner to fail. Use drain for routine maintenance; use revoke when
 +the token or host may be compromised.
++
 +Destroy/recreate with DigitalOcean:
++
 +```sh
 +doctl compute droplet list --tag-name shithub-actions-runner
 +doctl compute droplet delete <droplet-id>
 +```
++
 +After recreation, register a fresh runner token and let the provisioning
 +role write the new config. Do not reuse a token from a revoked runner.
++
 +Emergency controls:
++
 +- Stop all new claims: disable Actions at site level or drain every
 +  runner with `shithubd admin runner list --output json` followed by
 +  `shithubd admin runner drain --id <id>`.
 +- Pause one repo/org: set the repo/org Actions policy to disabled so the
 +  claim query leaves its queued jobs untouched.
 +- Cancel active work for a repo: use the repo Actions UI or job cancel
 +  API for each queued/running job.
 +- Revoke all runner tokens: iterate `shithubd admin runner revoke --id
 +  <id>` over every non-revoked runner.
 +- Fence the pool: block or destroy droplets tagged
 +  `shithub-actions-runner` in DigitalOcean, then rotate or revoke the
 +  affected runner tokens before allowing replacement hosts to connect.
++
  Equivalent config file:
  ```toml
  curl -fsS "$BASE/api/v1/runners/heartbeat" \
    -H "Authorization: Bearer $RUNNER_TOKEN" \
    -H "Content-Type: application/json" \
 -  -d '{"labels":["self-hosted","linux","ubuntu-latest","x64"],"capacity":1}' \
 +  -d '{"labels":["self-hosted","linux","ubuntu-latest","x64"],"capacity":1,"host_name":"curl-smoke","version":"manual"}' \
    | tee /tmp/shithub-claim.json
  ```
    jobs are terminal.
  - The PR Checks tab shows the matching check run as success.
  - `/metrics` includes runner registration, heartbeat, JWT, job
 -  cancellation, log-scrub, step-timeout, and retention counters.
 +  cancellation, log-scrub, step-timeout, retention, queue-depth by label,
 +  claim-latency, runner-online, runner-stale, runner-draining, and
 +  runner-revocation metrics.
  ## Retention Sweep

docs/public/api/actions-runner.mdmodified

  capacity match. Returns `204 No Content` when no job is available, or
  `200 OK` with a job payload and job JWT.
 +The runner may include `host_name` and `version` in the heartbeat body
 +for operator diagnostics. Drained runners return `204 No Content` and do
 +not claim new jobs. Revoked runners receive `401 Unauthorized`, and job
 +JWTs previously minted for that runner are rejected.
++
  ## Job log chunks
  ```