Actions runbook

This is the operator runbook for shithub Actions. Host provisioning lives in runner-deploy.md, and runner protocol details live in actions-runner-api.md.

Shape

git push / workflow_dispatch / schedule / pull_request
        |
        v
workflow:trigger worker job
        |
        v
workflow_runs + workflow_jobs + workflow_steps + check_runs
        |
        v
registered runner heartbeat claims a matching queued job
        |
        v
actions/checkout@v4 -> containerized run: steps
        |
        v
log chunks -> step/job status -> run rollup

The v1 executor supports host-side actions/checkout@v4 plus containerized run: steps. The checkout token is short-lived, repository-scoped, tied to a running job, and accepted only for read-only smart-HTTP fetches. shithub/upload-artifact@v1 and shithub/download-artifact@v1 are still reserved aliases and fail until artifact transfer is wired end to end.

First smoke

1. Confirm migrations are applied and the web process can enqueue workers.
2. Register one runner with a label that matches the workflow:

shithubd admin runner register \
  --name smoke-runner-1 \
  --labels self-hosted,linux,ubuntu-latest,x64 \
  --capacity 1 \
  --output json

1. Start shithubd-runner with the returned token. For production hosts, use one token per host and store it in ansible-vault, host vars, or the deployment secret store. The role writes /etc/shithubd-runner/config.toml with restrictive permissions. Use --expires-in only when the automation rotates the runner token before that deadline; the current runner uses the registration token for every heartbeat.
2. Push a run:-only workflow:

name: smoke
on: [push, workflow_dispatch]
jobs:
  hello:
    runs-on: ubuntu-latest
    env:
      RUN_ID: ${{ shithub.run_id }}
    steps:
      - run: echo "hello from shithub actions"
      - run: test -n "$RUN_ID"

1. Expected result:

• workflow:trigger enqueues a workflow run.
• A runner heartbeat claims the queued job within one idle poll interval.
• The Actions run page streams step logs while the job is running.
• The matching check run completes with success.
• /{owner}/{repo}/actions.atom includes the completed run.

Repeat with exit 1; the check should complete with failure.

Checkout smoke

After the run-only smoke passes, verify repository checkout with:

name: checkout smoke
on: [push, workflow_dispatch]
jobs:
  checkout:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: test -f README.md
      - run: test "$(git rev-parse HEAD)" = "${{ shithub.sha }}"

Expected result:

• The claim response contains checkout_url and checkout_token.
• Git smart HTTP sees the checkout token as Basic auth and permits git-upload-pack for the claimed repo.
• git-receive-pack rejects the same credential with 403.
• The job workspace contains the exact shithub.sha commit before the first run: step starts.

Live log tail

Step log pages open an SSE stream at:

/{owner}/{repo}/actions/runs/{run}/jobs/{job}/steps/{step}/log/stream

The stream sends event: chunk records with the chunk sequence as the SSE id. Browsers reconnect with Last-Event-ID; the handler also accepts ?after=<seq> for the first connection from a rendered log page. A terminal step sends event: done and closes the stream.

In shithubd, this route is mounted outside the normal app compression and 30-second timeout middleware. If a future route move puts live logs back under either middleware, EventSource clients will churn and logs can buffer despite the Caddy flush setting.

Log chunks are never sent through Postgres NOTIFY. Runner log writes append to workflow_step_log_chunks, then NOTIFY step_log_<step_id> with only the sequence number. Step completion notifies done.

Rate limit

Live tails use internal/ratelimit scope actions:logtail with five concurrent streams per viewer. Authenticated viewers key by user id; anonymous public-repo viewers key by client IP. The limiter uses a short lease TTL so a dropped connection cannot hold a slot permanently.

Caddy

The production Caddy template has a dedicated Actions log-stream route with:

flush_interval -1

The same route is excluded from gzip compression. If logs arrive only after several kilobytes accumulate, verify the deployed /etc/caddy/Caddyfile contains that route and reload Caddy:

sudo caddy reload --config /etc/caddy/Caddyfile

Runner health

On the runner host:

systemctl status shithubd-runner
journalctl -u shithubd-runner -n 100 --no-pager

On the app host, inspect runner registration and heartbeat state:

shithubd admin actions runner list

Inspect queued jobs by requested runs-on label:

shithubd admin runner queue
shithubd admin runner queue --output json

Important metrics:

• shithub_actions_queue_depth{resource="runs|jobs"}
• shithub_actions_active{resource="runs|jobs"}
• shithub_actions_runner_heartbeat_age_seconds{runner,status}
• shithub_actions_runner_capacity{runner,status}
• shithub_actions_runner_heartbeats_total{result="claimed|no_job"}
• shithub_actions_runner_jwt_total{result="issued|rejected|replay"}
• shithub_actions_runs_completed_total{event,conclusion}
• shithub_actions_run_duration_seconds{event,conclusion}
• shithub_actions_steps_completed_total{step_type,conclusion}
• shithub_actions_jobs_cancelled_total{reason="user|concurrency|timeout"}
• shithub_actions_log_scrub_replacements_total{location="server"}
• shithub_actions_log_chunks_total{location="server"}
• shithub_actions_log_chunk_bytes_total{location="server"}
• shithub_actions_storage_objects{kind="artifacts|step_logs|hot_log_chunks"}
• shithub_actions_storage_bytes{kind="artifacts|step_logs|hot_log_chunks"}
• shithub_actions_step_timeouts_total

The committed dashboard JSON lives at:

deploy/monitoring/grafana/dashboards/actions.json

Load harness

bench/k6/actions-load.js exercises the runner HTTP API under concurrent job claims. It does not create workflow runs itself; seed the queue first with pushes or workflow dispatches that produce jobs matching the runner labels.

Required environment:

SHITHUB_BASE_URL=https://shithub.example.test \
SHITHUB_RUNNER_TOKENS=token-a,token-b,token-c \
k6 run bench/k6/actions-load.js

Useful knobs:

• SHITHUB_ACTIONS_VUS=50 controls concurrent virtual users.
• SHITHUB_ACTIONS_DURATION=10m controls the steady-state window.
• SHITHUB_RUNNER_LABELS=self-hosted,linux,ubuntu-latest,x64 sets heartbeat labels.
• SHITHUB_RUNNER_CAPACITY=17 keeps three runners near the 50-concurrent target.
• SHITHUB_ACTIONS_LOG_BYTES=4096 controls emitted log chunk size.

Healthy run expectations:

• queued jobs drain without unbounded shithub_actions_queue_depth growth;
• runner heartbeats keep advancing and no runner deadlocks;
• log append p99 stays below five seconds;
• retention metrics catch up after the retention sweep.

Emergency cancel

Start with a dry run:

shithubd admin actions cancel-all --dry-run --limit 100

Scope to one repository when possible:

shithubd admin actions cancel-all --dry-run --repo-id 42

Then confirm:

shithubd admin actions cancel-all --confirm --repo-id 42

Queued jobs are marked cancelled immediately. Running jobs receive cancel_requested=true; the runner sees that through /cancel-check, kills the active container, and reports terminal cancelled.

Common failures

• Run never appears: confirm the workflow file is under .shithub/workflows/, parse it with shithubd admin actions parse <file>, and verify the trigger event matches on:.
• Run stays queued: open the run page to see the requested runner labels, then run shithubd admin runner queue and confirm a live runner is registered with matching labels and capacity. Unsupported hosted labels such as windows-latest and macos-latest intentionally remain queued until an operator registers matching runners.
• Step logs buffer: verify the Caddy route above and confirm the SSE route is still mounted outside compression and short timeouts.
• actions/checkout@v4 fails: confirm the job is still running, the repo URL in the runner claim points at this shithub instance, and the runner host can reach smart HTTP. The checkout token is not valid after the job leaves running.
• Artifact uses: step fails: expected for now. Replace with a run: step until artifact support lands.
• Secrets appear masked inconsistently: check shithub_actions_log_scrub_replacements_total{location="server"} and confirm the job was claimed after the secret was created or rotated. Mask snapshots are captured at claim time.