shithub Public

Watch 1 Fork 0 Star 0
markdown · 2929 bytes Raw Blame History

Performance benchmarking

The S36 perf-pass ships a small in-repo benchmark harness so perf regressions are visible in PR review, not discovered in production.

Targets (warm cache, MVP hardware)

The S36 spec defines:

  • Tree of root on 100k-file repo: < 200ms p95
  • Blame on 10k-line file: < 500ms p95
  • Commits list page on 1M-commit repo: < 250ms p95
  • Issue list (state filter) on 100k issues: < 200ms p95
  • Notifications inbox first page: < 150ms p95

These targets assume the big-fixture generators land. Until they do, make bench-small runs against the dev seed and serves as a floor regression detector for the harness itself + handler latency on the small dataset.

Running

# Defaults: target=http://localhost:8080, iters=20.
make bench-small

# Pin a different target / iteration count.
BENCH_TARGET=http://staging.shithub.sh BENCH_ITERS=100 make bench-small

Output is one JSON line per scenario:

{"scenario":"home","iters":20,"ok_count":20,"p50_us":432,"p95_us":5692,"p99_us":5692,"max_us":5692,"mean_us":693.15}

ok_count == 0 for a scenario means every probe missed the expected status (typically: the dev seed doesn't have the repo the scenario targets). The harness keeps running and reports zeros so the suite doesn't bail mid-run.

Adding scenarios

Append to bench/run.go::main's scenarios slice:

{"my-scenario", "GET", "/some/path", 200},

Scenarios are intentionally URL-shape probes, not deep state manipulators. If a scenario needs a logged-in user or a specific repo state, prefer staging via make seed over making the harness write its own state.

N+1 query auditing

Wire a route's integration test to assert max-queries:

import "github.com/tenseleyFlow/shithub/internal/web/middleware"

r.Use(middleware.CountQueries())
// ... drive a request through r ...
if got := middleware.QueriesFor(req); got > 8 {
    t.Fatalf("issuesList ran %d queries; threshold 8", got)
}

The pgx tracer (internal/infra/db.QueryCounter) increments a per-context counter on every Query/QueryRow/Exec; the middleware opts the request context in. Production paths pay one context.WithValue per request — cheap, but the assertion only fires in tests.

Big-fixture plan (deferred)

bench/fixtures/README.md documents the planned generators for the 1M-commit / 100k-file / 100k-issue / 5k-member-org fixtures. They aren't generated yet — the seed cost is non-trivial and the small dev seed is sufficient as a floor regression detector while the perf surface is still settling. When the generators land, make bench-full (currently a stub) hooks them up.

Profile dumps

bench/profiles/ is the canonical home for pprof captures referenced by the perf docs. The S36 spec asks for profile dumps for the slowest scenarios; until the big fixtures land, the captures are dev-machine pprof of make bench-small and aren't checked in by default.

View source
1 # Performance benchmarking
2
3 The S36 perf-pass ships a small in-repo benchmark harness so perf
4 regressions are visible in PR review, not discovered in production.
5
6 ## Targets (warm cache, MVP hardware)
7
8 The S36 spec defines:
9
10 - Tree of root on 100k-file repo: < 200ms p95
11 - Blame on 10k-line file: < 500ms p95
12 - Commits list page on 1M-commit repo: < 250ms p95
13 - Issue list (state filter) on 100k issues: < 200ms p95
14 - Notifications inbox first page: < 150ms p95
15
16 These targets assume the big-fixture generators land. Until they
17 do, `make bench-small` runs against the dev seed and serves as a
18 floor regression detector for the harness itself + handler latency
19 on the small dataset.
20
21 ## Running
22
23 ```sh
24 # Defaults: target=http://localhost:8080, iters=20.
25 make bench-small
26
27 # Pin a different target / iteration count.
28 BENCH_TARGET=http://staging.shithub.sh BENCH_ITERS=100 make bench-small
29 ```
30
31 Output is one JSON line per scenario:
32
33 ```json
34 {"scenario":"home","iters":20,"ok_count":20,"p50_us":432,"p95_us":5692,"p99_us":5692,"max_us":5692,"mean_us":693.15}
35 ```
36
37 `ok_count == 0` for a scenario means every probe missed the
38 expected status (typically: the dev seed doesn't have the repo the
39 scenario targets). The harness keeps running and reports zeros so
40 the suite doesn't bail mid-run.
41
42 ## Adding scenarios
43
44 Append to `bench/run.go::main`'s `scenarios` slice:
45
46 ```go
47 {"my-scenario", "GET", "/some/path", 200},
48 ```
49
50 Scenarios are intentionally URL-shape probes, not deep state
51 manipulators. If a scenario needs a logged-in user or a specific
52 repo state, prefer staging via `make seed` over making the
53 harness write its own state.
54
55 ## N+1 query auditing
56
57 Wire a route's integration test to assert max-queries:
58
59 ```go
60 import "github.com/tenseleyFlow/shithub/internal/web/middleware"
61
62 r.Use(middleware.CountQueries())
63 // ... drive a request through r ...
64 if got := middleware.QueriesFor(req); got > 8 {
65 t.Fatalf("issuesList ran %d queries; threshold 8", got)
66 }
67 ```
68
69 The pgx tracer (`internal/infra/db.QueryCounter`) increments a
70 per-context counter on every Query/QueryRow/Exec; the middleware
71 opts the request context in. Production paths pay one
72 context.WithValue per request — cheap, but the assertion only
73 fires in tests.
74
75 ## Big-fixture plan (deferred)
76
77 `bench/fixtures/README.md` documents the planned generators for
78 the 1M-commit / 100k-file / 100k-issue / 5k-member-org fixtures.
79 They aren't generated yet — the seed cost is non-trivial and the
80 small dev seed is sufficient as a floor regression detector while
81 the perf surface is still settling. When the generators land,
82 `make bench-full` (currently a stub) hooks them up.
83
84 ## Profile dumps
85
86 `bench/profiles/` is the canonical home for `pprof` captures
87 referenced by the perf docs. The S36 spec asks for profile dumps
88 for the slowest scenarios; until the big fixtures land, the
89 captures are dev-machine pprof of `make bench-small` and aren't
90 checked in by default.