shithub Public

Watch 1 Fork 0 Star 0

markdown · 2110 bytes Raw Blame History

Bench fixtures

S36 calls for four big-repo fixtures used by the nightly perf run:

big_repo_1m_commits/ — repository with 1M commits across a sensible branch graph
repo_100k_files/ — repository with 100k files in a reasonable directory structure
issues_100k/ — repo with 100k issues + 1M comments
users_org_5k_members/ — org with 5k members and 200 teams (one level)

These fixtures aren't generated yet — the seed cost is non-trivial (tens of minutes per fixture) and the current repo shape doesn't yet warrant the disk-and-CI-time spend. The S36 baseline uses the small dev seed fixtures via bench/run.go.

Generation plan (deferred)

Each fixture has a generator under bench/fixtures/<name>/seed.go that takes a fixed RNG seed and produces a deterministic on-disk shape. The CI integrity check re-runs the generator and asserts the resulting tree-hash matches the committed manifest, so a generator regression doesn't silently change what the bench measures.

Run order (when generators land):

go run ./bench/fixtures/big_repo_1m_commits -out=./bench/fixtures/big_repo_1m_commits
go run ./bench/fixtures/repo_100k_files     -out=./bench/fixtures/repo_100k_files
…

The fixtures themselves are gitignored (regenerable). The generators live in source.

Dev fixture (today)

The make seed flow in the repo root produces the small dev fixtures used by make bench-small. That's enough to catch regressions in the harness itself plus the small-scale handler latency floor; the big-fixture targets in S36's "Definition of done" land with the generators above.

Actions smoke fixture

actions/smoke-run-only.yml is the canonical first end-to-end Actions workflow fixture. Copy it into .shithub/workflows/smoke.yml in a repository with a registered ubuntu-latest runner.

actions/checkout-canary.yml is the next fixture once the run-only smoke is green. It verifies actions/checkout@v4 and pins that the runner workspace is on the workflow run's exact shithub.sha. Artifact aliases remain reserved until artifact execution is implemented.

View source

  
        1
        # Bench fixtures
      
        2
        
        3
        S36 calls for four big-repo fixtures used by the nightly perf run:
      
        4
        
        5
        - `big_repo_1m_commits/` — repository with 1M commits across a sensible branch graph
      
        6
        - `repo_100k_files/`   — repository with 100k files in a reasonable directory structure
      
        7
        - `issues_100k/`       — repo with 100k issues + 1M comments
      
        8
        - `users_org_5k_members/` — org with 5k members and 200 teams (one level)
      
        9
        
        10
        These fixtures aren't generated yet — the seed cost is non-trivial
      
        11
        (tens of minutes per fixture) and the current repo shape doesn't
      
        12
        yet warrant the disk-and-CI-time spend. The S36 baseline uses the
      
        13
        small dev seed fixtures via `bench/run.go`.
      
        14
        
        15
        ## Generation plan (deferred)
      
        16
        
        17
        Each fixture has a generator under `bench/fixtures/<name>/seed.go`
      
        18
        that takes a fixed RNG seed and produces a deterministic on-disk
      
        19
        shape. The CI integrity check re-runs the generator and asserts
      
        20
        the resulting tree-hash matches the committed manifest, so a
      
        21
        generator regression doesn't silently change what the bench measures.
      
        22
        
        23
        Run order (when generators land):
      
        24
        
        25
        ```
      
        26
        go run ./bench/fixtures/big_repo_1m_commits -out=./bench/fixtures/big_repo_1m_commits
      
        27
        go run ./bench/fixtures/repo_100k_files     -out=./bench/fixtures/repo_100k_files
      
        28
        …
      
        29
        ```
      
        30
        
        31
        The fixtures themselves are gitignored (regenerable). The generators
      
        32
        live in source.
      
        33
        
        34
        ## Dev fixture (today)
      
        35
        
        36
        The `make seed` flow in the repo root produces the small dev
      
        37
        fixtures used by `make bench-small`. That's enough to catch
      
        38
        regressions in the harness itself plus the small-scale handler
      
        39
        latency floor; the big-fixture targets in S36's "Definition of
      
        40
        done" land with the generators above.
      
        41
        
        42
        ## Actions smoke fixture
      
        43
        
        44
        `actions/smoke-run-only.yml` is the canonical first end-to-end Actions
      
        45
        workflow fixture. Copy it into `.shithub/workflows/smoke.yml` in a
      
        46
        repository with a registered `ubuntu-latest` runner.
      
        47
        
        48
        `actions/checkout-canary.yml` is the next fixture once the run-only smoke is
      
        49
        green. It verifies `actions/checkout@v4` and pins that the runner workspace is
      
        50
        on the workflow run's exact `shithub.sha`. Artifact aliases remain reserved
      
        51
        until artifact execution is implemented.

1	# Bench fixtures
2
3	S36 calls for four big-repo fixtures used by the nightly perf run:
4
5	- `big_repo_1m_commits/` — repository with 1M commits across a sensible branch graph
6	- `repo_100k_files/` — repository with 100k files in a reasonable directory structure
7	- `issues_100k/` — repo with 100k issues + 1M comments
8	- `users_org_5k_members/` — org with 5k members and 200 teams (one level)
9
10	These fixtures aren't generated yet — the seed cost is non-trivial
11	(tens of minutes per fixture) and the current repo shape doesn't
12	yet warrant the disk-and-CI-time spend. The S36 baseline uses the
13	small dev seed fixtures via `bench/run.go`.
14
15	## Generation plan (deferred)
16
17	Each fixture has a generator under `bench/fixtures/<name>/seed.go`
18	that takes a fixed RNG seed and produces a deterministic on-disk
19	shape. The CI integrity check re-runs the generator and asserts
20	the resulting tree-hash matches the committed manifest, so a
21	generator regression doesn't silently change what the bench measures.
22
23	Run order (when generators land):
24
25	```
26	go run ./bench/fixtures/big_repo_1m_commits -out=./bench/fixtures/big_repo_1m_commits
27	go run ./bench/fixtures/repo_100k_files -out=./bench/fixtures/repo_100k_files
28	…
29	```
30
31	The fixtures themselves are gitignored (regenerable). The generators
32	live in source.
33
34	## Dev fixture (today)
35
36	The `make seed` flow in the repo root produces the small dev
37	fixtures used by `make bench-small`. That's enough to catch
38	regressions in the harness itself plus the small-scale handler
39	latency floor; the big-fixture targets in S36's "Definition of
40	done" land with the generators above.
41
42	## Actions smoke fixture
43
44	`actions/smoke-run-only.yml` is the canonical first end-to-end Actions
45	workflow fixture. Copy it into `.shithub/workflows/smoke.yml` in a
46	repository with a registered `ubuntu-latest` runner.
47
48	`actions/checkout-canary.yml` is the next fixture once the run-only smoke is
49	green. It verifies `actions/checkout@v4` and pins that the runner workspace is
50	on the workflow run's exact `shithub.sha`. Artifact aliases remain reserved
51	until artifact execution is implemented.