Markdown pipeline (internal)

S25 ships shithub's canonical markdown renderer. One package owns goldmark + bluemonday; every other package routes through markdown.Render. The boundary is enforced by scripts/lint-markdown-boundary.sh.

Architecture

internal/markdown/
    markdown.go          — package doc + Ref/Mention public types
    version.go           — Version int32 = 1 (pipeline stamp)
    opts.go              — Options + Resolvers structs
    render.go            — Render() entry point + RenderHTML shim
    sanitize.go          — bluemonday policy
    markdown_test.go     — XSS fixture suite + golden render tests
    extensions/
        extensions.go    — single ASTTransformer for refs/mentions/commits/emoji
        emoji.go         — curated shortcode → unicode map

Render pipeline

source bytes
    │
    ▼
[goldmark.Convert]  CommonMark + GFM (tables, strikethrough,
    │                autolinks, task lists), html.WithUnsafe
    │                so raw HTML reaches the sanitizer
    │
    ▼
[ASTTransformer]    walks Document, skips code/codespan/link/
    │                image/HTML subtrees, runs reCombined regex
    │                on each Text segment, replaces matches with
    │                Link nodes (mentions/refs/commits) or String
    │                nodes (emoji + plain-text fallbacks).
    │
    ▼
[bluemonday.SanitizeBytes] strict UGC policy: scheme allowlist
    │                       (http/https/mailto), <details>/<summary>/
    │                       <kbd>/<sup>/<sub>, language-* class on
    │                       code, no <script>/<style>/<iframe>/data:
    │
    ▼
clean HTML bytes

Each Render call builds a fresh goldmark.Markdown so the extension's per-call Options can plug in without races. The build is ~µs and removes any cross-render contamination of the transformer state.

Reference resolution

Options.Resolvers is the seam between the renderer and the rest of the runtime. Each resolver is optional; a nil resolver means "render that flavor as plain text" — no link, no error, no existence leak.

Visibility gating happens inside the resolver. The transformer trusts the resolver's ok return — if the resolver returns ok=false, the displayed text passes through as plain text.

The handler/orchestrator that wires the resolver MUST honor:

• User: rejects suspended users; respects org-team visibility once S30/S31 land.
• Issue: rejects when the repo isn't visible to the viewer (the S15 policy.Can(ActionIssueRead) gate).
• Commit: rejects when the SHA prefix doesn't match a real commit; doesn't care about visibility (commit URLs respect repo visibility at request time).

Sanitizer policy decisions

The base is bluemonday.UGCPolicy(). We diverge by:

• Allowing <details>, <summary>, <kbd>, <sup>, <sub>. Common in READMEs; not security-relevant.
• Allowing id on h1-h6 so anchor links work (Goldmark's WithAutoHeadingID).
• Allowing class on <code>, <pre>, <span> matching ^(?:language-[A-Za-z0-9_+-]+|chroma|chroma-[a-zA-Z]+|nl|ln|line|hl)$ so Chroma's syntax-highlighted output passes through.
• Allowing <input type="checkbox" disabled checked> — Goldmark's task-list output. The disabled and checked attributes are HTML boolean attrs; we don't constrain their values.
• Restricting URL schemes to http, https, mailto (no data:, no javascript:, no vbscript:, no ftp:). This is stricter than UGCPolicy's default and is the documented divergence from GitHub.

Pipeline-version stamp

Version (in version.go) is bumped when:

• The sanitizer policy changes (added/removed tag, attribute, scheme).
• A new AST extension or rendering output change.
• Goldmark / bluemonday major-version upgrade with output drift.

Callers store the rendered version alongside body_html_cached columns. On read, callers compare the stored version to Version; if they differ, the cache is stale and the caller re-renders. We never run a one-shot "re-render every comment" job.

Why ASTTransformer instead of inline parsers

Goldmark inline parsers run during the main parse pass and need a Trigger() []byte plus careful interaction with Goldmark's existing inline disambiguation (link/codespan/emphasis priorities).

A single parser.ASTTransformer walks the document after parsing, visits text nodes whose ancestors aren't code/codespan/link/image/ HTML, and runs one combined regex per node. The replacement inserts *ast.Link (mentions/refs/commits) or *ast.String (emoji + plain fallbacks) before the original text node, then removes the original.

Tradeoffs:

• ✅ Simpler than custom inline parsers.
• ✅ Trivial to add new patterns (extend reCombined).
• ✅ Single-pass — runs once per text node.
• ❌ Can't span across other inline nodes (e.g. a mention split by *emphasis*). That's fine — we don't want that anyway.

Hostile-input fixture suite

TestRender_HostileInputs is the test of record. Every fixture is an XSS vector; the pass condition is "rendered HTML contains no executable JS surface" — no <script>, no href="javascript:", no on* handlers, no <iframe> / <object> / <embed> / <base> / <meta> / <style>, no expression(), no <annotation-xml>.

Adding a vector when a CVE / advisory lands is cheap; the test file lists ~40 vectors today.

Performance budget

• 50 KiB body, full extensions: <30 ms p99 on MVP hardware.
• Inputs above MaxRenderInputBytes (1 MiB) are rejected by the defensive check in Render. The API layer enforces tighter per-surface caps (64 KiB or 256 KiB depending on context); this is the renderer's last-resort guardrail.

Refactor pass (S25)

S17/S18/S21/S22/S23/S24 used a per-sprint internal/repos/markdown.RenderHTML wrapper. S25 deleted that package and updated every importer to github.com/tenseleyFlow/shithub/internal/markdown. The shim markdown.RenderHTML(src) (string, error) is preserved so the swap is a one-line import edit per file.

Callers that want richer behavior — resolved refs, mentions, viewer-aware visibility — should call markdown.Render directly. The interim RenderHTML keeps a sensible default (SoftBreakAsBR on, no resolvers).

README presentation chrome

Repository READMEs use the same sanitized HTML pipeline as comments and issues. The repo page adds GitHub-parity presentation chrome in the web layer only:

• <pre> blocks inside .markdown-body are wrapped client-side with a stable code-block container and a copy button. The copied text comes from the original <code> or <pre> textContent, not from injected controls.
• The README outline menu is populated client-side from rendered h1- h6 elements. Goldmark-generated IDs are reused. Raw HTML headings without IDs get deterministic page-local slugs so the outline can link to them.
• The JavaScript only reads sanitized text/IDs and mutates local chrome; it is not part of the trust boundary. Sanitization remains entirely in internal/markdown.

Lint guard

scripts/lint-markdown-boundary.sh fails CI when goldmark or bluemonday is imported outside internal/markdown/. Test files are exempt. Anything else triggers the alarm; the fix is to swap the import to the canonical package.

Wired into make ci via the lint-markdown target.

Deferred