tenseleyflow/shithub / cf62ed2

Browse files

Add org GitHub import UI

Authored by mfwolffe <wolffemf@dukes.jmu.edu>
SHA
cf62ed2d4a9a44b32f623395ca0d29cae96a60c3
Parents
c96e2b4
Tree
db51c16

12 changed files

StatusFile+-
M docs/internal/orgs.md 29 0
M docs/internal/repo-create.md 10 0
M docs/internal/worker.md 8 6
M internal/orgs/github_import.go 2 2
A internal/orgs/github_import_test.go 178 0
A internal/web/handlers/orgs/imports.go 147 0
M internal/web/handlers/orgs/orgs.go 59 10
M internal/web/static/css/shithub.css 158 0
A internal/web/templates/orgs/import_progress.html 62 0
M internal/web/templates/orgs/new.html 16 3
A internal/web/templates/orgs/settings_import.html 78 0
M internal/web/templates/orgs/settings_profile.html 1 0
docs/internal/orgs.mdmodified
@@ -34,6 +34,9 @@ POST /organizations/{org}/settings/profile
3434
 POST /organizations/{org}/settings/profile/avatar
3535
 POST /organizations/{org}/settings/profile/avatar/remove
3636
 POST /organizations/{org}/settings/delete
37
+GET  /organizations/{org}/settings/import
38
+POST /organizations/{org}/settings/import
39
+GET  /organizations/{org}/imports/{importID}
3740
 GET  /invitations/{token}          accept/decline view (auth required)
3841
 POST /invitations/{token}/accept
3942
 POST /invitations/{token}/decline
@@ -94,6 +97,28 @@ avatar pipeline and object store.
9497
 `orgs.SoftDelete` after an owner confirms the slug; the hard-delete
9598
 worker still owns permanent removal after the grace window.
9699
 
100
+`GET /organizations/{org}/settings/import` renders the owner-only
101
+GitHub organization import page. Owners can provide a GitHub
102
+organization name or `github.com/<org>` URL and, optionally, a token.
103
+Untokened imports discover public repositories only. Token-backed
104
+imports use GitHub's `type=all` repo listing, persist the token
105
+encrypted with the server secret box, and keep private upstream repos
106
+private on shithub.
107
+
108
+`POST /organizations/{org}/settings/import` creates an
109
+`org_github_imports` row and enqueues
110
+`worker.KindOrgGitHubImportDiscover`. `GET
111
+/organizations/{org}/imports/{importID}` shows a polling progress page
112
+with per-repo queued/importing/imported/skipped/failed state. The
113
+discover job pages through the GitHub API, records one
114
+`org_github_import_repos` row per repository, and enqueues an import job
115
+per repo. Each repo job creates the org-owned shithub repository, saves
116
+the GitHub source remote, fetches heads/tags with a temporary Git
117
+askpass helper when a token is present, updates the default branch from
118
+fetched refs, and enqueues indexing + size recalculation. Existing
119
+active repositories in the organization are skipped instead of
120
+overwritten.
121
+
97122
 Repo visibility is filtered through `policy.IsVisibleTo` using an actor
98123
 constructed from `middleware.CurrentUser`, including suspension,
99124
 site-admin, and impersonation write-mode fields. Anonymous viewers only
@@ -187,3 +212,7 @@ old slug for 301s during the rename cooldown.
187212
   re-clicking Invite doesn't spam the recipient.
188213
 * **Reserved slugs**: `auth.IsReserved` filter applies to org
189214
   slugs the same way it applies to usernames.
215
+* **Org import token storage**: the token never enters a Git URL, logs,
216
+  or plaintext database column. It is encrypted in
217
+  `org_github_imports.token_ciphertext` and supplied to git via a
218
+  temporary askpass script scoped to the fetch process.
docs/internal/repo-create.mdmodified
@@ -106,6 +106,16 @@ each submodule repo with its source remote, then create/import the parent
106106
 repo, and the pinned submodule links can hydrate exact detached tree
107107
 views on demand.
108108
 
109
+Organization GitHub imports reuse the same source-remote path in bulk.
110
+The org import worker creates one org-owned repository per discovered
111
+GitHub repo, persists the upstream clone URL in `repo_source_remotes`,
112
+fetches heads/tags, and then refreshes `default_branch` /
113
+`default_branch_oid` exactly like the single-repo import flow. Private
114
+repositories are only imported when the owner supplied a GitHub token;
115
+the token is stored encrypted on the import row and passed to git via
116
+temporary askpass environment, never embedded into the persisted remote
117
+URL.
118
+
109119
 ## Plumbing-only initial commit
110120
 
111121
 Why no working tree:
docs/internal/worker.mdmodified
@@ -30,12 +30,14 @@ backstop poll (every 5s by default) covers dropped notifications.
3030
 
3131
 ## Job kinds shipped in S14
3232
 
33
-| Kind                   | Trigger                              | Idempotent on            |
34
-| ---------------------- | ------------------------------------ | ------------------------ |
35
-| `push:process`         | post-receive hook per ref            | `push_events.processed_at` |
36
-| `repo:size_recalc`     | enqueued by `push:process`           | overwrite-last-wins        |
37
-| `jobs:purge_completed` | future cron / manual ad-hoc          | always safe to re-run      |
38
-| `trending:compute`     | recurring self-scheduled S42 job     | append-only snapshots      |
33
+| Kind                         | Trigger                              | Idempotent on                    |
34
+| ---------------------------- | ------------------------------------ | -------------------------------- |
35
+| `push:process`               | post-receive hook per ref            | `push_events.processed_at`       |
36
+| `repo:size_recalc`           | enqueued by `push:process`           | overwrite-last-wins              |
37
+| `org:github_import_discover` | org import request                   | `org_github_imports.status`      |
38
+| `org:github_import_repo`     | import discovery per GitHub repo     | `org_github_import_repos.status` |
39
+| `jobs:purge_completed`       | future cron / manual ad-hoc          | always safe to re-run            |
40
+| `trending:compute`           | recurring self-scheduled S42 job     | append-only snapshots            |
3941
 
4042
 Adding a new kind: write the handler in `internal/worker/jobs/<kind>.go`,
4143
 add the `Kind` constant to `internal/worker/types.go`, register it in
internal/orgs/github_import.gomodified
@@ -47,7 +47,7 @@ var (
4747
 	ErrImportTokenKeyNeeded = errors.New("orgs: import token encryption key is not configured")
4848
 )
4949
 
50
-var githubOrgRE = regexp.MustCompile(`^[A-Za-z0-9](?:[A-Za-z0-9-]{0,99})$`)
50
+var githubOrgRE = regexp.MustCompile(`^[A-Za-z0-9](?:[A-Za-z0-9-]{0,37}[A-Za-z0-9])?$`)
5151
 
5252
 // ImportDeps wires org-import orchestration.
5353
 type ImportDeps struct {
@@ -127,7 +127,7 @@ func NormalizeGitHubOrg(raw string) (string, error) {
127127
 	org = strings.TrimPrefix(org, "https://github.com/")
128128
 	org = strings.TrimPrefix(org, "http://github.com/")
129129
 	org = strings.Trim(org, "/")
130
-	if org == "" || strings.Contains(org, "/") || !githubOrgRE.MatchString(org) {
130
+	if org == "" || strings.Contains(org, "/") || strings.Contains(org, "--") || !githubOrgRE.MatchString(org) {
131131
 		return "", ErrInvalidGitHubOrg
132132
 	}
133133
 	return org, nil
internal/orgs/github_import_test.goadded
@@ -0,0 +1,178 @@
1
+// SPDX-License-Identifier: AGPL-3.0-or-later
2
+
3
+package orgs_test
4
+
5
+import (
6
+	"context"
7
+	"encoding/json"
8
+	"fmt"
9
+	"io"
10
+	"log/slog"
11
+	"net/http"
12
+	"strconv"
13
+	"strings"
14
+	"testing"
15
+
16
+	"github.com/tenseleyFlow/shithub/internal/auth/secretbox"
17
+	"github.com/tenseleyFlow/shithub/internal/orgs"
18
+	"github.com/tenseleyFlow/shithub/internal/worker"
19
+)
20
+
21
+func TestNormalizeGitHubOrg(t *testing.T) {
22
+	t.Parallel()
23
+	tests := []struct {
24
+		name    string
25
+		raw     string
26
+		want    string
27
+		wantErr bool
28
+	}{
29
+		{name: "bare", raw: "tenseleyFlow", want: "tenseleyFlow"},
30
+		{name: "url", raw: "https://github.com/FortranGoingOnForty/", want: "FortranGoingOnForty"},
31
+		{name: "path rejected", raw: "github.com/owner/repo", wantErr: true},
32
+		{name: "trailing hyphen rejected", raw: "bad-", wantErr: true},
33
+		{name: "double hyphen rejected", raw: "bad--name", wantErr: true},
34
+	}
35
+	for _, tt := range tests {
36
+		t.Run(tt.name, func(t *testing.T) {
37
+			got, err := orgs.NormalizeGitHubOrg(tt.raw)
38
+			if tt.wantErr {
39
+				if err == nil {
40
+					t.Fatalf("NormalizeGitHubOrg(%q) succeeded", tt.raw)
41
+				}
42
+				return
43
+			}
44
+			if err != nil {
45
+				t.Fatalf("NormalizeGitHubOrg(%q): %v", tt.raw, err)
46
+			}
47
+			if got != tt.want {
48
+				t.Fatalf("NormalizeGitHubOrg(%q)=%q, want %q", tt.raw, got, tt.want)
49
+			}
50
+		})
51
+	}
52
+}
53
+
54
+func TestGitHubClientListOrgReposPaginatesAndUsesToken(t *testing.T) {
55
+	t.Parallel()
56
+	var seenPages []string
57
+	rt := roundTripFunc(func(r *http.Request) (*http.Response, error) {
58
+		if got, want := r.URL.Path, "/orgs/tenseleyFlow/repos"; got != want {
59
+			t.Fatalf("path=%q, want %q", got, want)
60
+		}
61
+		if got, want := r.Header.Get("Authorization"), "Bearer test-token"; got != want {
62
+			t.Fatalf("Authorization=%q, want %q", got, want)
63
+		}
64
+		if got, want := r.URL.Query().Get("type"), "all"; got != want {
65
+			t.Fatalf("type=%q, want %q", got, want)
66
+		}
67
+		seenPages = append(seenPages, r.URL.Query().Get("page"))
68
+		var body strings.Builder
69
+		if r.URL.Query().Get("page") == "1" {
70
+			writeGitHubRepoList(t, &body, 100)
71
+		} else {
72
+			_, _ = io.WriteString(&body, `[{"id":101,"name":"last","full_name":"tenseleyFlow/last","clone_url":"https://github.com/tenseleyFlow/last.git","description":"last repo","default_branch":"trunk"}]`)
73
+		}
74
+		return &http.Response{
75
+			StatusCode: http.StatusOK,
76
+			Status:     "200 OK",
77
+			Header:     http.Header{"Content-Type": []string{"application/json"}},
78
+			Body:       io.NopCloser(strings.NewReader(body.String())),
79
+			Request:    r,
80
+		}, nil
81
+	})
82
+
83
+	repos, err := (orgs.GitHubClient{
84
+		HTTPClient: &http.Client{Transport: rt},
85
+		BaseURL:    "https://api.github.test",
86
+		UserAgent:  "shithub-test",
87
+	}).ListOrgRepos(context.Background(), "tenseleyFlow", "test-token")
88
+	if err != nil {
89
+		t.Fatalf("ListOrgRepos: %v", err)
90
+	}
91
+	if len(repos) != 101 {
92
+		t.Fatalf("len(repos)=%d, want 101", len(repos))
93
+	}
94
+	if got := fmt.Sprint(seenPages); got != "[1 2]" {
95
+		t.Fatalf("pages=%s, want [1 2]", got)
96
+	}
97
+	if repos[100].FullName != "tenseleyFlow/last" || repos[100].Description != "last repo" || repos[100].DefaultBranch != "trunk" {
98
+		t.Fatalf("last repo decoded incorrectly: %+v", repos[100])
99
+	}
100
+}
101
+
102
+type roundTripFunc func(*http.Request) (*http.Response, error)
103
+
104
+func (f roundTripFunc) RoundTrip(r *http.Request) (*http.Response, error) {
105
+	return f(r)
106
+}
107
+
108
+func writeGitHubRepoList(t *testing.T, w io.Writer, count int) {
109
+	t.Helper()
110
+	payload := make([]map[string]any, 0, count)
111
+	for i := 0; i < count; i++ {
112
+		payload = append(payload, map[string]any{
113
+			"id":             i + 1,
114
+			"name":           fmt.Sprintf("repo-%03d", i),
115
+			"full_name":      fmt.Sprintf("tenseleyFlow/repo-%03d", i),
116
+			"clone_url":      fmt.Sprintf("https://github.com/tenseleyFlow/repo-%03d.git", i),
117
+			"description":    nil,
118
+			"default_branch": "trunk",
119
+			"private":        false,
120
+			"fork":           false,
121
+		})
122
+	}
123
+	if err := json.NewEncoder(w).Encode(payload); err != nil {
124
+		t.Fatalf("encode GitHub response: %v", err)
125
+	}
126
+}
127
+
128
+func TestStartGitHubImportPersistsEncryptedTokenAndDiscoveryJob(t *testing.T) {
129
+	pool, deps, alice := setup(t)
130
+	row, err := orgs.Create(context.Background(), deps, orgs.CreateParams{
131
+		Slug: "acme", DisplayName: "Acme Inc", CreatedByUserID: alice,
132
+	})
133
+	if err != nil {
134
+		t.Fatalf("create org: %v", err)
135
+	}
136
+	key, err := secretbox.GenerateKey()
137
+	if err != nil {
138
+		t.Fatalf("GenerateKey: %v", err)
139
+	}
140
+	box, err := secretbox.FromBytes(key)
141
+	if err != nil {
142
+		t.Fatalf("FromBytes: %v", err)
143
+	}
144
+	imp, err := orgs.StartGitHubImport(context.Background(), orgs.ImportDeps{
145
+		Pool:   pool,
146
+		Box:    box,
147
+		Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
148
+	}, orgs.StartGitHubImportParams{
149
+		OrgID: row.ID, SourceOrg: "https://github.com/tenseleyFlow/",
150
+		RequestedByUserID: alice, Token: "ghp_secret",
151
+	})
152
+	if err != nil {
153
+		t.Fatalf("StartGitHubImport: %v", err)
154
+	}
155
+	if imp.SourceOrg != "tenseleyFlow" || !imp.IncludePrivate || !imp.TokenPresent {
156
+		t.Fatalf("unexpected import row: %+v", imp)
157
+	}
158
+	token, err := orgs.DecryptGitHubImportToken(imp, box)
159
+	if err != nil {
160
+		t.Fatalf("DecryptGitHubImportToken: %v", err)
161
+	}
162
+	if token != "ghp_secret" {
163
+		t.Fatalf("decrypted token=%q", token)
164
+	}
165
+	if string(imp.TokenCiphertext) == "ghp_secret" {
166
+		t.Fatal("token stored in plaintext")
167
+	}
168
+	var jobs int
169
+	if err := pool.QueryRow(context.Background(),
170
+		`SELECT count(*) FROM jobs WHERE kind = $1 AND payload->>'import_id' = $2`,
171
+		worker.KindOrgGitHubImportDiscover, strconv.FormatInt(imp.ID, 10),
172
+	).Scan(&jobs); err != nil {
173
+		t.Fatalf("query jobs: %v", err)
174
+	}
175
+	if jobs != 1 {
176
+		t.Fatalf("discovery jobs=%d, want 1", jobs)
177
+	}
178
+}
internal/web/handlers/orgs/imports.goadded
@@ -0,0 +1,147 @@
1
+// SPDX-License-Identifier: AGPL-3.0-or-later
2
+
3
+package orgs
4
+
5
+import (
6
+	"errors"
7
+	"net/http"
8
+	"net/url"
9
+	"strconv"
10
+	"strings"
11
+
12
+	"github.com/go-chi/chi/v5"
13
+	"github.com/jackc/pgx/v5"
14
+
15
+	orgdomain "github.com/tenseleyFlow/shithub/internal/orgs"
16
+	orgsdb "github.com/tenseleyFlow/shithub/internal/orgs/sqlc"
17
+	"github.com/tenseleyFlow/shithub/internal/web/middleware"
18
+)
19
+
20
+type importForm struct {
21
+	GitHubOrg   string
22
+	GitHubToken string
23
+}
24
+
25
+func (h *Handlers) settingsImport(w http.ResponseWriter, r *http.Request) {
26
+	org, ok := h.loadOrgSettingsOwner(w, r)
27
+	if !ok {
28
+		return
29
+	}
30
+	h.renderSettingsImport(w, r, org, importForm{}, "", importNotice(r.URL.Query().Get("notice")))
31
+}
32
+
33
+func (h *Handlers) settingsImportSubmit(w http.ResponseWriter, r *http.Request) {
34
+	org, ok := h.loadOrgSettingsOwner(w, r)
35
+	if !ok {
36
+		return
37
+	}
38
+	if err := r.ParseForm(); err != nil {
39
+		h.d.Render.HTTPError(w, r, http.StatusBadRequest, "form parse")
40
+		return
41
+	}
42
+	form := importForm{
43
+		GitHubOrg:   strings.TrimSpace(r.PostFormValue("github_org")),
44
+		GitHubToken: strings.TrimSpace(r.PostFormValue("github_token")),
45
+	}
46
+	viewer := middleware.CurrentUserFromContext(r.Context())
47
+	imp, err := orgdomain.StartGitHubImport(r.Context(), orgdomain.ImportDeps{
48
+		Pool: h.d.Pool, Box: h.d.SecretBox, Logger: h.d.Logger,
49
+	}, orgdomain.StartGitHubImportParams{
50
+		OrgID: org.ID, SourceOrg: form.GitHubOrg,
51
+		RequestedByUserID: viewer.ID, Token: form.GitHubToken,
52
+	})
53
+	if err != nil {
54
+		h.renderSettingsImport(w, r, org, form.withoutToken(), friendlyImportError(err), "")
55
+		return
56
+	}
57
+	http.Redirect(w, r, "/organizations/"+org.Slug+"/imports/"+strconv.FormatInt(imp.ID, 10), http.StatusSeeOther)
58
+}
59
+
60
+func (f importForm) withoutToken() importForm {
61
+	f.GitHubToken = ""
62
+	return f
63
+}
64
+
65
+func (h *Handlers) importProgress(w http.ResponseWriter, r *http.Request) {
66
+	org, ok := h.loadOrgSettingsOwner(w, r)
67
+	if !ok {
68
+		return
69
+	}
70
+	importID, err := strconv.ParseInt(chi.URLParam(r, "importID"), 10, 64)
71
+	if err != nil || importID <= 0 {
72
+		h.d.Render.HTTPError(w, r, http.StatusNotFound, "")
73
+		return
74
+	}
75
+	q := orgsdb.New()
76
+	progress, err := q.GetOrgGithubImportProgress(r.Context(), h.d.Pool, orgsdb.GetOrgGithubImportProgressParams{
77
+		ID:    importID,
78
+		OrgID: org.ID,
79
+	})
80
+	if err != nil {
81
+		if errors.Is(err, pgx.ErrNoRows) {
82
+			h.d.Render.HTTPError(w, r, http.StatusNotFound, "")
83
+			return
84
+		}
85
+		h.d.Render.HTTPError(w, r, http.StatusInternalServerError, "")
86
+		return
87
+	}
88
+	items, err := q.ListOrgGithubImportRepos(r.Context(), h.d.Pool, importID)
89
+	if err != nil {
90
+		h.d.Render.HTTPError(w, r, http.StatusInternalServerError, "")
91
+		return
92
+	}
93
+	if err := h.d.Render.RenderPage(w, r, "orgs/import_progress", map[string]any{
94
+		"Title":        org.Slug + " - GitHub import",
95
+		"CSRFToken":    middleware.CSRFTokenForRequest(r),
96
+		"Org":          org,
97
+		"AvatarURL":    "/avatars/" + url.PathEscape(org.Slug),
98
+		"ActiveOrgNav": "settings",
99
+		"Progress":     progress,
100
+		"Items":        items,
101
+		"IsTerminal":   orgdomain.IsTerminalImportStatus(progress.Status),
102
+	}); err != nil {
103
+		h.d.Logger.ErrorContext(r.Context(), "org import: render progress", "error", err)
104
+	}
105
+}
106
+
107
+func (h *Handlers) renderSettingsImport(w http.ResponseWriter, r *http.Request, org orgsdb.Org, form importForm, errMsg, notice string) {
108
+	imports, err := orgsdb.New().ListOrgGithubImportsForOrg(r.Context(), h.d.Pool, orgsdb.ListOrgGithubImportsForOrgParams{
109
+		OrgID: org.ID,
110
+		Limit: 10,
111
+	})
112
+	if err != nil {
113
+		h.d.Logger.WarnContext(r.Context(), "org import: list imports", "error", err, "org_id", org.ID)
114
+	}
115
+	_ = h.d.Render.RenderPage(w, r, "orgs/settings_import", map[string]any{
116
+		"Title":        org.Slug + " - repository import",
117
+		"CSRFToken":    middleware.CSRFTokenForRequest(r),
118
+		"Org":          org,
119
+		"AvatarURL":    "/avatars/" + url.PathEscape(org.Slug),
120
+		"ActiveOrgNav": "settings",
121
+		"Form":         form,
122
+		"Error":        errMsg,
123
+		"Notice":       notice,
124
+		"Imports":      imports,
125
+		"SecretBoxOK":  h.d.SecretBox != nil,
126
+	})
127
+}
128
+
129
+func friendlyImportError(err error) string {
130
+	switch {
131
+	case errors.Is(err, orgdomain.ErrInvalidGitHubOrg):
132
+		return "GitHub organization must be a valid organization name or github.com organization URL."
133
+	case errors.Is(err, orgdomain.ErrImportTokenKeyNeeded):
134
+		return "GitHub token imports require the server secret key to be configured."
135
+	default:
136
+		return "Could not start the GitHub organization import."
137
+	}
138
+}
139
+
140
+func importNotice(code string) string {
141
+	switch code {
142
+	case "start-failed":
143
+		return "The organization was created, but the import could not be started. Try again here."
144
+	default:
145
+		return ""
146
+	}
147
+}
internal/web/handlers/orgs/orgs.gomodified
@@ -10,6 +10,9 @@
1010
 //	POST /{org}/people/{user}/role                          change role
1111
 //	POST /{org}/people/{user}/remove                        remove member
1212
 //	GET  /organizations/{org}/settings/profile              profile settings
13
+//	GET  /organizations/{org}/settings/import               GitHub org import
14
+//	POST /organizations/{org}/settings/import               start GitHub org import
15
+//	GET  /organizations/{org}/imports/{importID}            GitHub org import progress
1316
 //	GET  /organizations/{org}/settings/{secrets,variables}/actions
1417
 //	POST /organizations/{org}/settings/{secrets,variables}/actions
1518
 //	GET  /invitations/{token}                               accept/decline view
@@ -89,6 +92,9 @@ func (h *Handlers) MountCreate(r chi.Router) {
8992
 	r.Post("/organizations/{org}/settings/profile/avatar", h.settingsAvatarUpload)
9093
 	r.Post("/organizations/{org}/settings/profile/avatar/remove", h.settingsAvatarRemove)
9194
 	r.Post("/organizations/{org}/settings/delete", h.settingsDelete)
95
+	r.Get("/organizations/{org}/settings/import", h.settingsImport)
96
+	r.Post("/organizations/{org}/settings/import", h.settingsImportSubmit)
97
+	r.Get("/organizations/{org}/imports/{importID}", h.importProgress)
9298
 	r.Get("/organizations/{org}/settings/secrets/actions", h.settingsActionsSecrets)
9399
 	r.Post("/organizations/{org}/settings/secrets/actions", h.settingsActionsSecretSet)
94100
 	r.Post("/organizations/{org}/settings/secrets/actions/{name}/delete", h.settingsActionsSecretDelete)
@@ -157,7 +163,15 @@ func (h *Handlers) newForm(w http.ResponseWriter, r *http.Request) {
157163
 		http.Redirect(w, r, "/login?next=/organizations/new", http.StatusSeeOther)
158164
 		return
159165
 	}
160
-	h.renderNewForm(w, r, "", "")
166
+	h.renderNewForm(w, r, orgCreateForm{}, "")
167
+}
168
+
169
+type orgCreateForm struct {
170
+	Slug         string
171
+	DisplayName  string
172
+	BillingEmail string
173
+	GitHubOrg    string
174
+	GitHubToken  string
161175
 }
162176
 
163177
 func (h *Handlers) createSubmit(w http.ResponseWriter, r *http.Request) {
@@ -170,28 +184,63 @@ func (h *Handlers) createSubmit(w http.ResponseWriter, r *http.Request) {
170184
 		h.d.Render.HTTPError(w, r, http.StatusBadRequest, "")
171185
 		return
172186
 	}
173
-	slug := strings.TrimSpace(r.PostFormValue("slug"))
174
-	displayName := strings.TrimSpace(r.PostFormValue("display_name"))
175
-	billingEmail := strings.TrimSpace(r.PostFormValue("billing_email"))
187
+	form := orgCreateForm{
188
+		Slug:         strings.TrimSpace(r.PostFormValue("slug")),
189
+		DisplayName:  strings.TrimSpace(r.PostFormValue("display_name")),
190
+		BillingEmail: strings.TrimSpace(r.PostFormValue("billing_email")),
191
+		GitHubOrg:    strings.TrimSpace(r.PostFormValue("github_org")),
192
+		GitHubToken:  strings.TrimSpace(r.PostFormValue("github_token")),
193
+	}
194
+	if form.GitHubOrg != "" {
195
+		if _, err := orgs.NormalizeGitHubOrg(form.GitHubOrg); err != nil {
196
+			h.renderNewForm(w, r, form, "GitHub organization must be a valid organization name or github.com organization URL.")
197
+			return
198
+		}
199
+		if form.GitHubToken != "" && h.d.SecretBox == nil {
200
+			h.renderNewForm(w, r, form.withoutToken(), "GitHub token imports require the server secret key to be configured.")
201
+			return
202
+		}
203
+	}
176204
 
177205
 	row, err := orgs.Create(r.Context(), h.deps(), orgs.CreateParams{
178
-		Slug:            slug,
179
-		DisplayName:     displayName,
180
-		BillingEmail:    billingEmail,
206
+		Slug:            form.Slug,
207
+		DisplayName:     form.DisplayName,
208
+		BillingEmail:    form.BillingEmail,
181209
 		CreatedByUserID: viewer.ID,
182210
 	})
183211
 	if err != nil {
184
-		h.renderNewForm(w, r, slug, friendlyOrgErr(err))
212
+		h.renderNewForm(w, r, form.withoutToken(), friendlyOrgErr(err))
213
+		return
214
+	}
215
+	if form.GitHubOrg != "" {
216
+		imp, err := orgs.StartGitHubImport(r.Context(), orgs.ImportDeps{
217
+			Pool: h.d.Pool, Box: h.d.SecretBox, Logger: h.d.Logger,
218
+		}, orgs.StartGitHubImportParams{
219
+			OrgID: row.ID, SourceOrg: form.GitHubOrg,
220
+			RequestedByUserID: viewer.ID, Token: form.GitHubToken,
221
+		})
222
+		if err != nil {
223
+			h.d.Logger.WarnContext(r.Context(), "orgs: start GitHub import after create", "error", err, "org_id", row.ID)
224
+			http.Redirect(w, r, "/organizations/"+row.Slug+"/settings/import?notice=start-failed", http.StatusSeeOther)
225
+			return
226
+		}
227
+		http.Redirect(w, r, "/organizations/"+row.Slug+"/imports/"+strconv.FormatInt(imp.ID, 10), http.StatusSeeOther)
185228
 		return
186229
 	}
187230
 	http.Redirect(w, r, "/"+row.Slug, http.StatusSeeOther)
188231
 }
189232
 
190
-func (h *Handlers) renderNewForm(w http.ResponseWriter, r *http.Request, slug, errMsg string) {
233
+func (f orgCreateForm) withoutToken() orgCreateForm {
234
+	f.GitHubToken = ""
235
+	return f
236
+}
237
+
238
+func (h *Handlers) renderNewForm(w http.ResponseWriter, r *http.Request, form orgCreateForm, errMsg string) {
191239
 	if err := h.d.Render.RenderPage(w, r, "orgs/new", map[string]any{
192240
 		"Title":     "New organization",
193241
 		"CSRFToken": middleware.CSRFTokenForRequest(r),
194
-		"Slug":      slug,
242
+		"Slug":      form.Slug,
243
+		"Form":      form,
195244
 		"Error":     errMsg,
196245
 	}); err != nil {
197246
 		h.d.Logger.ErrorContext(r.Context(), "orgs: render", "tpl", "orgs/new", "error", err)
internal/web/static/css/shithub.cssmodified
@@ -3592,6 +3592,164 @@ button.shithub-contrib-setting-item:hover {
35923592
   margin: 0.15rem 0 0;
35933593
   color: var(--fg-muted);
35943594
 }
3595
+.shithub-org-import-create {
3596
+  display: grid;
3597
+  gap: 0.75rem;
3598
+  margin: 1rem 0;
3599
+  padding: 1rem;
3600
+  border: 1px solid var(--border-default);
3601
+  border-radius: 6px;
3602
+}
3603
+.shithub-org-import-create legend {
3604
+  padding: 0 0.35rem;
3605
+  font-weight: 600;
3606
+}
3607
+.shithub-org-import-create p {
3608
+  margin: 0;
3609
+  color: var(--fg-muted);
3610
+}
3611
+.shithub-org-import-form {
3612
+  max-width: 640px;
3613
+}
3614
+.shithub-org-import-history {
3615
+  overflow: hidden;
3616
+  border: 1px solid var(--border-default);
3617
+  border-radius: 6px;
3618
+}
3619
+.shithub-org-import-history-row {
3620
+  display: grid;
3621
+  grid-template-columns: minmax(0, 1fr) auto auto;
3622
+  gap: 1rem;
3623
+  align-items: center;
3624
+  padding: 0.75rem 1rem;
3625
+  border-top: 1px solid var(--border-muted, var(--border-default));
3626
+  color: var(--fg-default);
3627
+  text-decoration: none;
3628
+}
3629
+.shithub-org-import-history-row:first-child {
3630
+  border-top: 0;
3631
+}
3632
+.shithub-org-import-history-row:hover {
3633
+  background: var(--canvas-subtle);
3634
+  text-decoration: none;
3635
+}
3636
+.shithub-org-import-progress {
3637
+  max-width: 960px;
3638
+  margin: 0 auto;
3639
+  padding: 1.5rem 1rem 3rem;
3640
+}
3641
+.shithub-org-import-progress-head {
3642
+  display: flex;
3643
+  justify-content: space-between;
3644
+  gap: 1rem;
3645
+  align-items: center;
3646
+  padding-bottom: 1rem;
3647
+  border-bottom: 1px solid var(--border-default);
3648
+}
3649
+.shithub-org-import-progress-head h1 {
3650
+  margin: 0.15rem 0 0;
3651
+  font-size: 1.5rem;
3652
+}
3653
+.shithub-org-import-status {
3654
+  display: inline-flex;
3655
+  align-items: center;
3656
+  gap: 0.35rem;
3657
+  width: max-content;
3658
+  padding: 0.15rem 0.5rem;
3659
+  border: 1px solid var(--border-default);
3660
+  border-radius: 999px;
3661
+  color: var(--fg-muted);
3662
+  font-size: 0.75rem;
3663
+  font-weight: 600;
3664
+  text-transform: capitalize;
3665
+}
3666
+.shithub-org-import-status.is-imported,
3667
+.shithub-org-import-status.is-completed {
3668
+  color: var(--success-fg);
3669
+  border-color: color-mix(in srgb, var(--success-fg) 45%, transparent);
3670
+}
3671
+.shithub-org-import-status.is-failed {
3672
+  color: var(--danger-fg);
3673
+  border-color: color-mix(in srgb, var(--danger-fg) 45%, transparent);
3674
+}
3675
+.shithub-org-import-status.is-importing,
3676
+.shithub-org-import-status.is-discovering {
3677
+  color: var(--accent-fg);
3678
+  border-color: color-mix(in srgb, var(--accent-fg) 45%, transparent);
3679
+}
3680
+.shithub-org-import-counts {
3681
+  display: grid;
3682
+  grid-template-columns: repeat(5, minmax(0, 1fr));
3683
+  gap: 0;
3684
+  margin: 1rem 0;
3685
+  overflow: hidden;
3686
+  border: 1px solid var(--border-default);
3687
+  border-radius: 6px;
3688
+  background: var(--canvas-default);
3689
+}
3690
+.shithub-org-import-counts div {
3691
+  display: grid;
3692
+  gap: 0.15rem;
3693
+  padding: 0.8rem 1rem;
3694
+  border-left: 1px solid var(--border-muted, var(--border-default));
3695
+}
3696
+.shithub-org-import-counts div:first-child {
3697
+  border-left: 0;
3698
+}
3699
+.shithub-org-import-counts strong {
3700
+  font-size: 1.2rem;
3701
+}
3702
+.shithub-org-import-counts span {
3703
+  color: var(--fg-muted);
3704
+  font-size: 0.8rem;
3705
+}
3706
+.shithub-org-import-list {
3707
+  overflow: hidden;
3708
+  border: 1px solid var(--border-default);
3709
+  border-radius: 6px;
3710
+  background: var(--canvas-default);
3711
+}
3712
+.shithub-org-import-row {
3713
+  display: grid;
3714
+  grid-template-columns: minmax(0, 1fr) auto;
3715
+  gap: 1rem;
3716
+  align-items: center;
3717
+  padding: 0.85rem 1rem;
3718
+  border-top: 1px solid var(--border-muted, var(--border-default));
3719
+}
3720
+.shithub-org-import-row:first-child {
3721
+  border-top: 0;
3722
+}
3723
+.shithub-org-import-row-title {
3724
+  display: flex;
3725
+  flex-wrap: wrap;
3726
+  gap: 0.45rem;
3727
+  align-items: center;
3728
+  font-weight: 600;
3729
+}
3730
+.shithub-org-import-row p {
3731
+  margin: 0.2rem 0 0;
3732
+  color: var(--fg-muted);
3733
+}
3734
+.shithub-org-import-row .shithub-org-import-error {
3735
+  color: var(--danger-fg);
3736
+}
3737
+.shithub-spinner {
3738
+  width: 12px;
3739
+  height: 12px;
3740
+  border: 2px solid currentColor;
3741
+  border-right-color: transparent;
3742
+  border-radius: 50%;
3743
+  animation: shithub-spin 0.75s linear infinite;
3744
+}
3745
+.shithub-org-import-actions {
3746
+  display: flex;
3747
+  gap: 0.5rem;
3748
+  margin-top: 1rem;
3749
+}
3750
+@keyframes shithub-spin {
3751
+  to { transform: rotate(360deg); }
3752
+}
35953753
 .shithub-org-danger-box {
35963754
   margin-top: 0.75rem;
35973755
 }
internal/web/templates/orgs/import_progress.htmladded
@@ -0,0 +1,62 @@
1
+{{ define "page" -}}
2
+<section class="shithub-org-settings-page">
3
+  {{ if not .IsTerminal }}<script>setTimeout(function(){ window.location.reload(); }, 3000);</script>{{ end }}
4
+  <header class="shithub-org-pagehead shithub-org-settings-pagehead">
5
+    <div class="shithub-org-pagehead-inner">
6
+      <a class="shithub-org-pagehead-title" href="/{{ .Org.Slug }}">
7
+        <img src="{{ .AvatarURL }}" alt="" width="30" height="30">
8
+        <span>{{ if .Org.DisplayName }}{{ .Org.DisplayName }}{{ else }}{{ .Org.Slug }}{{ end }}</span>
9
+      </a>
10
+    </div>
11
+  </header>
12
+
13
+  <main class="shithub-org-import-progress">
14
+    <div class="shithub-org-import-progress-head">
15
+      <div>
16
+        <p class="shithub-muted">github.com/{{ .Progress.SourceOrg }}</p>
17
+        <h1>Import repositories</h1>
18
+      </div>
19
+      <span class="shithub-org-import-status is-{{ .Progress.Status }}">{{ .Progress.Status }}</span>
20
+    </div>
21
+
22
+    {{ if .Progress.LastError.Valid }}
23
+    <p class="shithub-flash shithub-flash-error" role="alert">{{ .Progress.LastError.String }}</p>
24
+    {{ end }}
25
+
26
+    <div class="shithub-org-import-counts" aria-label="Import progress">
27
+      <div><strong>{{ .Progress.ImportedCount }}</strong><span>Imported</span></div>
28
+      <div><strong>{{ .Progress.ImportingCount }}</strong><span>Importing</span></div>
29
+      <div><strong>{{ .Progress.QueuedCount }}</strong><span>Queued</span></div>
30
+      <div><strong>{{ .Progress.SkippedCount }}</strong><span>Skipped</span></div>
31
+      <div><strong>{{ .Progress.FailedCount }}</strong><span>Failed</span></div>
32
+    </div>
33
+
34
+    <div class="shithub-org-import-list">
35
+      {{ range .Items }}
36
+      <div class="shithub-org-import-row">
37
+        <div>
38
+          <div class="shithub-org-import-row-title">
39
+            {{ if eq .Status "imported" }}
40
+              <a href="/{{ $.Org.Slug }}/{{ .TargetName }}">{{ .SourceName }}</a>
41
+            {{ else }}
42
+              <span>{{ .SourceName }}</span>
43
+            {{ end }}
44
+            <span class="shithub-repo-visibility">{{ .TargetVisibility }}</span>
45
+          </div>
46
+          <p>{{ if .Description }}{{ .Description }}{{ else }}{{ .SourceFullName }}{{ end }}</p>
47
+          {{ if .LastError.Valid }}<p class="shithub-org-import-error">{{ .LastError.String }}</p>{{ end }}
48
+        </div>
49
+        <span class="shithub-org-import-status is-{{ .Status }}">{{ if eq .Status "queued" }}{{ octicon "history" }}{{ else if eq .Status "importing" }}<span class="shithub-spinner" aria-hidden="true"></span>{{ else if eq .Status "imported" }}{{ octicon "check" }}{{ else if eq .Status "skipped" }}{{ octicon "dash" }}{{ else }}{{ octicon "x" }}{{ end }} {{ .Status }}</span>
50
+      </div>
51
+      {{ else }}
52
+      <p class="shithub-empty-note">Repository discovery has not finished yet.</p>
53
+      {{ end }}
54
+    </div>
55
+
56
+    <p class="shithub-org-import-actions">
57
+      <a class="shithub-button" href="/organizations/{{ .Org.Slug }}/settings/import">Back to imports</a>
58
+      <a class="shithub-button" href="/orgs/{{ .Org.Slug }}/repositories">View repositories</a>
59
+    </p>
60
+  </main>
61
+</section>
62
+{{- end }}
internal/web/templates/orgs/new.htmlmodified
@@ -9,16 +9,29 @@
99
       <input type="text" name="slug" required
1010
              pattern="[a-z0-9](?:[a-z0-9-]{0,37}[a-z0-9])?"
1111
              title="lowercase letters, digits, and hyphens; 1–39 chars; cannot start or end with a hyphen"
12
-             value="{{ .Slug }}" autofocus>
12
+             value="{{ .Form.Slug }}" autofocus>
1313
     </label>
1414
     <label>
1515
       <span>Display name</span>
16
-      <input type="text" name="display_name">
16
+      <input type="text" name="display_name" value="{{ .Form.DisplayName }}">
1717
     </label>
1818
     <label>
1919
       <span>Billing email (optional)</span>
20
-      <input type="email" name="billing_email" autocomplete="email">
20
+      <input type="email" name="billing_email" autocomplete="email" value="{{ .Form.BillingEmail }}">
2121
     </label>
22
+    <fieldset class="shithub-org-import-create">
23
+      <legend>Import repositories from GitHub</legend>
24
+      <p>Optionally mirror repositories from a GitHub organization after this organization is created.</p>
25
+      <label>
26
+        <span>GitHub organization</span>
27
+        <input type="text" name="github_org" placeholder="github-org" value="{{ .Form.GitHubOrg }}" autocomplete="off">
28
+      </label>
29
+      <label>
30
+        <span>GitHub token (optional)</span>
31
+        <input type="password" name="github_token" autocomplete="off" placeholder="Required for private repositories">
32
+      </label>
33
+      <p class="shithub-auth-aside">Without a token, shithub imports public repositories only. With a token, repositories visible to that token are imported and private repositories stay private.</p>
34
+    </fieldset>
2235
     <button type="submit" class="shithub-button shithub-button-primary">Create organization</button>
2336
   </form>
2437
   <p class="shithub-auth-aside">
internal/web/templates/orgs/settings_import.htmladded
@@ -0,0 +1,78 @@
1
+{{ define "page" -}}
2
+<section class="shithub-org-settings-page">
3
+  <header class="shithub-org-pagehead shithub-org-settings-pagehead">
4
+    <div class="shithub-org-pagehead-inner">
5
+      <a class="shithub-org-pagehead-title" href="/{{ .Org.Slug }}">
6
+        <img src="{{ .AvatarURL }}" alt="" width="30" height="30">
7
+        <span>{{ if .Org.DisplayName }}{{ .Org.DisplayName }}{{ else }}{{ .Org.Slug }}{{ end }}</span>
8
+      </a>
9
+    </div>
10
+  </header>
11
+
12
+  <div class="shithub-org-settings-layout">
13
+    <aside class="shithub-org-settings-sidebar" aria-label="Organization settings">
14
+      <h1>Settings</h1>
15
+      <nav class="shithub-org-settings-menu">
16
+        <a href="/organizations/{{ .Org.Slug }}/settings/profile">{{ octicon "organization" }} General</a>
17
+        <a class="is-selected" href="/organizations/{{ .Org.Slug }}/settings/import" aria-current="page">{{ octicon "repo" }} Import repositories</a>
18
+        <span aria-disabled="true">{{ octicon "people" }} People</span>
19
+        <span aria-disabled="true">{{ octicon "repo" }} Repository defaults</span>
20
+        <span aria-disabled="true">{{ octicon "lock" }} Member privileges</span>
21
+        <h2>Code, planning, and automation</h2>
22
+        <span aria-disabled="true">{{ octicon "play" }} Actions</span>
23
+        <span aria-disabled="true">{{ octicon "table" }} Projects</span>
24
+        <span aria-disabled="true">{{ octicon "package" }} Packages</span>
25
+        <h2>Security</h2>
26
+        <span aria-disabled="true">{{ octicon "shield-check" }} Code security</span>
27
+        <span aria-disabled="true">{{ octicon "globe" }} Domains</span>
28
+        <h2>Access</h2>
29
+        <span aria-disabled="true">{{ octicon "gear" }} Integrations</span>
30
+        <span aria-disabled="true">{{ octicon "history" }} Audit log</span>
31
+      </nav>
32
+    </aside>
33
+
34
+    <div class="shithub-org-settings-main">
35
+      {{ with .Error }}<p class="shithub-flash shithub-flash-error" role="alert">{{ . }}</p>{{ end }}
36
+      {{ with .Notice }}<p class="shithub-flash shithub-flash-success" role="status">{{ . }}</p>{{ end }}
37
+
38
+      <div class="Subhead">
39
+        <h2 class="Subhead-heading Subhead-heading--large">Import repositories</h2>
40
+      </div>
41
+
42
+      <form method="POST" action="/organizations/{{ .Org.Slug }}/settings/import" class="shithub-org-settings-form shithub-org-import-form" novalidate>
43
+        <input type="hidden" name="csrf_token" value="{{ .CSRFToken }}">
44
+        <label>
45
+          <span>GitHub organization</span>
46
+          <input type="text" name="github_org" value="{{ .Form.GitHubOrg }}" placeholder="github-org" required autocomplete="off">
47
+        </label>
48
+        <label>
49
+          <span>GitHub token <small>(optional)</small></span>
50
+          <input type="password" name="github_token" placeholder="Required for private repositories" autocomplete="off">
51
+        </label>
52
+        {{ if not .SecretBoxOK }}
53
+        <p class="shithub-flash shithub-flash-error" role="status">Token-backed imports are disabled until the server secret key is configured.</p>
54
+        {{ end }}
55
+        <p class="shithub-empty-note">Public imports need no token. Token-backed imports include private repositories visible to that token and keep them private on shithub.</p>
56
+        <button type="submit" class="shithub-button shithub-button-primary">Start import</button>
57
+      </form>
58
+
59
+      <section class="shithub-org-settings-section" aria-labelledby="org-import-history-heading">
60
+        <div class="Subhead Subhead--spacious border-bottom-0 mb-0 tmp-mb-0">
61
+          <h2 id="org-import-history-heading" class="Subhead-heading">Recent imports</h2>
62
+        </div>
63
+        <div class="shithub-org-import-history">
64
+          {{ range .Imports }}
65
+          <a href="/organizations/{{ $.Org.Slug }}/imports/{{ .ID }}" class="shithub-org-import-history-row">
66
+            <span>{{ .SourceOrg }}</span>
67
+            <span class="shithub-org-import-status is-{{ .Status }}">{{ .Status }}</span>
68
+            <span>{{ relativeTime .CreatedAt.Time }}</span>
69
+          </a>
70
+          {{ else }}
71
+          <p class="shithub-empty-note">No imports have been started for this organization.</p>
72
+          {{ end }}
73
+        </div>
74
+      </section>
75
+    </div>
76
+  </div>
77
+</section>
78
+{{- end }}
internal/web/templates/orgs/settings_profile.htmlmodified
@@ -14,6 +14,7 @@
1414
       <h1>Settings</h1>
1515
       <nav class="shithub-org-settings-menu">
1616
         <a class="is-selected" href="/organizations/{{ .Org.Slug }}/settings/profile" aria-current="page">{{ octicon "organization" }} General</a>
17
+        <a href="/organizations/{{ .Org.Slug }}/settings/import">{{ octicon "repo" }} Import repositories</a>
1718
         <span aria-disabled="true">{{ octicon "people" }} People</span>
1819
         <span aria-disabled="true">{{ octicon "repo" }} Repository defaults</span>
1920
         <span aria-disabled="true">{{ octicon "lock" }} Member privileges</span>