# Operations Guide — deploys, OAuth login & token setup

**Last updated:** 2026-09-23 (full rewrite — the previous version was an
April-2026 Fly.io/Vercel runbook; Fly was decommissioned 2026-09-09 and the
dashboard moved to forge-control. Closes this file's findings from the
2026-09-23 docs audit: pervasive stale Fly/Vercel content, the missing
break-glass secrets inventory, and the absent JWT-drift triage entry.)

**Where things actually run (verified 2026-09-23):**

| Piece | Location |
|---|---|
| Registry API | forge-control **app-3** (`116.203.97.105`), container `packr-registry-registry-1-1`, loopback `:4873` behind host Caddy → `api.packr.blueforge.studio` |
| Dashboard/site | same host, container `packr-registry-site-1-1`, loopback `:3555` → `packr.blueforge.studio` (Next.js 16, live since 2026-08-20) |
| Postgres | `postgres-shared` on the fleet network, DB `app_packr_registry`, **SSL disabled** (`?sslmode=disable` is mandatory) |
| Blobs | fleet MinIO (`app-packr-registry`); Backblaze `packr-registry-cache` is the **nightly Postgres backup target**, not a blob store (`docs/backups.md`) |
| Secrets | `forge vault` (project `packr-registry`) + the host env files below — **not** Fly |

Related runbooks: [`registry-outage-runbook.md`](registry-outage-runbook.md) (incident response),
[`backups.md`](backups.md), [`deployment-architecture.md`](deployment-architecture.md),
[`docs/05-cli.md`](05-cli.md) (CLI reference), [`docs/08-cli-auth.md`](08-cli-auth.md) (login/troubleshooting).

---

## Deploys

Two paths exist. Both end at `docker compose` on app-3; they differ in who
writes the compose/env files and which env file the compose reads.

### Normal path (forge-control worker)

Push to **`main`** on forge-git → Gitea webhook → build-worker builds the
image on app-3, writes `/opt/packr-registry/registry.docker-compose.yml` +
`site.docker-compose.yml`, and `docker compose up -d`. Feature-branch pushes
are cancelled by design; pushes that touch only non-code paths (docs,
`changelog.d/`) are also cancelled — neither rebuilds production.

```bash
# Watch a deploy (forge CLI):
forge app deploys packr-registry          # records + error messages
forge app status packr-registry           # container, image, health, domains
forge deployments show <full-uuid>        # stage-by-stage lifecycle
```

**Known landmines (tracked in forge-control):**
- The worker's env render has twice diverged from vault/files — it dropped
  `?sslmode=disable` from `DATABASE_URL` (registry crash-loops with
  `pq: SSL is not enabled on the server`, rolls back) and wrote a site
  `INTERNAL_API_SECRET` that didn't match the registry's. **After any worker
  deploy, verify** `/-/ping`, `health?strict=1`, and a `packr-cli doctor`
  before trusting it. (forge-control#1502)
- Rollback messages name the *deployment record's* commit, not necessarily the
  image that is actually serving — trust `/-/ping`'s `commit`, not the message.
- A failed registry deploy rolls back the registry only; the site half may
  have shipped. Check both containers.

### Manual fallback (`scripts/deploy-fleet.sh`)

Use when the worker path is broken. Builds the image **on the host** from the
exact commit, installs `deploy/fleet/registry.docker-compose.yml`, and
verifies `/-/ping` on loopback + publicly.

```bash
scripts/deploy-fleet.sh            # VERSION from HEAD's tag, else "dev"
scripts/deploy-fleet.sh v0.54.0    # explicit (must match the VERSION file)
```

- Refuses a dirty tree (the image is built from `git archive HEAD`).
- **Never writes env files** — runtime env stays whatever is on the host.
- Ships the deploy-artefact scripts (`backup-db.sh`, `regression-recovery.sh`,
  `restore-db.sh`, …) to `/opt/packr-registry/` so `/etc/cron.d/packr-backup`
  keeps working.
- ⚠️ The site container appears as an "orphan" in its output — it is a
  **separate compose file under the same project**. Never add
  `--remove-orphans`; it would delete the site.
- Rollback: `ssh root@app-3 'cd /opt/packr-registry && PACKR_IMAGE_TAG=<old-tag>
  docker compose -p packr-registry -f registry.docker-compose.yml up -d'`
  (previous image tags are kept; e.g. `packr-registry:v0.52.0`).

### Releases

Tagged `v*` pushes run `release.yml` → fold `changelog.d/` fragments → tag →
`forge app-deploy`. The step list lives in `CONTRIBUTING.md`; fragment
conventions in `changelog.d/README.md`.

---

## Secrets & env files

The registry reads env from **two files that must stay in sync** — which one
is live depends on which compose is installed:

| File | Read by | Notes |
|---|---|---|
| `/opt/packr-registry/registry.secrets.env` | **worker path** (its compose `env_file:`) | rewritten by deploys |
| `/opt/packr-registry/registry.env` | **manual path** (`deploy/fleet/registry.docker-compose.yml`) | never written by scripts; hand-maintained |
| `/opt/packr-registry/site.secrets.env` | site compose | must pair with the registry's `INTERNAL_API_SECRET` |

**Rotating a secret therefore means updating BOTH registry files** (and the
site's, when it is `INTERNAL_API_SECRET`), then **recreating** the container —
`docker restart` reuses the captured env and silently keeps the old value.

Canonical values live in **forge vault** (`forge vault get --project
packr-registry --key <KEY>`): `JWT_SECRET`, `SESSION_SECRET`,
`INTERNAL_API_SECRET`, `DATABASE_URL`, plus `SUPER_ADMIN_TOKEN` and
`OAUTH_GITHUB_*` (added 2026-09-23). Hash-compare before and after; never
print values.

Current break-glass inventory (as of 2026-09-23, hash-verified):

| Secret | In vault | In both registry env files | Consequence if missing |
|---|---|---|---|
| `JWT_SECRET` | ✓ | ✓ | every token minted before the last change 401s on admin routes while still *publishing* (row fallback) — see JWT-drift below |
| `SESSION_SECRET` | ✓ | ✓ | dashboard sessions drop every boot |
| `INTERNAL_API_SECRET` | ✓ | ✓ | site→registry calls fail (device callback, OAuth session mint) — **must equal the site's copy** |
| `SUPER_ADMIN_TOKEN` | ✓ | ✓ | every `X-Super-Admin-Token` route fails **closed**, even with the correct local secret |
| `OAUTH_GITHUB_CLIENT_ID/SECRET` | ✓ | ✓ | device flow: `unknown_provider` |
| `DATABASE_URL` | ✓ | ✓ (needs `?sslmode=disable`) | registry crash-loops: `pq: SSL is not enabled on the server` |

Vault is the source of truth; the env files are the delivery. If a deploy
writes a value that disagrees with vault, the deploy is wrong — that is
forge-control#1502's bug class, not a reason to "fix" vault.

---

## OAuth login setup

### Prerequisites

- [x] Registry at `https://api.packr.blueforge.studio` (app-3)
- [x] GitHub OAuth App configured (client ID: `Ov23ligJE4NpN7NysXzL`)
- [x] `OAUTH_GITHUB_CLIENT_ID` / `OAUTH_GITHUB_CLIENT_SECRET` set in **both**
      registry env files (device flow reads them from the registry process)
- [x] `OAUTH_REDIRECT_BASE=https://packr.blueforge.studio`
- [x] GitHub OAuth App callback URL =
      `https://packr.blueforge.studio/api/auth/callback/github`
- [x] Dashboard live at `https://packr.blueforge.studio` (forge-control, app-3)
- [x] `INTERNAL_API_SECRET` **equal** on registry and site (see Secrets above)
- [x] `packr-cli` built at `bin/packr-cli`

> The GitHub OAuth **client secret** was written out in full in this repo's
> history, so treat any value that was ever committed as burned: rotate it in
> the OAuth App and update both env files + vault. The client *ID* is not a
> secret and needs no rotation.

### First login

```bash
./bin/packr-cli login --provider github --registry https://api.packr.blueforge.studio
```

1. CLI prints a verification URL + code (`ABCD-EFGH`)
2. Browser opens `https://packr.blueforge.studio/cli/auth` (live since 2026-08-20)
3. Enter the code → authorize on GitHub's consent screen
4. Browser: "CLI Authorized." → CLI: `Logged in as <name> (maintainer) via github — expires <date>`

Creates `~/.packr/credentials.json` (mode 0600) and updates `~/.npmrc`.

> **Headless/CI:** no browser to open? Use the password fallback —
> `packr-cli login --password` — see `docs/08-cli-auth.md` § "Password login
> escape hatch". On a live registry, `whoami` reads the *local* file and can
> claim a dead token is fine; verify server-side with `packr-cli doctor` or
> `packr-cli token verify`.

### Verify install path

```bash
pnpm install @blueforge-studio/some-package   # resolves via the token in ~/.npmrc
```

---

## Creating scoped CI tokens

`packr-cli token create` uses the **admin API** — it needs a working login
(or the operator route, below), plus `--registry` or `PACKR_REGISTRY`.

```bash
export PACKR_REGISTRY=https://api.packr.blueforge.studio

# Read-only (installs)
./bin/packr-cli token create ci-read --role ci-readonly

# Publish, scoped to the namespace (releases)
./bin/packr-cli token create ci-publish --role ci-publish --scope @blueforge-studio \
  --output-file /tmp/tok        # never let the JWT hit scrollback/CI logs
```

### The operator route: minting when your login is the broken thing

After a `JWT_SECRET` change, your stored session may 401 on every admin route
while publish still works — and `token create` goes through the admin API. The
break-glass path needs **only** the super-admin secret, no session:

```bash
export PACKR_SUPER_ADMIN_TOKEN=<from ~/.forge-secrets/packr-super-admin.token or vault>
./bin/packr-cli token create replacement --role admin --scope @blueforge-studio \
  --as-user web:<id> --output-file /tmp/tok
```

`--as-user` posts to `POST /api/v1/admin/tokens/mint` with
`X-Super-Admin-Token` and **no** `Authorization` header (the audit row names
the secret, not a stale session). Requires `SUPER_ADMIN_TOKEN` to be set
*server-side* — see the Secrets table above. Retiring a credential you can't
authenticate as is the mirror operation:
`packr-cli token revoke <id> --super-admin-token <tok>`.

### JWT drift — "publish works, every admin call 401s"

The classic post-rotation symptom, and the reason the operator route exists:

| Path | Check | Result after a `JWT_SECRET` change |
|---|---|---|
| publish / install | sha256 row lookup | still works |
| `/api/v1/admin/*` | JWT signature vs current secret | `AUTH_INVALID`, despite `exp` months away |

**Fix:** `packr-cli login --password` (fresh signature) or the operator mint
above; then retire the old credential with `packr-cli token revoke <id>
--super-admin-token …`. Full detail: `docs/08-cli-auth.md` § troubleshooting.

### GitHub org secrets — pitfalls

**Use stdin with `printf`**, not `--body` (avoids line-wrap corruption):

```bash
unset GITHUB_TOKEN                 # otherwise gh ignores keychain auth
gh auth refresh -h github.com -s admin:org
printf '%s' '<token>' | gh secret set PACKR_TOKEN --org blueforge-studio
printf '%s' '<token>' | gh secret set PACKR_TOKEN_PUBLISH --org blueforge-studio
```

**Repo-level secrets shadow org-level ones.** A leftover repo `PACKR_TOKEN`
silently wins:

```bash
cd /path/to/repo && gh secret list          # repo-level
gh secret list --org blueforge-studio       # org-level
gh secret delete PACKR_TOKEN               # remove the shadowing duplicate
```

### `PACKR_TOKEN` vs `PACKR_TOKEN_PUBLISH`

The original design was: use `PACKR_TOKEN` (read-only) for `pnpm install`,
`PACKR_TOKEN_PUBLISH` (publish + scoped) for releases.

**It does not survive a single root `.npmrc`:** both install and publish read
the same `_authToken` line, and `NODE_AUTH_TOKEN` loses to it. Production runs
**one** `ci-publish` token scoped to `@blueforge-studio` (the scope is the real
authorization boundary), set as both secrets:

```bash
printf '%s' '<ci-publish-token>' | gh secret set PACKR_TOKEN --org blueforge-studio
printf '%s' '<ci-publish-token>' | gh secret set PACKR_TOKEN_PUBLISH --org blueforge-studio
```

Real split-in-two needs per-command temp `.npmrc`s — documented, not yet the
default.

---

## Initializing consumer repos

```bash
./bin/packr-cli init --scope @blueforge-studio --registry https://api.packr.blueforge.studio
```

Writes `.packr.json` (registry + scope) and an `.npmrc` scope line with a
`${PACKR_TOKEN}` reference (safe to commit). Target every repo consuming
`@blueforge-studio/*`: `blueforge-org`, `mailstack-os`, and siblings.

---

## Dashboard/site environment

The site is a forge-control app, not a Vercel project. Its env lives in
`/opt/packr-registry/site.secrets.env` on app-3 (same host as the registry);
`blueforge.config.json#apps.site.secrets` declares what belongs there.
Key entries:

```
GITHUB_OAUTH_CLIENT_ID / GITHUB_OAUTH_CLIENT_SECRET   # web sign-in (rotate if burned — see above)
INTERNAL_API_SECRET                                   # MUST equal the registry's value
SESSION_SECRET                                        # site sessions
STRIPE_*                                              # billing
```

After editing: recreate the site container (`docker compose -f
site.docker-compose.yml up -d site-1`) — restart does not re-read env.

> ⚠️ The worker has written a site `INTERNAL_API_SECRET` that didn't match
> the registry's (forge-control#1502). If dashboard login or device-flow
> callbacks start failing after a deploy, hash-compare the two files first —
> this exact drift has now been repaired by hand twice.

---

## Troubleshooting

### Device flow / OAuth callback

1. Callback URL in the OAuth App must be
   `https://packr.blueforge.studio/api/auth/callback/github` (not `/cli/auth/callback`)
2. `OAUTH_REDIRECT_BASE=https://packr.blueforge.studio` in the **registry** env
3. `INTERNAL_API_SECRET` equal on registry and site (hash-compare — never print)
4. `OAUTH_GITHUB_CLIENT_ID/SECRET` present in the registry env
   (`unknown_provider` from `/-/v1/device/authorize` = they are missing)
5. Site route live: `curl -I https://packr.blueforge.studio/cli/auth` → 200

### Admin calls 401 while publish works

JWT drift — see [above](#jwt-drift--publish-works-every-admin-call-401s).
Do not chase the token's `exp`; it lies. `packr-cli token verify` reports the
signature and stored-row paths separately.

### Publish fails with 403 in CI

1. Repo-level `PACKR_TOKEN` shadowing the org-level secret (see pitfalls above)
2. Decode the JWT actually in the secret and check `permissions`/`scopes` —
   `echo "$TOKEN" | cut -d. -f2 | base64 -d 2>/dev/null | python3 -m json.tool`
3. Confirm the token's `scopes` actually cover the namespace you are about to
   publish into. Publishing a brand-new `@scope/*` name needs `publish` plus
   a matching scope — **not** `admin` (README § No `admin` role is needed to
   publish a new package)
4. Registry logs: `ssh root@app-3 'docker logs packr-registry-registry-1-1
   --tail 100 | grep publish:'`

### Token expired / session keeps dropping

```bash
packr-cli login --provider github --registry https://api.packr.blueforge.studio
```

If sessions drop on **every** registry boot: `SESSION_SECRET` (and
`JWT_SECRET`) must be set persistently — unset, the server generates a random
one per boot and silently logs everyone out while `whoami` still reports the
local file as valid.

### Keychain storage (optional)

```bash
./bin/packr-cli config set credential-store keychain
```

Service `packr-cli`, account = registry URL (`docs/08-cli-auth.md` § credential
stores). Headless/SSH sessions should stay on `file`.

### Logs & health

```bash
ssh root@app-3 'docker logs packr-registry-registry-1-1 --tail 100'
curl -s https://api.packr.blueforge.studio/health?strict=1   # strict: blob+db must be ok
packr-cli doctor                                              # full local→server sweep
```

`/health` (bare) answers 200 even when degraded — a liveness probe must not
restart a container over a bad bucket credential. Use `?strict=1` or `doctor`
when the question is "is it *actually* serving".

---

## Token role reference

| Role | Permissions | Use for |
|------|------------|---------|
| `ci-readonly` | `read` | CI install jobs, local dev installs |
| `ci-publish` | `read`, `publish` | CI publish workflows |
| `maintainer` | `read`, `publish`, `unpublish` | Developer accounts (OAuth login default) |
| `admin` | all | Registry administration — token and user management. **Not** needed to publish a new package name; that needs `publish` + a matching scope |

Per-plan token ceilings (default `free` = 20 active) gate **every** mint path;
manage with `packr-cli token-limit get|set|clear web:<id>`.

---

## Lifting a rate limit (super-admin escape hatch)

The login limiter is 10/min per IP. A `packr-cli login` device flow that polls
can spend that, and the operator is then locked out of the one action that
would let them fix anything — including whatever made them retry. Observed
2026-09-02.

```bash
curl -X POST https://api.packr.blueforge.studio/api/v1/admin/maintenance/rate-limit/reset \
  -H "X-Super-Admin-Token: $SUPER_ADMIN_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"limiter":"login","key":"203.0.113.7"}'
# X-Internal-Secret: <INTERNAL_API_SECRET> is accepted by the same handler
```

- `limiter` — `login`, `publish`, `admin`, `search`, or `all`. An unknown name
  is a 400, not a silent no-op: "cleared nothing" and "worked" must not look
  the same to someone acting in an incident.
- `key` — the client IP to clear. Omit it to clear every bucket on that limiter.

Gated by `X-Super-Admin-Token` or `X-Internal-Secret`, like the other
`maintenance/*` routes. Deliberately **not** behind a session: an operator
locked out by the login limiter has no session to present, and gating it behind
one would make it unreachable exactly when it is needed. Both secrets must be
set **server-side** (see the Secrets table) — `secret_compare.go` fails closed,
so a correct local value still 401s against an unprovisioned registry.

**Budgeted at 3 per rolling 24h** (`RATE_LIMIT_RESETS_PER_DAY`). A reset switch
with no budget is not a safety valve, it is the removal of the limit — anyone
holding the token could clear the login limiter on every request and brute-force
behind it. The window slides, so a budget spent at 23:00 refills through the
following day rather than at midnight.

Every reset is written to the audit log as `ratelimit.reset` with the limiter,
key and what was cleared. The budget keeps those entries rare enough to read.

The response reports what is left:

```json
{"cleared":{"login":1},"resetsRemaining":2,"budgetWindowHours":24}
```

> If sessions are being invalidated rather than rate-limited, the cause is
> different: `SESSION_SECRET` unset makes the server generate a random one at
> boot, so every restart silently logs everyone out while the CLI still reports
> a valid local session. Set `SESSION_SECRET` and `JWT_SECRET` persistently.

Full audit surface (`scope=all` for super-admin rows, including operator mints):
`GET /api/v1/admin/audit` — `docs/API.md` § Audit Log.