From 73f39da54a9a58614cdb0460be7646e15425e1fa Mon Sep 17 00:00:00 2001 From: Abdulmuizz Hamzat <103611443+buildwithhamzat@users.noreply.github.com> Date: Sat, 1 Aug 2026 21:07:31 +0100 Subject: [PATCH] feat(workflows): add Railway preview environment and teardown workflows (#766) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(workflows): add Railway preview environment and teardown workflows * fix(workflows): enhance security by binding branch names to env variables in Railway workflows * fix(workflows): ensure preview URLs are live by blocking on service healthchecks during deployment * temp: auto-run railway-preview.yml on push for demo recording workflow_dispatch requires the file on the default branch before "Run workflow" is available; add a scoped push trigger + input fallbacks so this branch can produce a real run to record. Revert before merge. * fix(workflows): pass --workspace to railway link for non-interactive CI Railway CLI now requires --workspace when running `railway link` non-interactively; without it the preview workflow fails at the link step. Defaults to the "cio projects" workspace the staging project lives in, overridable via a RAILWAY_WORKSPACE repo variable. * fix(workflows): read RAILWAY_PROJECT_ID from secrets, not vars It's configured as a repository secret (has been for weeks), but the workflow read it via the vars context, which is a separate namespace and always resolved empty -> railway link failed with Project "" not found. Read it via secrets to match how it's actually stored. * fix(workflows): pass --workspace to environment new/delete, stop masking failures railway environment new needed --workspace in non-interactive mode too (same as railway link), but its failure was unconditionally swallowed by `|| echo`, so the workflow proceeded to "Link" as if the preview environment existed when it never got created. Add --workspace to both environment new and environment delete, and only swallow the "already exists" case -- any other failure now fails the run where it actually happened instead of surfacing as a confusing error two steps later. * fix(workflows): stop passing --project/--workspace to railway environment new/delete Verified against the actual CLI (railway environment new --help / delete --help): those subcommands take NO --project/--workspace/--environment flags at all, and were silently rejecting them, which is why environment creation kept failing and getting masked by the fallback echo. Only `railway link` accepts --project/--workspace/--environment, so link to the base project first, then create/delete environments by name, then re-link to switch into the new one for deploy. * fix(workflows): force path-style addressing for mc against Railway's MinIO domain mc alias set defaults to --path auto, which can pick virtual-host-style bucket addressing (bucket.host/key). Railway's edge only routes the exact configured domain, not wildcard bucket subdomains, so mb then fails with a misleading "specified bucket does not exist" even though the alias/auth check (a GET) succeeds. Force --path on, matching the app's own OBJECT_STORAGE_FORCE_PATH_STYLE=true for this same endpoint. * chore: retrigger preview workflow after fixing cio-minio start command staging's cio-minio now has the correct Custom Start Command, and the stale pr-chore-staging-setup preview env (duplicated before that fix) has been deleted. Empty commit to trigger a fresh deploy run. * fix(workflows): generate cio-minio's public domain in the preview env Confirmed staging's cio-minio has its own domain, but environment duplication doesn't carry it to the new preview environment, so RAILWAY_PUBLIC_DOMAIN came back empty there and produced an invalid "https://" alias URL. Generate one in CI, same as the existing cio-dashboard domain step, pinned to port 9000 since the minio/minio image exposes both the S3 API (9000) and console (9001). * chore(workflows): revert temp push-trigger testing scaffold, trim comments Remove the push trigger and || fallback expressions added solely to get a run before this file exists on main (workflow_dispatch requires that for the "Run workflow" UI). Back to workflow_dispatch-only, with comments trimmed from a debugging narrative down to the durable facts future readers need. All real fixes found during testing (link --workspace, secrets vs vars, environment new/delete flag handling, mc --path on, cio-minio domain generation) are unchanged. * fix(workflows): fix teardown link/secrets bugs, harden domain reads railway-preview-teardown.yml had the same class of bugs already found and fixed in railway-preview.yml during testing: - RAILWAY_PROJECT_ID read from vars.* while the deploy workflow (and the actual GitHub config) uses secrets.* — always resolved empty. - environment delete was called without linking first, and passed an unsupported --project flag; the resulting CLI error was swallowed by `|| echo`, so teardown silently no-opped on every run and every preview environment leaked indefinitely. Also hardens railway-preview.yml against a repeat of the cio-minio domain bug: two other "generate domain, then read it back" spots (cio-minio bucket seeding, cio-dashboard's final URL report) had the same unguarded `|| true` swallow with no check that the resulting host is non-empty. Both now fail loudly instead of silently proceeding — the dashboard one matters most since it's the final step and would otherwise report success with a broken link. Resolves the now-verified SPIKE comment on environment duplication, and trims a few comments that restated code instead of explaining non-obvious behavior. * fix(workflows): address CodeRabbit findings on preview lifecycle - Unify concurrency groups between deploy and teardown (was railway-preview- vs railway-preview-teardown-, so they never coordinated). A PR-close teardown and a manual deploy for the same branch could otherwise race, letting a deploy recreate an environment right after teardown deleted it. - Pin the Railway CLI to 5.30.1 (the version verified against all session) in both workflows, instead of an unpinned `npm install -g` that could silently pick up different undocumented CLI behavior later, same class of bug this session hit repeatedly. - Append a short hash of the original branch string to the derived environment name in both workflows. The slug alone is lossy (/, _, and . all collapse to the same separator), so e.g. "foo/bar" and "foo_bar" previously collided onto the same environment name. * fix(workflows): scope teardown job to same-repo PRs only The header comment already documented that this only applies to same-repo branches (secrets are withheld from fork PR runs), but nothing enforced it -- the job ran on every closed PR including forks, where the empty RAILWAY_API_TOKEN/RAILWAY_PROJECT_ID made railway link fail with an auth error and show a red run for a PR that never had a preview environment to clean up. * docs: migrate RAILWAY_PREVIEW.md to PREVIEW_ENV.md Replace docker/docs/RAILWAY_PREVIEW.md with PREVIEW_ENV.md at the repo root, restructured per PR review feedback around the two things a reader actually needs: how to deploy to staging, and how to deploy a PR to a preview environment. Drops the one-time Railway project setup section (not needed on this page, per review). --------- Co-authored-by: Rotimi Best --- .../workflows/railway-preview-teardown.yml | 61 +++++ .github/workflows/railway-preview.yml | 213 ++++++++++++++++++ PREVIEW_ENV.md | 73 ++++++ 3 files changed, 347 insertions(+) create mode 100644 .github/workflows/railway-preview-teardown.yml create mode 100644 .github/workflows/railway-preview.yml create mode 100644 PREVIEW_ENV.md diff --git a/.github/workflows/railway-preview-teardown.yml b/.github/workflows/railway-preview-teardown.yml new file mode 100644 index 00000000..92a6b690 --- /dev/null +++ b/.github/workflows/railway-preview-teardown.yml @@ -0,0 +1,61 @@ +name: Railway Preview Teardown + +# Auto-delete a preview environment created by railway-preview.yml when its PR closes, +# so manually-spun previews don't linger and accrue Railway cost. (Native Railway PR +# environments clean themselves up; this covers the pr- envs made by the manual +# workflow.) Deletion is best-effort — if no matching env exists, the run is a no-op. +# +# Uses the same RAILWAY_API_TOKEN + RAILWAY_PROJECT_ID secrets as the deploy workflow. +# Note: secrets are unavailable to PRs from forks, so this only tears down previews for +# same-repo branches (which is where manual previews come from anyway). + +on: + pull_request: + types: [closed] + +concurrency: + group: railway-preview-${{ github.event.pull_request.head.ref }} + cancel-in-progress: false + +jobs: + teardown: + runs-on: ubuntu-latest + if: github.event.pull_request.head.repo.full_name == github.repository + env: + RAILWAY_API_TOKEN: ${{ secrets.RAILWAY_API_TOKEN }} + RAILWAY_PROJECT_ID: ${{ secrets.RAILWAY_PROJECT_ID }} + RAILWAY_WORKSPACE: ${{ vars.RAILWAY_WORKSPACE || 'cio projects' }} + BASE_ENV: ${{ vars.RAILWAY_BASE_ENV || 'staging' }} + steps: + - name: Install Railway CLI + run: npm install -g @railway/cli@5.30.1 + + - name: Link to base project + # `railway environment delete` takes no --project/--workspace/--environment + # flags — only `railway link` does, so it must run first to establish that + # context (same requirement as railway-preview.yml). + run: | + railway link --project "$RAILWAY_PROJECT_ID" --workspace "$RAILWAY_WORKSPACE" --environment "$BASE_ENV" + + - name: Delete the preview environment + # Mirror the name derivation in railway-preview.yml so the env name matches. + # HEAD_REF is bound via env (not interpolated into the script) so a malicious + # branch name can never be executed as shell — Actions expands ${{ }} as text + # BEFORE bash parses it, so a raw `${{ ...head.ref }}` here is a script-injection hole. + env: + HEAD_REF: ${{ github.event.pull_request.head.ref }} + run: | + slug=$(echo "$HEAD_REF" | tr '[:upper:]/_.' '[:lower:]---' | sed 's/[^a-z0-9-]//g' | sed 's/--*/-/g; s/^-//; s/-$//') + hash=$(echo -n "$HEAD_REF" | sha256sum | cut -c1-8) + env_name="pr-${slug}-${hash}" + echo "Attempting to delete environment: ${env_name}" + if ! output=$(railway environment delete "${env_name}" --yes 2>&1); then + echo "$output" + if echo "$output" | grep -qi "not found"; then + echo "No matching preview environment (or already deleted) — nothing to do." + else + exit 1 + fi + else + echo "$output" + fi diff --git a/.github/workflows/railway-preview.yml b/.github/workflows/railway-preview.yml new file mode 100644 index 00000000..600918f4 --- /dev/null +++ b/.github/workflows/railway-preview.yml @@ -0,0 +1,213 @@ +name: Railway Preview Environment + +# Manually spin up (or tear down) a short-lived Railway preview environment for a +# branch/PR, without waiting for a merge. Railway builds each app service FROM SOURCE +# (docker/Dockerfile.*) so the preview runs the branch's exact code — this is the +# controlled counterpart to Railway's native PR environments. +# +# Prerequisites (one-time): +# - A Railway project (e.g. "classroomio-staging") whose `staging` environment is +# fully configured (Postgres, Redis, cio-minio, cio-api, cio-dashboard, cio-jobs). +# - Repo/Org secret RAILWAY_API_TOKEN -> a Railway ACCOUNT or TEAM token (NOT a +# project token; env-scoped project tokens can't create environments). +# - Repo/Org secret RAILWAY_PROJECT_ID -> the target project's id. +# - Repo/Org variable RAILWAY_WORKSPACE -> the Railway workspace the project lives in +# (default: "cio projects"); required by `railway link` in non-interactive mode. +# - Repo/Org variable RAILWAY_BASE_ENV -> the env to duplicate from (default: staging). +# +# The DB needs no explicit migration step here: the cio-api image self-migrates on boot +# (docker/entrypoint-api.sh -> `pnpm --filter @cio/db db:setup`, which runs drizzle +# migrations + essential seed). The optional `seed` input adds the full demo dataset. + +on: + workflow_dispatch: + inputs: + branch: + description: "Branch (or PR head ref) to deploy into the preview env" + required: true + type: string + action: + description: "deploy = create/update the preview; destroy = delete it" + required: true + type: choice + default: deploy + options: + - deploy + - destroy + seed: + description: "Seed the full demo dataset (admin@test.com etc.) after deploy" + required: false + type: boolean + default: true + +# One run per branch at a time; a newer run supersedes an in-flight one. +concurrency: + group: railway-preview-${{ inputs.branch }} + cancel-in-progress: true + +env: + RAILWAY_API_TOKEN: ${{ secrets.RAILWAY_API_TOKEN }} + RAILWAY_PROJECT_ID: ${{ secrets.RAILWAY_PROJECT_ID }} + RAILWAY_WORKSPACE: ${{ vars.RAILWAY_WORKSPACE || 'cio projects' }} + BASE_ENV: ${{ vars.RAILWAY_BASE_ENV || 'staging' }} + # The three app services that build from source. Postgres/Redis/cio-minio come along + # via environment duplication and need no `railway up`. + APP_SERVICES: "cio-api cio-dashboard cio-jobs" + +jobs: + preview: + runs-on: ubuntu-latest + steps: + - name: Checkout target branch + uses: actions/checkout@v4 + with: + ref: ${{ inputs.branch }} + + - name: Install Railway CLI + run: npm install -g @railway/cli@5.30.1 + + - name: Derive a safe environment name + id: env + # BRANCH is bound via env (not interpolated into the script) so an attacker-supplied + # value can't be executed as shell. "feat/foo_bar" -> "pr-feat-foo-bar-" — the + # slug alone is lossy (e.g. "foo/bar" and "foo_bar" both slug to "foo-bar"), so a + # hash of the original branch string disambiguates collisions. + env: + BRANCH: ${{ inputs.branch }} + run: | + slug=$(echo "$BRANCH" | tr '[:upper:]/_.' '[:lower:]---' | sed 's/[^a-z0-9-]//g' | sed 's/--*/-/g; s/^-//; s/-$//') + hash=$(echo -n "$BRANCH" | sha256sum | cut -c1-8) + echo "name=pr-${slug}-${hash}" >> "$GITHUB_OUTPUT" + + - name: Link to base project + # `railway environment new`/`delete` take no --project/--workspace/--environment + # flags — only `railway link` does, so it must run first to establish that + # context for every other command in this job. + run: | + railway link --project "$RAILWAY_PROJECT_ID" --workspace "$RAILWAY_WORKSPACE" --environment "$BASE_ENV" + + # ---------------------------------------------------------------- destroy + - name: Destroy preview environment + if: ${{ inputs.action == 'destroy' }} + env: + ENV_NAME: ${{ steps.env.outputs.name }} + run: | + railway environment delete "$ENV_NAME" --yes + echo "Deleted environment $ENV_NAME." >> "$GITHUB_STEP_SUMMARY" + + # ---------------------------------------------------------------- deploy + - name: Create preview environment (duplicate base) + if: ${{ inputs.action == 'deploy' }} + # `--duplicate` copies service config and variables but not public domains (see + # domain-generation steps below). Only "already exists" is swallowed here — any + # other failure must surface, not silently continue into a "Link" step that can't + # find an environment that was never created. + env: + ENV_NAME: ${{ steps.env.outputs.name }} + run: | + if ! output=$(railway environment new "$ENV_NAME" --duplicate "$BASE_ENV" 2>&1); then + echo "$output" + if echo "$output" | grep -qi "already exists"; then + echo "Environment may already exist; continuing." + else + exit 1 + fi + else + echo "$output" + fi + + - name: Link project + preview environment + if: ${{ inputs.action == 'deploy' }} + env: + ENV_NAME: ${{ steps.env.outputs.name }} + run: | + railway link --project "$RAILWAY_PROJECT_ID" --workspace "$RAILWAY_WORKSPACE" --environment "$ENV_NAME" + + - name: Pin the Dockerfile per service + if: ${{ inputs.action == 'deploy' }} + # `railway up` uploads local source but does NOT carry the dashboard's "Dockerfile + # Path" build setting, so without this Railway falls back to Railpack and builds + # the whole monorepo as a generic Node app. (Same gotcha documented in the setup guide.) + run: | + railway variables --service cio-api --set "RAILWAY_DOCKERFILE_PATH=docker/Dockerfile.api" + railway variables --service cio-dashboard --set "RAILWAY_DOCKERFILE_PATH=docker/Dockerfile.dashboard" + railway variables --service cio-jobs --set "RAILWAY_DOCKERFILE_PATH=docker/Dockerfile.jobs" + + - name: Seed MinIO buckets (fresh volume starts empty) + if: ${{ inputs.action == 'deploy' }} + # Railway has no minio-init sidecar and a duplicated env's MinIO volume is empty, + # so create the buckets the app expects. Root creds carry over via duplication, + # but the public domain doesn't, so generate one here (same as "Report the preview + # URL" does for cio-dashboard). `--ignore-existing` keeps re-deploys safe. + run: | + # --port 9000: the minio/minio image exposes both 9000 (S3 API) and 9001 + # (console); pin it so domain generation doesn't land on the wrong one. + railway domain --service cio-minio --port 9000 || true + # `cut -d= -f2-` (not `awk -F= '{print $2}'`) so a value containing '=' — common in + # base64/auto-generated secrets like MINIO_ROOT_PASSWORD — isn't silently truncated. + MINIO_HOST=$(railway variables --service cio-minio --kv | grep '^RAILWAY_PUBLIC_DOMAIN=' | cut -d= -f2-) + MINIO_USER=$(railway variables --service cio-minio --kv | grep '^MINIO_ROOT_USER=' | cut -d= -f2-) + MINIO_PASS=$(railway variables --service cio-minio --kv | grep '^MINIO_ROOT_PASSWORD=' | cut -d= -f2-) + if [ -z "$MINIO_HOST" ]; then + echo "::error::cio-minio has no public domain — 'railway domain' may have failed." + exit 1 + fi + # Verify mc against MinIO's published SHA256 before executing it. + curl -sSL https://dl.min.io/client/mc/release/linux-amd64/mc -o mc + curl -sSL https://dl.min.io/client/mc/release/linux-amd64/mc.sha256sum -o mc.sha256sum + echo "$(cut -d' ' -f1 mc.sha256sum) mc" | sha256sum -c - + chmod +x mc + # --path on forces path-style bucket addressing (host/bucket/key), same as the + # app's own OBJECT_STORAGE_FORCE_PATH_STYLE=true — this Railway domain doesn't + # route wildcard bucket subdomains, so virtual-host style (mc's "auto" default) + # fails bucket creation with a misleading "does not exist" error. + ./mc alias set cio "https://${MINIO_HOST}" "$MINIO_USER" "$MINIO_PASS" --path on + ./mc mb cio/videos cio/documents cio/media --ignore-existing + ./mc anonymous set download cio/media + + - name: Deploy each app service from this branch + if: ${{ inputs.action == 'deploy' }} + # `--ci` (not `--detach`) blocks until each service's deploy/healthcheck succeeds, + # so the URL printed later is actually live. Trade-off: builds run sequentially. + run: | + for svc in $APP_SERVICES; do + echo "::group::railway up $svc" + railway up --service "$svc" --ci + echo "::endgroup::" + done + + - name: Seed the full demo dataset (optional) + if: ${{ inputs.action == 'deploy' && inputs.seed }} + # SPIKE: `railway run` injects the service's env vars and runs the command ON THIS + # RUNNER, so it needs a DB URL reachable from GitHub (Railway's public TCP proxy, + # DATABASE_PUBLIC_URL) — NOT the private railway.internal one. If cio-api only + # exposes a private DATABASE_URL, add DATABASE_PUBLIC_URL to this step, or drop + # this step and rely on the api boot-time db:setup (migrations + essential seed). + run: | + corepack enable + railway run --service cio-api -- pnpm --filter @cio/db db:setup:seed || \ + echo "Demo seed failed (likely DB not reachable from runner) — migrations + essential seed still ran on api boot." + + - name: Report the preview URL + if: ${{ inputs.action == 'deploy' }} + # BRANCH / ENV_NAME bound via env (never interpolated into the script). + env: + BRANCH: ${{ inputs.branch }} + ENV_NAME: ${{ steps.env.outputs.name }} + run: | + railway domain --service cio-dashboard || true + URL=$(railway variables --service cio-dashboard --kv | grep '^RAILWAY_PUBLIC_DOMAIN=' | cut -d= -f2-) + if [ -z "$URL" ]; then + echo "::error::cio-dashboard has no public domain — 'railway domain' may have failed." + exit 1 + fi + { + echo "### Railway preview ready :rocket:" + echo "" + echo "- Environment: \`$ENV_NAME\` (branch \`$BRANCH\`)" + echo "- Dashboard: https://${URL}" + echo "" + echo "> All services passed their Railway deploy/healthcheck before this link was printed, so it should be live now." + echo "" + echo "Tear down with this workflow using **action = destroy** and the same branch." + } >> "$GITHUB_STEP_SUMMARY" diff --git a/PREVIEW_ENV.md b/PREVIEW_ENV.md new file mode 100644 index 00000000..7ed3490f --- /dev/null +++ b/PREVIEW_ENV.md @@ -0,0 +1,73 @@ +# Railway Staging & PR Preview Environments + +ClassroomIO's production SaaS runs on Render + Cloudflare — **not** Railway. This +setup adds a **separate, internal Railway project** that provides a long-lived **staging** +environment plus **short-lived preview environments** for testing a branch/PR of the full +stack (api, dashboard, jobs, Postgres, Redis, MinIO) before it merges to `main`. + +Real users are never routed here; it's a team-only sandbox. + +## How to deploy to staging + +Push or merge your change to the branch connected to the staging environment (staging). Railway +picks it up automatically and redeploys — no manual step required. + +## How to deploy a PR to a preview environment + +Two mechanisms, both enabled: + +| | Native PR environments | Manual workflow (`railway-preview.yml`) | +|---|---|---| +| Trigger | Automatic on every opened PR | You run it from the **Actions** tab | +| Teardown | Automatic on PR merge/close | `action = destroy`, or auto via `railway-preview-teardown.yml` on PR close | +| Control | Whole base env is duplicated | Same, but you choose when and can re-seed the DB | +| Requires | App services connected to the GitHub repo (Dockerfile builder) | A Railway account/team token in CI | + +> A service that deploys from a **fixed Docker Hub image tag can never get a PR preview** — +> an image tag has no branch. That's why the Railway app services build **from the repo** +> via `docker/Dockerfile.*`, not from `docker-compose.images.yaml`. + +### Using the manual workflow + +> 🎥 Demo video: TODO + +**Actions → "Railway Preview Environment" → Run workflow:** +- `branch` — the branch/PR head to deploy. +- `action` — `deploy` (create/update) or `destroy` (delete). +- `seed` — also load the full demo dataset (`admin@test.com` etc.); migrations + essential + seed always run on api boot regardless. + +The run prints the preview dashboard URL in its job **summary**. The env is named +`pr-`. + +The deploy step runs `railway up --ci` (no `--detach`), which streams build logs and **blocks until +each service passes its deploy/healthcheck** before the next one starts — so the printed URL is live +the moment the run finishes, and a failed healthcheck fails the run rather than reporting a broken +preview. + +## Database behavior + +- Each environment has its **own isolated Postgres** — previews start fresh. +- `cio-api` **self-migrates on boot** (`docker/entrypoint-api.sh` → `pnpm --filter @cio/db + db:setup` → `drizzle-kit migrate` + essential seed). No manual migrate step needed. +- **The branch must include a committed Drizzle migration** (`pnpm --filter @cio/db + db:generate`) — the boot path runs committed migrations only, not schema `push`. + +## Verify before relying on it (spike, do on a throwaway env) + +Railway's docs are thin on a couple of points the workflow depends on — confirm these once: + +1. **`railway environment new --duplicate ` copies service config *and* variables.** + If your CLI version lacks `--duplicate`, use the GraphQL + `environmentCreate(sourceEnvironmentId)` mutation instead. +2. **A duplicated env's Postgres starts empty** (so boot migration builds a clean schema). +3. **The demo-seed step's DB reachability.** `railway run` executes on the GitHub runner, so + it needs a DB URL reachable from CI (Railway's public TCP proxy `DATABASE_PUBLIC_URL`), + not the private `railway.internal` one. If unreachable, drop the seed step and rely on the + boot-time essential seed. These caveats are marked `# SPIKE:` in `railway-preview.yml`. + +## Cost + +Every environment (staging + each preview) runs the full 6 services incl. a MinIO volume, so +previews are not free while alive. Destroy them promptly — the teardown workflow handles PR +close automatically; use `action = destroy` for previews whose PR is still open.