Environments and Deploys

OpsDNA runs two environments — prod and staging — in the same AWS account and region, as structurally identical CDK stack sets. Code promotes through them in one direction: feature branch → staging → main, with staging validating every change on real infrastructure before a human dispatches the prod release — a manual gate that exists because the org’s GitHub plan lacks environment required-reviewers, so the human dispatch is the approval step. This page is the how-to for that flow.

How env selection works

The CDK app reads one context value:


# staging: synthesizes only the OpsdnaStaging* stacks
cdk deploy --all --context env=staging
 
# prod: no context at all — the historical default
cdk deploy --all

infra/lib/env.ts (resolveEnvConfig) maps that to an EnvConfig. The load-bearing invariant:

For prod, every derived value must equal the historical hardcoded value — empty physical-name suffix, empty stack prefix, /opsdna/prod SSM prefix — so a cdk diff against the live prod stacks is a no-op. A changed physical name means CloudFormation replaces the resource, which for the RDS instance would be catastrophic. Never make prod naming depend on new logic without re-verifying a clean diff.

What differs between the two:

	prod	staging
Stack IDs	`OpsdnaDatabaseStack`, …	`OpsdnaStagingDatabaseStack`, …
SSM prefix	`/opsdna/prod`	`/opsdna/staging`
API custom domain	`api.opsdna.ai` (default on — a bare deploy once tore the live domain down before this default)	none (raw HTTP API endpoint)
RDS deletion protection	on	off (staging can be torn down)
Backup retention	30 days	1 day
Lambda provisioned concurrency	on	off
`CiStack` / `SecurityBaselineStack`	created here (account-global)	not synthesized

The portal has the same split on Cloudflare: prod worker opsdna-portal at app.opsdna.ai, staging worker opsdna-portal-staging at staging-app.opsdna.ai (one-level subdomain on purpose — Universal SSL only covers *.opsdna.ai one level deep).

The promotion pipeline

Three enforcement points:

CI gates every PR into staging and main identically (ci.yml: the build / typecheck / test job, the ~20-gate gates job, and the real-Postgres postgres integration job).
enforce-branch-flow.yml fails any PR into main whose head branch is not staging — main only accepts promotions.
Prod deploys are manual by design. prod-deploy.yml is workflow_dispatch only. The org’s GitHub plan doesn’t support environment required-reviewers, so the human choosing to dispatch is the approval gate. Merging to main deploys nothing.

Both deploy workflows authenticate to AWS via the github-ci-opsdna OIDC role (created by CiStack) — no stored AWS credentials. Cloudflare credentials come from GitHub environment secrets.

Both run the same ordered pipeline in one job, so ordering is guaranteed by construction:

Migrate — infra/scripts/ci-migrate.sh <env> runs bun migrate:latest over the SSM tunnel against the real DB. A failed migration fails the job here; the code never ships.
Backend — cdk deploy --all (with --context env=staging for staging) swaps the Lambda code, now safe because the schema is ahead.
Frontend — build + deploy the portal Worker, only reached if the backend succeeded (the frontend never goes live ahead of the backend it calls). The vite build needs NODE_OPTIONS=--max-old-space-size=8192 — it OOMs at the default ~2 GB heap.
(prod only) Smoke — curl https://app.opsdna.ai and fail on a non-2xx/3xx.

This is expand/contract end to end; see Database operations for why the migrate-first ordering is non-negotiable.

How to ship a change to prod

Open a PR against `staging`

Never against main. Run bun run verify locally first (see Getting started) — it mirrors CI’s build/typecheck/test job and iterates in minutes instead of CI round-trips. If your change touches schema, also run the real-Postgres integration lane locally (see /persistence).

Merge — staging deploys itself

The push to staging triggers staging-deploy.yml: migrate against the staging DB, cdk deploy --all --context env=staging, then the staging portal. This is where migrate-before-deploy gets rehearsed against a real database — a bad migration fails here, never in prod.

Validate on staging

Exercise the change at staging-app.opsdna.ai / the staging API.

Promote via PR from `staging` to `main`

enforce-branch-flow.yml blocks any other head branch. CI runs again on the promotion PR.

Dispatch the prod release

From the GitHub Actions UI (or CLI), run Prod Deploy on main:


gh workflow run prod-deploy.yml --ref main

Watch it through migrate → CDK → portal → smoke. concurrency: cancel-in-progress: false means an in-flight prod deploy is never interrupted; queue behind it.

Pitfalls

Never run wrangler deploy --env staging for the portal. The @cloudflare/vite-plugin build flattens wrangler.jsonc to a single resolved environment and drops the env blocks; --env staging then falls into legacy-env mode, keeps the prod routes and vars, and hijacks app.opsdna.ai. Select the env at build time instead: CLOUDFLARE_ENV=staging bun run build:cf && wrangler deploy --config dist/server/wrangler.json — which is exactly what bun run deploy:cf:staging does.
Don’t unset the prod API custom domain. It defaults on so that a bare cdk deploy doesn’t tear down api.opsdna.ai (that footgun fired once). Override only via API_CUSTOM_DOMAIN when you know why.
Staging needs one-time bootstrap before its first deploy works: the staging RDS role setup (bootstrap-app-role.sh), put-secrets.sh for /opsdna/staging/*, and GRANT rds_iam TO opsdna_admin for the migrate connect. Until then the migrate step fails — which is the point: it fails there, not in prod.
Push-to-main does not deploy prod. If prod looks stale after a promotion merge, nobody dispatched the release.

feature-preview.yml / preview-cleanup.yml / neon-preview-base.yml — per-PR preview environments on Neon branch-per-preview; the long-lived preview base branch is refreshed automatically after each successful Staging Deploy (workflow_run trigger).
landing-page-deploy.yml — the landing page (Cloudflare Pages) has no backend dependency and deploys independently.
legacy-migration-replay.yml — frozen pre-baseline migration replay, nightly/manual only.