Deploy runbook — shipping the integration branch to production

Step-by-step checklist for releasing integration/ops-upgrade (or any branch that carries migrations and edge-function changes) to the production Supabase project yzpfwdxpwalmfuodkxni and Cloudflare Pages. Rules behind it: PLT-010 … PLT-043.

Production is on the Supabase Free plan

There is no point-in-time recovery. If a migration corrupts data, the only way back is the logical backup you take in step 1 — and anything written after that backup is lost on restore. Do not skip step 1.

Roles for the window: Operator (runs commands), Checker (reads every command before it runs, ticks the list), Business tester (runs the smoke tests in step 7). One person may not be both Operator and Checker (ACC-020).

0. Go / no-go (day before)

[ ] Branch is merged (or ready to merge) to main and CI is green, including db-tests and deno-check.
[ ] node scripts/check-migrations.mjs → 0 errors; no warnings in new migrations.
[ ] Locally, from a clean DB: supabase db reset then scripts/db-test.sh → all passed.
[ ] If a staging project exists (PLT-001): steps 2–6 have been run there first and passed.
[ ] Every secret in §4 that the release needs has a value in hand (password manager), especially the new WHATSAPP_APP_SECRET.
[ ] Window agreed: low traffic (after 21:00 IST or before 07:00 IST — the backup cron runs at 07:30 IST). Staff told that finance/booking screens may be briefly unavailable.
[ ] Supabase CLI ≥ 2.70 installed (supabase --version), gh authenticated, psql available.

Assume this release is not backward compatible with the frontend currently live (PLT-014) unless staging proves otherwise: the security-lockdown migration moves finance-PIN checking into the database and hides User secrets, and several edge functions now require a signed-in caller. Plan for migrations → secrets → functions → frontend in one sitting, with as little time as possible between the database step and the frontend going live.

1. Fresh backup

Trigger the backup workflow and wait for it:

gh workflow run supabase-backup.yml
gh run list --workflow supabase-backup.yml --limit 1     # note the run id
gh run watch <run-id>

UI alternative: Actions → Supabase Backup → Run workflow.

Confirm the run is ✅ and its summary lists roles.sql.gz, schema.sql.gz, data.sql.gz with non-trivial sizes (compare with yesterday's run).

Download the artifact to a safe place outside the repo and verify checksums:

gh run download <run-id> -D ~/alhuda-backups/$(date -u +%Y%m%dT%H%M%SZ)
cd ~/alhuda-backups/<dir>/supabase-backup-*/ && sha256sum -c checksums.txt   # macOS: shasum -a 256 -c

Record in the release notes: run id, timestamp, file sizes.

Stop if the backup failed. Fix the backup first.

2. Compare remote and local migrations

supabase link --project-ref yzpfwdxpwalmfuodkxni     # asks for the DB password
supabase migration list --linked

The output has a Local and a Remote column per version. Expected for this release: every row has both columns filled except the new local-only versions:

Version	File
20260917100000	booking_status_rejected
20260917100100	release_group_booked_count
20260917120000	audit_trail_hardening
20260917130000	cancellation_chain
20260917140000	release_single_seat
20260917150000	security_lockdown_access_control
20260917160000	airline_cancellation_filing_failure

(Plus whatever later waves add — regenerate this table from git diff --name-status main...HEAD -- supabase/migrations.)

Stop and investigate if:

a version appears Remote-only (someone applied SQL by hand — PLT-010; pull it into the repo first with supabase migration fetch/db pull in a separate PR),
an older version is Local-only (a migration was skipped; it will be applied out of order — review it before continuing),
20260408 looks odd — that legacy file (20260408_whatsapp.sql) is expected to show with its short version.

Then dry-run:

supabase db push --dry-run

It must list exactly the expected files, in order.

3. Apply migrations

supabase db push

Do not add --include-all unless step 2 showed an older Local-only migration that you reviewed and intend to apply.
The CLI applies files in version order and stops at the first failure; files before it stay applied. If a file fails: do not edit it and retry blindly — read the error, decide with the Checker between a forward fix (new migration) and restore (§9).

Verify:

supabase migration list --linked      # no Local-only rows left

Quick sanity queries (SQL editor or psql "$SUPABASE_DB_URL", read-only):

select count(*) from "Booking";                                   -- same order of magnitude as before
select proname from pg_proc where proname in ('finance_pin_matches','user_is_active');   -- 2 rows
select count(*) from pg_policies where schemaname = 'public';    -- non-zero

4. Secrets

SUPABASE_URL, SUPABASE_ANON_KEY and SUPABASE_SERVICE_ROLE_KEY are injected by Supabase automatically — never set them by hand.

supabase secrets list          # names + digests only; values are never shown

Set anything missing (one supabase secrets set NAME=value … call; quote values with spaces; never paste secrets into shell history on shared machines — use supabase secrets set --env-file ./prod.env from a file outside the repo and delete it afterwards).

Secret	Used by	Required?	If missing
`WHATSAPP_APP_SECRET` — new in this release	whatsapp-webhook	Yes	Webhook accepts unsigned payloads (fails open, logs a warning) — PLT-013
`TURNSTILE_SECRET_KEY`	lead-intake, visa-intake	Yes	Public forms can't verify CAPTCHA
`WHATSAPP_ACCESS_TOKEN`, `WHATSAPP_PHONE_NUMBER_ID`, `WHATSAPP_VERIFY_TOKEN`	whatsapp-webhook, mailer (via `_shared/whatsapp.ts`)	Yes (if WhatsApp is used)	Webhook verification / sends fail
`WHATSAPP_DEFAULT_COUNTRY_CODE`, `WHATSAPP_GRAPH_VERSION` (default `v22.0`), `WHATSAPP_OTP_TEMPLATE_NAME`, `WHATSAPP_TEMPLATE_LANGUAGE`	mailer, whatsapp-webhook	Optional	Defaults / OTP outside 24h window fails
`COMMUNICATION_DISPATCH_KEY`	communications-dispatcher	Recommended	Scheduler must use the service-role key
Vault: `communications_dispatcher_url`, `communications_dispatcher_key` — once, in the database, not as a function secret	the pg_cron job `alhuda-communications-dispatcher` (`comms_dispatcher_run()`, every five minutes)	Yes	The job logs "not configured" every five minutes and sends nothing; queued e-mails (a new booking's, LC-007) leave only on the caller's kick or a hand run. The nightly WhatsApp template sync (`alhuda-whatsapp-templates-sync`, AUD-024) uses the same two secrets — it posts to `whatsapp-templates-sync` next to the dispatcher's URL — and also does nothing until they are stored
`RESEND_API_KEY`, `RESEND_FROM`	admin-users, lead-intake, visa-intake, mailer, secret-selftest	Yes	Invites / notifications not sent
`MAILJET_API_KEY`, `MAILJET_SECRET`, `MAIL_FROM`	mailer, secret-selftest	Fallback mailer	No fallback when Resend fails
`MAIL_GMAIL_SENDER`, `MAIL_GMAIL_FROM`, `MAIL_REPLY_TO` (with the Drive service account's `GOOGLE_DRIVE_CLIENT_EMAIL` / `GOOGLE_DRIVE_PRIVATE_KEY`)	mailer, integration-test	Optional (COMM-040)	Without them mail goes through Resend, then Mailjet. Values: sender `admin@alhudatravels.in`, From `Alhuda Travels <noreply@alhudatravels.in>`
`LEADS_NOTIFY_EMAIL`, `VISA_INTAKE_NOTIFY_EMAIL`	lead-intake, visa-intake	Yes	Nobody is notified of new leads / visa intakes
`RAZORPAY_KEY_ID`, `RAZORPAY_KEY_SECRET`	razorpay-order	Yes (online payment in the app)	The app's Pay online answers 503 "not switched on yet" and points to I have paid; nothing else breaks. The key id is also handed to the phone for Checkout; the secret never leaves the function
`RAZORPAY_WEBHOOK_SECRET`	razorpay-webhook	Yes	Payment webhooks rejected — an online payment taken through the app is then never recorded until the secret is set and Razorpay retries
`VAPID_PUBLIC_KEY`, `VAPID_PRIVATE_KEY`, `VAPID_SUBJECT`	push-send	Yes (push)	Push notifications fail
`GOOGLE_DRIVE_CLIENT_EMAIL`, `GOOGLE_DRIVE_PRIVATE_KEY`, `GOOGLE_DRIVE_FOLDER_ID`, `GOOGLE_DRIVE_IMPERSONATE_EMAIL`	upload-customer-doc, upload-partner-doc, upload-employee-doc, drive-upload, drive-file, lead-intake, visa-intake, chat-upload, issue-document	Yes	No file can be stored or opened anywhere (ACC-074: files live only on the Shared Drive) — see Google Drive
`DRIVE_LINK_SECRET`	drive-file, issue-document	Recommended	The ten-minute file links are signed with the service role key instead; rotating this secret alone revokes every outstanding link
`VIRUSTOTAL_API_KEY` (`VIRUSTOTAL_BASE_URL` optional)	upload-scan	Yes	Uploads not scanned
`AIRLABS_API_KEY`	flight-lookup	Optional	Fetch flight details on Airline blocks and FIT finds nothing at AirLabs (the function answers 503) and fills from flights already in inventory only. Replaces the old browser variable `VITE_AIRLABS_API_KEY`, which must not be set anywhere (PLT-015)
`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GEMINI_API_KEY`, `GROQ_API_KEY`, `OCR_SPACE_API_KEY`	extract-passport, finance-copilot, package-copilot	At least one AI key	Passport extraction / copilots fail
`GDRIVE_ACCESS_TOKEN`, `CLOUDINARY_CLOUD_NAME`, `CLOUDINARY_PRESET`, `B2_AUTHORIZATION`, `B2_BUCKET`, `B2_UPLOAD_URL`, `IDRIVE_AUTH`, `IDRIVE_UPLOAD_URL`	none since 27 Sep 2026 — `chat-upload` stores on the Shared Drive only	No	Unused; delete them
`RAZORPAY_WEBHOOK_TEST`, `RAZORPAY_ORDER_TEST`	razorpay-webhook / razorpay-order tests only	Never set in production	—

Regenerate this table when functions change:

grep -rhoE "Deno\.env\.get\(\s*['\"][A-Z0-9_]+['\"]" supabase/functions | sort | uniq -c

After setting secrets, secret-selftest can confirm mail providers are reachable (call it as a signed-in admin).

The dispatcher's schedule (Vault, once)

20260929110100_the_dispatcher_runs_every_five_minutes.sql schedules the pg_cron job alhuda-communications-dispatcher (every five minutes). It calls comms_dispatcher_run(), which posts to the communications-dispatcher function with a URL and a key it reads from Supabase Vault — a key is never written into a migration. The owner stores both once, in the SQL editor of the hosted project (never in the repo, never in a migration):

select vault.create_secret('https://<project-ref>.supabase.co/functions/v1/communications-dispatcher', 'communications_dispatcher_url');
select vault.create_secret('<the COMMUNICATION_DISPATCH_KEY value>', 'communications_dispatcher_key');

Use the COMMUNICATION_DISPATCH_KEY you set above (the service-role key also works; prefer the dispatch key so the service key stays out of Vault). Check it is alive: select public.comms_dispatcher_run(); answers requested:<id> (a pg_net request was queued) or not_configured; select * from cron.job where jobname = 'alhuda-communications-dispatcher'; shows the schedule, and cron.job_run_details its runs. Until the secrets exist the job logs "not configured" and does nothing — a new booking's e-mail then leaves on the caller's kick (the browser or the app asks the dispatcher to send its own rows right away) or when the dispatcher is called by hand.

5. Deploy edge functions

Type-check first (same as CI):

scripts/deno-check-functions.sh

This release changed _shared/auth.ts, which is bundled into every function that imports it, so deploy all importers plus the directly changed functions:

for fn in admin-users auth-admin-mfa-reset customer-exchange extract-passport finance-copilot \
          mailer permissions-admin push-send push-subscribe secret-selftest session-manager \
          upload-customer-doc upload-scan whatsapp-webhook chat-upload \
          customer-signup partner-signup upload-partner-doc upload-employee-doc \
          drive-file drive-upload issue-document razorpay-order flight-lookup; do
  supabase functions deploy "$fn" --project-ref yzpfwdxpwalmfuodkxni || break
done

(Find importers for a future release with grep -l "_shared/<file>.ts" supabase/functions/*/index.ts; find changed functions with git diff --name-only <last-release-tag>..HEAD -- supabase/functions.)

For each deployed function confirm in Dashboard → Edge Functions: the new version timestamp, and that Verify JWT matches supabase/config.toml (functions listed there with verify_jwt = false must show it off, otherwise ES256 user tokens and webhooks get 401).

6. Frontend

Merge to main (squash) → CI → Cloudflare Pages publishes. Watch Cloudflare → Workers & Pages → Deployments until the new build is live, then hard-refresh (the PWA service worker may serve the old bundle until "Update available" is accepted).

Squash merges can carry [skip ci]

A squash merge's default message lists every commit in the PR. The integration branch contains the previous release's chore(release): x.y.z [skip ci] commit, so the squash message contains [skip ci] — and both GitHub Actions and Cloudflare Pages then skip the commit: no release, no frontend build, no error. This happened on 2026-09-19 (#379). Always merge a release with your own message:

gh pr merge <n> --squash --subject "Release: <summary> (#<n>)" --body "See #<n>."

Then confirm the live bundle changed — a Cloudflare status on the commit is not enough, because production builds may not report one.

7. Smoke tests

Run immediately after the frontend is live, on production, with real (not admin) accounts where possible. Record ✅/❌ and the time for each.

#	Test	Expected
1	Staff login at `/auth` as a sales user, then as a finance user	Dashboard loads; menu shows only permitted sections; no console errors / Sentry spike
2	Customer sign-up at `/customer/auth` with a new email (Turnstile visible)	Account created, lands on `/customer`; the new user has only the customer role (no staff menus)
3	Partner sign-up at `/partner/auth`	Account created with the agent/partner role only; `/partner` loads; staff pages return "not authorised"
4	Finance PIN — finance user opens Finance, enters the correct PIN, then a wrong PIN several times	Correct PIN unlocks; wrong PINs are rejected and lock out after the configured attempts (verified in the database, not the browser)
5	Create booking for a test customer on a test group	Booking saved with a booking number; group booked count increases by the passenger count
6	Record payment against that booking (cash, small amount), then verify it as a different finance user	Payment shows; journal entry balanced (receivable credited, cash debited); customer balance reduced; if maker ≠ checker (ACC-020) is enforced in this release, the recording user cannot verify it
7	Cancel the test booking's passenger	Seat released back to its block/FIT; audit trail shows who did it
8	WhatsApp webhook — send a message to the business number	Appears in communications; function log shows no "not signature-verified" warning
9	Lead intake / visa intake public form	Submission saved; notification email received

Clean up the test booking, payment and accounts afterwards (reverse/cancel through the app — do not delete rows by SQL).

Watch for 30–60 minutes: Supabase → Logs (Postgres errors, 4xx/5xx on Edge Functions, Auth errors) and Sentry.

8. Record the release

In docs/rules/decisions-log.md or the release notes: date/time, commit, migration versions applied, functions deployed, secrets added (names only), backup run id, smoke-test results, who was Operator/Checker.

9. Rollback plan

Decide within the window using this order — least destructive first.

Symptom	Action
Frontend bug only (API and DB fine)	Cloudflare Pages → Deployments → Rollback to the previous build. Caution: the previous frontend is not compatible with the new database (§0) — prefer a forward fix unless the old UI is verified to work.
One edge function broken	Redeploy the previous version: `git checkout <previous-tag> -- supabase/functions/<fn> supabase/functions/_shared && supabase functions deploy <fn>`, then `git checkout HEAD -- supabase/functions`. If the new function needs a secret that is wrong, fix the secret instead.
Migration failed half-way	Nothing after the failing file ran. Fix forward with a new migration (review + push), or, if the applied part is harmful, write an undo migration. Never edit an applied file.
RLS / permission change locks users out	Forward-fix migration that restores the specific grant or policy (emergency: Operator + Checker apply it via `supabase db push` from a hotfix branch — record it as an incident per PLT-010).
Data corrupted or lost	Restore from the step-1 backup following backup-restore.md. Everything written since the backup is lost, so first export affected new rows (bookings, payments) made during the window. Management approves a restore.

After any rollback: re-run the smoke tests, post an incident note, and add a DB test (supabase/tests/) that would have caught the problem (PLT-022).