Skip to content

Release rehearsal — integration/ops-upgrade against production data (2026-09-18)

Verdict: GO WITH CLEANUP, conditional on the fix committed alongside this document (20260921120000_drop_legacy_policies_and_portal_grants.sql).

Without that migration the call is NO-GO: the release's audit-trail restriction does not take effect on production, and partner/customer logins keep staff permissions.

What this was: the 2026-09-18 production backup restored into a throw-away local Postgres, the release applied to it the way a deploy would, and then every test and integrity check we have run against the result. No connection was made to the production project at any point; the only thing taken from it is the backup artifact that the nightly workflow already publishes.


1. What was restored

Source: GitHub Actions run 35316380252, artifact supabase-backup-20260918T064706Z (397 KB), produced by .github/workflows/supabase-backup.yml — three supabase db dump outputs (roles.sql, schema.sql schema-only, data.sql data-only COPY), gzipped, with SHA-256 checksums. All three checksums verified.

Restored into a local stack on ports 593xx (project alhuda-rehearsal) with the documented one-shot command:

psql --single-transaction -v ON_ERROR_STOP=1 \
     -f roles.sql -f schema.sql \
     -c 'SET session_replication_role = replica' -f data.sql

Restore time: 1.9 s. Zero errors, zero warnings. 99 tables, 2,727 rows.

Table Rows Table Rows
RolePermission 1,090 UserRole 19
UserSession 186 Role 18
AuditLog 179 LedgerEntry 18
Permission 168 Customer 14
Account 89 Supplier 14
CommunicationsLog 83 UserPermission 13
JournalLine 77 User 12
exchange_rate_snapshots 52 Agent 10
exchange_rates 42 TravelGroup 8
ChatMessage 37 BookingPassenger 7
JournalEntry 34 Invoice 7
ChatDMMember 28 TicketRecord 7
MfaBackupCode 20 Booking 6

auth.users: 18. storage.objects: 0. Every operational table below Booking is single digits — this is a young dataset, which matters for the timing numbers below.

Three things about the backup itself

  1. The backup does not contain supabase_migrations.schema_migrations. supabase db dump covers public (plus auth/storage data), not the migration ledger. A restore from these artifacts therefore comes up with an empty migration history, and the next supabase db push would try to replay all 304 migrations onto a database that already has the schema. For the rehearsal the ledger was reconstructed from git ls-tree main (263 versions) after confirming structurally that production matches main's head. Recommendation: add --schema supabase_migrations (or a fourth dump step) to the backup workflow, and say so in backup-restore.md. This is a disaster-recovery gap, not a release blocker.

  2. The local GoTrue is older than production's. Seven auth tables in the dump do not exist locally (custom_oauth_providers, mfa_recovery_code_sets, mfa_recovery_codes, scim_tokens, scim_users, webauthn_challenges, webauthn_credentials) and five tables have columns the local version lacks. Every one of those rows is empty except auth.one_time_tokens.expires_at (3 rows) and storage.buckets.versioning_status (1 row), so they were dropped for the restore. Irrelevant to this release — but anyone restoring to a local stack for real will hit it, so it belongs in the restore runbook.

  3. Production was confirmed to be at main's head before the release: the markers from the last four main migrations are all present (Account delete authenticated policy, the buyer/supplier split columns on B2BFlightOfferCancellation, auth_user_has_permission(), all 41 public functions with a pinned search_path) and none of the release's objects are (BookingStatus.REJECTED, Incident*, InventoryHold, drift functions, incident permissions).


2. What ran

41 migrations were pending (40 from the branch + the fix this rehearsal produced). Applied in filename order, one transaction per file plus its ledger insert — the same shape as supabase db push.

supabase db push --db-url could not be used locally: the CLI forces TLS and the local stack refuses it (tls error (server refused TLS connection)), even with sslmode=disable. Against production, where TLS is real, db push is the right command; the psql loop is an exact stand-in for the rehearsal.

Every migration applied cleanly. No failures, no rollbacks, no manual intervention.

Migrations applied 41
Failures 0
Wall-clock 9 s
psql time 4.4 s
Slowest file 20260918100000_data_isolation_portals.sql, 0.24 s

All 41, with per-file timings, are in §8.

Notices worth a human's attention

Everything else in the 214-line notice log is DROP … IF EXISTS idempotency chatter. These four are substantive:

NOTICE:  Access model seeded: 20 roles, 181 permissions, 1000 grants
NOTICE:  SUPER_ADMIN assignment: rule=email users=1 newly-added=1
NOTICE:  Role bundles applied: [{"role":"CEO","granted":1,"removed":93},
         {"role":"GM","granted":9,"removed":74}, {"role":"IT_ADMIN","granted":0,"removed":144},
         {"role":"ADMIN_HR","granted":0,"removed":69}, {"role":"SUPER_ADMIN","granted":171,"removed":0},
         {"role":"CHARTERED_ACCOUNTANT","granted":38,"removed":0}, …]
NOTICE:  INV-012: 1 inventory row(s) are already over-allocated and were left
         untouched — see inventory_drift_report()

The INV-012 line is the designed behaviour of 20260920100000_inventory_counter_integrity.sql: it backfills every counter to the derived truth but refuses to invent seats, logging the row instead of failing the migration (AUD-010). See §5.

Did any migration silently skip?

No. 20260920100000 adds eleven non-negative CHECK constraints NOT VALID and then tries to VALIDATE each, downgrading a check_violation to a notice. All eleven validated — no INV-013 … has rows violating notice was raised. The only unvalidated constraint left in the database is CustomerRequest_status_check, which 20260918130000 deliberately declares NOT VALID and never validates (the table is empty in production, so nothing is hiding behind it).

For completeness, all 303 migrations also apply to an empty database in 52 s with zero failures, and the full supabase/tests suite is 18/18 green there — so nothing below is a pre-existing branch failure.


3. What broke, and what was fixed

Two of the eighteen DB test suites fail on restored production data and pass on a freshly migrated one. Both are real, both are security-relevant, and both are fixed by the migration committed with this document.

3.1 NO-GO (fixed) — the audit trail stayed world-readable · AUD-004

supabase/tests/audit_trail.sql:

ERROR:  FAIL sales user can read raw AuditLog (1126 rows)

20260917120000_audit_trail_hardening.sql drops four named policies on AuditLog and replaces SELECT with auth_user_has_permission('admin.audit.view'). Production also carries a fifth:

policy "auth_access" ON public."AuditLog" FOR ALL TO authenticated
  USING (true) WITH CHECK (true)

No migration in this repository creates that policy. grep -rn auth_access supabase/migrations/ finds auth_access created on nine other tables by 20260406010000_fix_rls_policies.sql — never on AuditLog. It was applied to the project out of band, and it is the only surviving auth_access policy in the database; 20260917150000 cleans up the other nine.

Postgres ORs permissive policies together, so the hardening migration would have landed on production and changed nothing: every signed-in user — including the two AGENT and two CUSTOMER portal logins — could still SELECT all 1,126 audit rows, names, masked identity numbers and all. The append-only triggers do hold, so this is a confidentiality failure, not a tampering one.

This is the most important finding of the rehearsal, and it is invisible to the runbook's §2 check: supabase migration list compares migration versions, and a hand-made policy has no version. Only a diff of the actual catalogue, or a test run against real data, finds it.

Fix: the new migration drops every policy on AuditLog that is not one of the two the release owns — so any other out-of-band policy on that table goes with it, not just the one we happened to find.

3.2 NO-GO (fixed) — portal roles held staff permissions · ACC-001, ACC-012

supabase/tests/access_model.sql:

ERROR:  FAIL ACC-001 a portal role holds staff permissions:
        AGENT → dashboard.view, AGENT → customers.view,
        AGENT → quotations.view, AGENT → bookings.view,
        CUSTOMER → dashboard.view

PERMISSIONS.md §4 and §5 both say portal roles hold none ("Portal-only; no staff permissions"). Production has five such grants, from before the portal model was settled. apply_role_bundles() narrows the roles it has a bundle for; AGENT and CUSTOMER have an empty bundle and are left alone, which on a seeded database is correct and on real data is a no-op where a cleanup was needed. Combined with 3.1, this is exactly how a partner login ends up reading the company's audit log.

Fix: the new migration deletes RolePermission rows for AGENT and CUSTOMER. No permission is added or removed, so src/test/permissions.matrix.test.ts is unaffected; PERMISSIONS.md §10 gains a changelog row.

3.3 Verified

Restored again from the dump, applied all 41 migrations, re-ran the suites:

NOTICE:  AUD-004: dropping non-release policy auth_access on AuditLog
NOTICE:  ACC-001: removed 5 staff permission grant(s) from portal roles AGENT/CUSTOMER

access_model.sql and audit_trail.sql now pass against production data.


4. Test results on the migrated database

scripts/db-test.sh, all suites, against restored production data + the release:

14 passed, 4 failed (of 18). All four remaining failures are the test fixtures colliding with pre-existing rows, not release defects — each one passes on a freshly migrated database.

Suite Why it fails on real data
tickets_visa_comms.sql inserts CommunicationSetting('whatsapp_phone_id'); production already has that key → duplicate key value violates unique constraint
bookings_paging.sql walks booking_page() and asserts an exact total: "walk returned 66 rows, expected 60" — the 6 real bookings
role_dashboards.sql asserts exact finance-inbox counts; production adds 18 journals / 4 booking approvals to the fixture's numbers
inventory_integrity.sql calls inventory_repair_drift() with no id filter, so it sweeps in the genuinely over-allocated production FIT row (§5.1) and reports it under failed

These suites are written against supabase db reset state and documented that way in testing.md, so this is not a regression — but it does mean db-test.sh cannot be used as a post-deploy smoke check on production. Making the four hermetic (unique fixture keys, counts relative to a baseline, id-scoped repair) would give us a suite that can be pointed at a restored copy of production on any day. Worth doing; not a blocker.

Residue: a full suite run was sandwiched between two complete row counts of all 124 tables. diff is empty — every suite rolls back cleanly, including the four that fail mid-file.

Key read paths

Executed as the owner's admin@alhuda.co.in account under RLS (SET ROLE authenticated + request.jwt.claims):

Path Result
get_dashboard_stats() returns
get_booking_journey() on a booking with passengers returns
get_group_journey() returns
get_customer_360() returns
get_audit_timeline('booking', …) 8 events
get_journey_board_page() returns
booking_page() 6 rows (all live bookings)
finance_day_book_page(−180 d … today) returns
finance_aging_report(), finance_account_totals() return
list_incidents_page() returns

No read path regressed.


5. Data integrity under the new rules

Real data predates these rules, so findings were expected. None of them blocks the release — per AUD-010 they are reported here and repaired only on approval.

5.1 Inventory counter drift — 1 row · BLOCKS NOTHING, FIX SOON

inventory_drift_report():

Rows 1 of 2 FIT rows / 5 airline blocks / 1 hotel / 0 food / 0 ground
Severity over_allocated, also flagged oversoldRisk
Counter availableSeats: stored 0, derived −3000, difference 3000

Anonymised example — a 3-seat FIT block whose cancelledChargedSeats is 3000:

totalSeats 3 | availableSeats 0 | allocatedSeats 0
cancelledFocSeats 3 | cancelledChargedSeats 3000 | lostSeats 0

3,000 charged cancellations on a three-seat block is not a counter that drifted; it looks like a rupee amount written into a seat-count column by whatever recorded that cancellation. inventory_repair_drift() correctly refuses it — "no counter edit can conjure a seat" — so it will sit in the drift report until someone decides the right number.

Action: an operator confirms the intended value against the airline manifest, then inventory_repair_drift() (needs inventory.counters.repair, demands a written reason, writes an InventoryDriftRepair row). Until then the FIT block reads as unsellable, which is the safe direction. Worth a separate look at whichever code path wrote it.

5.2 Group bookedCount drift — 2 of 8 groups · BLOCKS NOTHING

Group stored bookedCount passengers on live bookings
(anonymised, 3 live bookings) 0 3
(anonymised, 3 live bookings) 6 7

This is precisely the leak 20260917100100_release_group_booked_count.sql documents: the rollback paths called try_increment_group_booked_count() with a negative value, which it rejects, and supabase-js returned the failure as { error } instead of throwing — so every failed booking or transfer leaked seats. The release stops the leak going forward; it does not recount history.

Both directions appear (one under-counts, one over-counts), so a blind recount is safe but should still be an approved repair, and AUD-010 lists "group counts vs passengers" as not yet covered by a drift function. Recommend a follow-up that adds group_counter_drift() alongside inventory_drift_report() and a one-off approved recount.

5.3 Booking passengerCount vs passenger rows — 3 of 6 · ADVISORY

Three bookings carry passengerCount = 1 with zero BookingPassenger rows. All three are PENDING_OPS — the intake stage, before passengers are entered — so this is the expected shape of a booking at that point, not drift. No new constraint or trigger touches it, and all three are readable and editable after the release. Flagged only so nobody is surprised by the number.

5.4 Clean

Check Result
finance_opening_balance_drift() [] — no drift
Journal entries where Σdebit ≠ Σcredit 0 of 34
LedgerEntry rows pointing at a missing booking 0 of 18
Dangling soft references (every un-FK'd *Id text column against its obvious target) 0
New CHECK / NOT NULL / FK violations 0 (see §2)

After the release the database carries 149 CHECK, 179 foreign-key, 124 primary key and 15 unique constraints and 175 non-internal triggers on public — all valid but the one deliberate NOT VALID.


6. The access model after the release

20 roles, 181 permissions, 995 grants, 12 app users.

Per-role

Role Users Permissions
SUPER_ADMIN 1 181
ADMIN_HR 1 96
CEO 1 83
GM 2 83
FINANCE_MANAGER 1 78
AUDITOR 0 63
OPS_MANAGER 1 61
ACCOUNTANT 0 50
B2B_MANAGER 1 40
CHARTERED_ACCOUNTANT 0 39
SALES_MANAGER 1 38
OPS_EXEC 1 36
SALES_EXEC 1 33
IT_ADMIN 1 27
B2B_EXEC 1 25
TICKET_MANAGER 1 24
VISA_OFFICER 1 24
CASHIER 1 14
AGENT 2 0 (portal)
CUSTOMER 2 0 (portal)

Per-user, before and after

User (anonymised) Roles Before After
admin@alhuda.co.in ADMIN_HR, IT_ADMIN → +SUPER_ADMIN 168 181
ceo@alhuda.co.in CEO 166 83
9458ddaf GM 139 83
e10d1af3 (inactive) GM 139 83
96f98f46 FINANCE_MANAGER 82 78
1821e97b 9 ops/sales/B2B roles 77 82
4066fec7, dd73ab67 AGENT (portal) 4 0
6153dc23, 811e3e87 CUSTOMER (portal) 1 0
2a83864a (inactive) none 0 0
53444f91 none 0 0

The narrowing landed as intended

  • At least one account holds SUPER_ADMIN: yes, exactly one — admin@alhuda.co.in, assigned by the release's email rule (SUPER_ADMIN assignment: rule=email users=1 newly-added=1). Break-glass exists.
  • CEO/GM approve-only: confirmed. Their finance set is approvals (journals.approve/reject/bulk_approve, bookings.approve_finance, refunds.approve, cancellations.approve_airline/_b2b, periods.lock, approvals.high_value) plus *.view/*.export. Zero finance create/edit/delete/post/record/void permissions. incidents.manage is the only non-view write they keep — duty of care, deliberate.
  • HR out of finance: ADMIN_HR holds 0 permissions matching finance%.
  • Cashier records only: 14 permissions — twelve *.view, plus finance.view and finance.payments.record. Nothing else.

Lockout risk

Two users end up with zero effective permissions — 2a83864a (already deactivated) and 53444f91 (active). Neither is caused by the release: both have had no UserRole and no UserPermission row since before it (verified directly in the backup: 12 users, 10 with a role, 13 overrides across 2 users). The active one should be given a role or deactivated, but that is housekeeping, not a release consequence.

The real operational risk is the CEO account dropping from 166 to 83 permissions, and GM from 139 to 83. That is the owner-approved F1 narrowing (ACC-012), and everything they need to approve and see is intact — but the owner will notice the difference the morning after, so tell them before, not after. The admin@alhuda.co.in account gains SUPER_ADMIN and can grant anything back without a migration.

No other staff user loses working access. The nine-role ops/sales user actually gains (77 → 82).

Pre-existing account hygiene (not caused by this release)

  • 7 auth.users rows have no public."User" row — they can authenticate and then hold nothing.
  • 1 public."User" row has no auth.users row — it cannot sign in at all.

Worth an access review (ACC-042) after the release, not before it.


7. Timings, and the window production is inconsistent in

Step Measured (local)
Restore roles + schema + data 1.9 s
41 release migrations 9 s wall, 4.4 s psql
All 303 migrations onto an empty DB 52 s wall, 31 s psql
db-test.sh, all 18 suites ~4 min (dominated by money_integrity and finance_paging, which generate their own volume)

The dataset is 2,727 rows and the largest table is RolePermission at 1,090, so nothing in this release rewrites a large table — the cost is DDL and catalogue work, not data. The local 9 s is essentially pure execution; on the hosted database the per-statement round trip over the session pooler dominates, so budget 2–5 minutes for supabase db push and treat anything past ten minutes as a signal to look, not to panic.

That is also the window in which the database is ahead of the frontend. Two changes make the old frontend actively wrong during it — portal roles losing their staff grants, and CEO/GM losing the narrowed permissions — so the old UI will render controls that now 403. Deploy the frontend immediately after the migrations, in the same maintenance window, and prefer a low-traffic slot (the backup already assumes 07:30 IST is quiet).


8. Migrations applied

Migration Time
20260917100000_booking_status_rejected.sql 0.06s
20260917100100_release_group_booked_count.sql 0.06s
20260917120000_audit_trail_hardening.sql 0.11s
20260917130000_cancellation_chain.sql 0.09s
20260917140000_release_single_seat.sql 0.06s
20260917150000_security_lockdown_access_control.sql 0.19s
20260917160000_airline_cancellation_filing_failure.sql 0.07s
20260918100000_data_isolation_portals.sql 0.24s
20260918110000_money_integrity.sql 0.17s
20260918120000_booking_lifecycle.sql 0.15s
20260918130000_tickets_visa_comms.sql 0.19s
20260919100000_finance_journal_controls.sql 0.10s
20260919100100_role_types_super_admin_ca.sql 0.06s
20260919100200_finance_role_bundles.sql 0.18s
20260919100300_finance_payment_permissions.sql 0.07s
20260919100400_finance_supplier_tds_controls.sql 0.10s
20260919100500_finance_document_numbering.sql 0.07s
20260919100600_finance_opening_balances_reports.sql 0.08s
20260919100700_finance_approval_limits.sql 0.10s
20260919100800_finance_journal_same_txn_fix.sql 0.07s
20260919100900_finance_receipt_date_and_batch.sql 0.09s
20260919120000_journey_customer_360.sql 0.14s
20260919130000_incident_permissions.sql 0.06s
20260919130100_duty_of_care_incidents.sql 0.19s
20260919140000_role_dashboards.sql 0.14s
20260920100000_inventory_counter_integrity.sql 0.14s
20260920100100_inventory_archive_not_delete.sql 0.10s
20260920100200_inventory_drift_check.sql 0.09s
20260920100300_block_utilisation_true_sold.sql 0.06s
20260920110000_inventory_holds_deadlines.sql 0.16s
20260920110100_inventory_hold_release_functions.sql 0.11s
20260920110200_seed_inventory_hold_release_permissions.sql 0.07s
20260920110300_inventory_deadline_dashboards.sql 0.07s
20260920115900_release_approve_large_leadership.sql 0.06s
20260920120000_inventory_ia_ib_join.sql 0.07s
20260920120200_list_paging_indexes.sql 0.13s
20260920120300_list_paging_rpcs.sql 0.08s
20260920130000_bookings_paging_search.sql 0.08s
20260921100000_dashboard_stats_ticket_table.sql 0.06s
20260921110000_dr_seed_roles_and_permission_grants.sql 0.07s
20260921120000_drop_legacy_policies_and_portal_grants.sql (new, this rehearsal) 0.06s

9. The deploy, step by step

Follows deploy-runbook.md; this is the rehearsed version with what we learned. Operator + Checker, both present.

Rollback point = step 2. Everything before it is reversible by doing nothing; everything after it is reversible only by restoring that dump and losing what was written since.

  1. Pre-flight, on the branch. CI=true npx vitest run · npm run typecheck:ratchet · node scripts/check-migrations.mjs — all three green (they are, see §10).
  2. Fresh backup — this is the rollback point.
    gh workflow run supabase-backup.yml
    gh run watch <run-id>
    gh run download <run-id> -D ~/alhuda-backups/$(date -u +%Y%m%dT%H%M%SZ)
    shasum -a 256 -c checksums.txt
    
    Record the run id and the three file sizes. Stop if it failed.
  3. Confirm what will run.
    supabase link --project-ref yzpfwdxpwalmfuodkxni
    supabase migration list --linked     # expect exactly the 41 in §8 as Local-only
    supabase db push --dry-run
    
    Stop on any Remote-only version or any older Local-only one.
  4. Capture the "before" numbers (read-only, SQL editor), so §5's findings can be compared afterwards:
    select count(*) from "Booking";
    select count(*) from "RolePermission";
    select polname from pg_policy where polrelid = 'public."AuditLog"'::regclass;
    
    That last one should list auth_access — if it does not, production has changed since 2026-09-18 and §3.1 needs re-checking before you proceed.
  5. Apply. supabase db push — no --include-all. Expect 2–5 minutes. If a file fails: stop, read the error, decide forward-fix vs restore with the Checker. Never edit an applied file.
  6. Verify the two fixes landed.
    select polname from pg_policy where polrelid = 'public."AuditLog"'::regclass;
    -- expect exactly: AuditLog select audit viewers, AuditLog insert authenticated
    select count(*) from "RolePermission" rp join "Role" r on r.id = rp."roleId"
     where r.name::text in ('AGENT','CUSTOMER');                    -- expect 0
    select count(*) from "UserRole" ur join "Role" r on r.id = ur."roleId"
     where r.name::text = 'SUPER_ADMIN';                            -- expect >= 1
    
  7. Deploy edge functions and the frontend immediately (runbook §5–6). The database is ahead of the UI until this finishes — see §7.
  8. Smoke-test as a human (runbook §7): sign in as the owner, open a booking with passengers, a group, a customer 360, a finance report and an audit timeline. Then sign in as a partner and confirm the portal still works and /admin and the audit log are refused.
  9. Run the drift report and hand it over.
    select public.inventory_drift_report();
    
    Expect the one FIT row from §5.1. Give it to operations with §5.2's group counts; repair only with approval (AUD-010).
  10. Record the release (runbook §8) and tell the owner about the CEO/GM permission change before they find it.

If it goes wrong

Symptom Action
A migration fails part-way Nothing after it ran. Forward-fix with a new migration; do not edit the applied file.
Someone is locked out Forward-fix migration restoring the specific grant — or, faster, admin@alhuda.co.in (SUPER_ADMIN) grants it through the admin UI and it gets recorded.
Data damaged Restore the step-2 dump per backup-restore.md; export anything written since first. Management approves.

10. Repo state

Check Result
CI=true npx vitest run 1,477 passed (108 files)
npm run typecheck:ratchet 22 errors, 22 in baseline — no new errors
node scripts/check-migrations.mjs 304 files, 0 errors, 0 warnings (+249 grandfathered)
scripts/db-test.sh on a freshly migrated DB 18/18 green

11. Follow-ups this rehearsal opened

Ordered by how much they would have helped today.

  1. Back up the migration ledger. Add supabase_migrations.schema_migrations to the backup workflow — without it a restore cannot be migrated forward (§1.1).
  2. Detect out-of-band schema changes. supabase migration list cannot see a hand-made policy (§3.1). A CI job that diffs the production catalogue against a database built from migrations alone would have caught it months ago.
  3. Make the four listed DB suites hermetic so db-test.sh can be run against a restored copy of production (§4).
  4. group_counter_drift(), to close the AUD-010 gap for group counts vs passengers, plus an approved one-off recount (§5.2).
  5. Investigate whoever wrote cancelledChargedSeats = 3000 on a 3-seat block (§5.1) — a counter that holds an amount suggests a bug that is still live.
  6. Access review (ACC-042): 7 auth-only accounts, 1 app-only account, 1 active user with no role (§6).
  7. Document the GoTrue-version caveat in the restore runbook (§1.2).

Rehearsal run 2026-09-18 on a local Postgres (project alhuda-rehearsal, ports 593xx), torn down afterwards. No connection was made to the production project. The restored data and the dump live only under the session scratchpad, outside the repository, and are deleted with it.