Repository navigation
refactor(db): Postgres on effect-orm, queries and migrations - #1353
Conversation
All 71 application tables as PG.table definitions in @maple/db/tables, checked against drizzle-kit's head snapshot and against the catalog of a database the bundled migrations build. Database.execute hands services db.orm, an effect-orm database over the same client, whose statements join an open drizzle transaction and land on the call's span.
Services run their Postgres statements through db.orm with the typed @maple/db/tables definitions: rows decode through the table codecs, timestamps are epoch milliseconds end to end, ids are branded and every value is bound. Drizzle remains only under the platform layer.
…ad with PG.undecoded Service code reads stored jsonb documents it decodes itself through PG.undecoded instead of hand-written templates, and CASE/greatest/least/ boolean templates use the builder. Tests seed and read through db.orm; the typed inserts rejected fixtures drizzle had accepted (an investigation subject type that does not exist), which are fixed.
…ices effect-orm MapleDb is now effect-orm's database: services call db.run(...) and db.transaction(effect), statements reach the call's span through effect-orm's observe hook, and DatabaseError absorbs effect-orm's errors. drizzle-orm leaves backend and api; packages/db keeps drizzle-kit and its schema only to generate and apply migrations. Platform and integration tests run on effect-orm, the latter verified against a real Postgres.
…is gone The baseline is drizzle-kit's last snapshot, and the tables generate nothing against it. Migrations stay migration.sql in the same folder, so the deploy applies them unchanged. Tests and db:migrate run the effect-orm runner; db:migrate adopts a drizzle-migrated database once. check-schema-drift.ts compares a live schema with the migrations.
… extensionless imports
ai_triage_runs and ai_triage_settings.fanout_enabled left the schema without a migration dropping them. ai_triage_settings.investigation_mode (every row the default) and alert_destinations_org_id_replident_idx (not the replica identity; the table is FULL) exist only in prd, created outside the migrations. The parity test no longer allows any drift.
…for the Effect lint
|
Important Review skippedToo many files! This PR contains 235 files, which is 85 over the limit of 150. To get a review, reduce the PR to 150 files or fewer by splitting it into smaller PRs or changing its base branch. Upgrade to a paid plan to raise the limit. This review couldn't start because sufficient usage credits or metered capacity aren't available. Add credits or update usage-based reviews in the billing tab, then retry. ⚙️ Run configuration
⛔ Files ignored due to path filters (1)
📒 Files selected for processing (235)
You can disable this status message by setting the
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Note A newer push replaced |
…drop deps only drizzle used
🍁 Maple PR previewNote Preview resources were removed when this pull request closed. Final commit |
Maple review🟡 Confidence 5/10 · needs attention Moves every Postgres query and the migration tooling from drizzle-orm to
Before merge
Findings🟠 Warning · F1 ·
|
| Issue | Service | Occurrences | File |
|---|---|---|---|
| @maple/http/errors/WarehouseAuthError | maple-ai | 20 | packages/backend/src/platform/DatabaseLive.ts |
| @maple/http/errors/WarehouseQueryError | maple-ai | 16 | packages/backend/src/platform/DatabaseLive.ts |
| @maple/api/errors/McpOAuthProtocolError | maple-api | 15 | packages/backend/src/services/org/OnboardingChecklistService.ts |
| TypeError | maple-api | 10 | packages/backend/src/services/org/OnboardingChecklistService.ts |
| @maple/http/errors/WarehouseQueryError | maple-chat | 8 | packages/backend/src/platform/DatabaseLive.ts |
| @maple/http/errors/WarehouseQueryError | maple-ai | 8 | packages/backend/src/platform/DatabaseLive.ts |
| @maple/http/errors/WarehouseQueryError | maple-chat | 8 | packages/backend/src/platform/DatabaseLive.ts |
| @maple/http/errors/WarehouseAuthError | maple-chat | 8 | packages/backend/src/platform/DatabaseLive.ts |
| @maple/http/errors/WarehouseUpstreamError | maple-cli | 8 | packages/db/src/schema/errors.ts |
| @maple/http/errors/WarehouseQueryError | maple-ai | 8 | packages/backend/src/platform/DatabaseLive.ts |
After this merges, Maple checks whether they stop.
Production traffic of the changed files (last 7 days)
| File | Calls/day | Busiest operations |
|---|---|---|
.env.example |
325k | maple 325k/day, 99.3% err |
packages/backend/src/services/alerts/AlertsService.ts |
289k | AlertsService.processEvaluation 284k/day, 0.0% errAlertsService.processQueuedDeliveries 2.8k/day, 0.0% errAlertsService.runSchedulerTick 2.8k/day, 0.0% err |
packages/backend/src/services/alerts/AlertRuleModel.ts |
173k | AlertsService.normalizeRuleRow 173k/day, 0.0% errAlertsService.normalizeRuleRow 2/day, 0.0% errAlertsService.normalizeRuleRow 1/day, 0.0% err |
packages/backend/src/services/errors/ErrorsService.ts |
42k | ErrorsService.processOrg 39k/day, 0.3% errErrorsService.processNotificationOutbox 2.8k/day, 0.0% err |
packages/db/src/tables/scrape-targets.ts |
8.2k | POST /api/internal/scrape-results 8.2k/day, 0.0% err |
packages/backend/src/services/alerts/AnomalyDetectionService.ts |
7.0k | AnomalyDetectionService.loadSettingsRow 7.0k/day, 0.0% errAnomalyDetectionService.listIncidents 11/day, 0.0% err |
packages/backend/src/services/errors/IssueFixVerificationService.ts |
5.6k | IssueFixVerification.settledRuns 2.8k/day, 0.0% errIssueFixVerification.dueVerifications 2.8k/day, 0.0% errIssueFixVerification.hydrateLinks 1/day, 0.0% err |
packages/backend/src/services/auth/oauth/connection-helpers.ts |
5.0k | OAuthConnectionHelpers.requireConnection 5.0k/day, 0.0% err |
packages/backend/src/platform/span-name.ts |
4.4k | SELECT alert_rules 2.9k/day, 0.0% errDatabase.execute 1.2k/day, 0.0% errSELECT alert_rules 238/day, 0.0% err |
packages/backend/src/services/alerts/EscalationService.ts |
2.8k | EscalationService.runEscalationTick 2.8k/day, 0.0% errEscalationService.processOne 11/day, 0.0% err |
packages/db/src/client.ts |
1.2k | Database.execute 1.2k/day, 0.0% err |
packages/backend/src/services/integrations/RailwayMetricsService.ts |
1.2k | RailwayMetricsService.emitMetrics 647/day, 0.0% errRailwayMetricsService.pollAllOrgs 504/day, 0.0% err |
packages/backend/src/services/integrations/CloudflareAnalyticsService.ts |
579 | CloudflareAnalyticsService.pollAllOrgs 579/day, 0.0% err |
packages/backend/src/services/integrations/PlanetScaleService.ts |
579 | PlanetScaleService.pollAllOrgs 579/day, 0.0% err |
packages/backend/src/services/integrations/vcs/VcsRepository.ts |
370 | VcsRepository.getPrReviewSettings 191/day, 0.0% errVcsRepository.getPrReviewSettings 174/day, 0.0% errVcsRepository.listAllInstallations 5/day, 0.0% err |
What was checked
selectDistinctOrgIdskeeps the loose index scan and now takes the table, not a column (distinct-org-ids.ts:31)toDatabaseErroranddriverSqlErrorstill unwrap the driverSqlErrorfor SQLSTATE classification (DatabaseLive.ts:81)- Span capture survives the logger swap:
DatabaseOrm.test.ts:63assertsSELECT api_keysanddb.query.textwith$1, never the value
Observability coverage: 1 of 1 changes observable
| Change | Kind | Observable | Evidence |
|---|---|---|---|
| SQL statement per Database.execute call | db | yes | makeMapleDb observe -> MapleStatementCollector; DatabaseOrm.test.ts asserts span name and db.query.text |
Files not reviewed (136)
The review ended before it read these diffs, so nothing above vouches for them.
.env.example.github/workflows/ingest-rust-tests.yml.oxfmtrc.jsoncapps/api/package.jsonapps/api/src/routes/scraper-internal.router.test.tsapps/api/src/routes/webhooks/planetscale.http.test.tsapps/api/test/integration/pg-connection-scope.integration.test.tsknip.jsonpackages/backend/package.jsonpackages/backend/src/platform/DatabaseLive.test.tspackages/backend/src/platform/DatabasePgLive.test.tspackages/backend/src/platform/DatabasePgliteLive.tspackages/backend/src/platform/distinct-org-ids.test.tspackages/backend/src/platform/pg-connection-scope.test.tspackages/backend/src/platform/postgres-errors.test.tspackages/backend/src/platform/raw-rows.tspackages/backend/src/platform/span-name.tspackages/backend/src/services/alerts/AlertDeliveryDispatch.providers.test.tspackages/backend/src/services/alerts/AlertDeliveryDispatch.test.tspackages/backend/src/services/alerts/AlertDestinationHydration.test.tspackages/backend/src/services/alerts/EscalationService.test.tspackages/backend/src/services/alerts/anomaly/detector-state-batch.tspackages/backend/src/services/alerts/delivery/context.tspackages/backend/src/services/alerts/delivery/delivery-spans.test.tspackages/backend/src/services/alerts/delivery/transports/chat.test.tspackages/backend/src/services/alerts/delivery/transports/render.test.tspackages/backend/src/services/alerts/delivery/transports/telegram.test.tspackages/backend/src/services/auth/MembershipRevocationService.test.tspackages/backend/src/services/auth/oauth/connection-helpers.test.tspackages/backend/src/services/digest/DigestService.test.ts- and 106 more
6260c07 · Updated on every push. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.
…y again Both reads tolerated a stored document in another shape before the migration: a strict column decode failed the whole tick or the whole target list instead of the one row.
Maple review🟢 Confidence 9/10 · safe to merge The last commit's hunks read stored jsonb leniently on the two paths that touch it — pr-review telemetry and scrape-target labels — and carry epoch-ms timestamps with
Before merge
Fixed since the last review
Production impactOpen errors in the changed files
After this merges, Maple checks whether they stop. Production traffic of the changed files (last 7 days)
What was checked
|
|
On the before-merge checklist:
|
Summary
Postgres moves from drizzle-orm to
@maple-dev/effect-orm0.5.0, for queries and for migrations. Every query is typed by its table, including the ones that used to be inline raw SQL.Tables and queries
packages/db/src/tables(@maple/db/tables), with branded ids, literal-union enums, schema-typed jsonb and epoch-ms timestamps (PG.timestamptzMillis).db.run(PG.from(...))anddb.transaction(effect)throughDatabase.execute, which still opens one span per call and records the statement.$n, not written into the SQL.db.query.textno longer carries key hashes or emails.PG.sqltemplates with typed results: advisory locks,UPDATE ... FROM (VALUES), jsonb operators.=== null, never by truthiness, so epoch 0 isn't read as "not set". The three places that did this are fixed.PG.undecodedand decode it themselves, so a bad stored document can't turn into a 503.Migrations
20261009195715_effect_orm_baselineis drizzle-kit's last snapshot, and generating against it produces nothing, so the tables match the existing migrations exactly.effect-orm.config.tsusesemit: "sql", so new migrations are<ts>_<name>/migration.sqlin the same folder. The deploy applies the folder exactly as before.db:generate,db:check,db:statusanddb:verifyrun the effect-orm kit.db:migrateruns the effect-orm runner. On a database drizzle-kit migrated, the first run records drizzle's applied migrations as done, then applies only the newer ones.20261009202241_drop_schema_leftoversdrops four objects that no table definition has and nothing uses. All four useIF EXISTS.ai_triage_runsandai_triage_settings.fanout_enabled: left behind by schema changes that never dropped them.ai_triage_settings.investigation_modeandalert_destinations_org_id_replident_idx: exist only in prd. Everyinvestigation_modevalue is the default. The index isn't the replica-identity index; the table stays REPLICA IDENTITY FULL.Verification
parity.test.tschecks the tables against the newest snapshot, and compares a database built by the migrations with one built from the definitions.PgConnectionScopeintegration test passes 9/9.scripts/check-row-decoding.ts).scripts/check-schema-drift.ts).db:migratehad just adopted.Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.