Skip to content

docs(db): Generate the schema, and move its rationale into the schema - #193

Merged
bitbiter-dev merged 11 commits into
masterfrom
chore/schema-documentation
Sep 28, 2026
Merged

bitbiter-dev merged 11 commits into
masterfrom
chore/schema-documentation

Conversation

@bitbiter-dev

Copy link
Copy Markdown
Owner

Why

The hand-written description of the database had drifted, and nothing could tell:

  • Invite was missing from the ER diagram entirely — an entire table, in both the wiki's copy and a separate documentation repository's copy
  • Columns were named in snake_case throughout; the model has only ever emitted PascalCase. The one explicitly-named column in the whole schema is Invites.xmin
  • live_photo_pair_id does not exist — the column is PairedAssetId
  • Metadata was documented as carrying a colour space. There is no such column
  • ProxyFile was documented as having a blurhash string. There is no such column — BlurHash is a ProxyType value whose content is a file
  • The unique index on (StorageConfigId, FilePath) appeared in no document at all

None of that was undocumented. It was duplicated — restated by hand while the authoritative copy sat in AnichronDbContextModelSnapshot.cs, committed and diffed in every PR. The copy rotted because nothing compared it to anything.

What changed

Three layers, none of them hand-written prose about columns:

Layer Source Committed?
Facts dotnet ef dbcontext script → docs/schema.sql yes, CI-gated
Rationale .HasComment(...) → COMMENT ON in the DDL yes, via the model
Navigable view tbls against a throwaway database no — CI artefact

scripts/derive-db-docs.sh calls a tool; it renders nothing itself. --check is the CI gate.

Rationale now lives in the schema

COMMENT ON COLUMN "MediaAssets"."ContentHash" IS 'XXHash64 of the file contents.
Dedup is deliberately CONFIG-SCOPED: the same bytes under two storage configs are
two assets, which is what makes multi-user work. A global unique index on this
column alone would break that and must never be added.';

24 such comments. They travel into PostgreSQL, so any schema-doc tool pointed at a live instance picks them up. There is no separate prose file left to drift.

What was deleted, and why it mattered

A first attempt generated Markdown from the EF model in ~200 lines of C# with a test asserting the committed file matched. It worked. It was still wrong: a third copy of the same facts, and a bespoke renderer to own forever, in a space where mature tools exist.

Do not hand-build a layer a maintained tool already covers, and do not restate a fact version control already holds.

Trade-offs, stated plainly

  • ⚠️ A one-word comment fix is now a migration, not a text edit. That is the price of putting rationale in the schema. AddSchemaComments is 48 comment alterations and zero structural statements, so it carries no data risk.
  • ⚠️ EF exposes no HasComment for an index, so index reasoning stays in code comments and does not reach the database.
  • ⚠️ We give up a browsable schema document in the repository. Deliberate: a committed human-readable copy is exactly the thing that drifted.
  • ⚠️ database-docs is continue-on-error: true and not a required check — it documents, it does not gate.

Two bugs found and fixed while writing the tests

  1. --check overwrote the file it was checking. It regenerated in place then asked git, so a hand-edited schema.sql was silently repaired and reported "current" — it only ever caught a stale commit. Now it derives to a sibling file and diffs, leaving the target untouched, so CI no longer mutates its own checkout either.
  2. A test assertion expected CHANGED after repairing an uncommitted edit. The generator reports against HEAD, and the file correctly matched HEAD again.

The comparison file sits beside the target rather than in /tmp or behind <(...): process substitution hands diff a /dev/fd path some sandboxes refuse, and a fixed temp path collides between concurrent runs. All three variants were tried; both alternatives fail as Operation not permitted, which reads like a broken script rather than like drift.

Verification

  • dotnet build src/ — 0 warnings, 0 errors (Debug and Release)
  • dotnet test src/ — 635/635
  • dotnet format --verify-no-changes — clean
  • All 5 script suites pass; derive-db-docs.test.sh adds 12 assertions including the failure path
  • ⚠️ Not verified locally: the database-docs job needs a live PostgreSQL and Docker. Its first real run is on CI.

See docs/adr/0004-schema-documentation.md.

🤖 Generated with Claude Code

bitbiter-dev and others added 11 commits September 27, 2026 16:55
The hand-written description of the database had drifted and nothing could tell.
Invite was missing from the ER diagram entirely. Columns were named in snake_case
throughout; the model has only ever emitted PascalCase. `live_photo_pair_id` does
not exist — the column is PairedAssetId. Metadata was documented as carrying a
colour space, and ProxyFile a blurhash string; neither column exists. The unique
index on (StorageConfigId, FilePath) appeared in no document at all.

None of that was undocumented. It was DUPLICATED — restated by hand while the
authoritative copy sat in AnichronDbContextModelSnapshot.cs, committed and diffed
in every pull request. The copy rotted because nothing compared it to anything.

Three layers now, none of them hand-written prose about columns:

- Facts: docs/schema.sql, `dotnet ef dbcontext script` output, committed. CI
  regenerates it and fails on any change.
- Rationale: PostgreSQL COMMENTs via .HasComment(...) in AnichronDbContext, beside
  the Fluent config they explain. They travel into the database and therefore into
  docs/schema.sql and any schema-doc tool pointed at a live instance.
- Navigable view: an ER diagram from tbls against a throwaway database, published
  as a CI artefact and never committed — an artefact cannot go stale.

⚠️ A first attempt generated Markdown from the EF model in ~200 lines of C# with a
test asserting the file matched. It worked, and it was still wrong: a third copy of
the same facts, and a bespoke renderer to own forever, in a space where mature tools
exist. It is deleted. The rule: do not hand-build a layer a maintained tool covers,
and do not restate a fact version control already holds.

⚠️ AddSchemaComments is 48 comment alterations and zero structural statements, so it
carries no data risk — but a one-word comment fix is now a migration, not a text
edit. That trade is recorded in ADR-0004.

The --check gate regenerates in place and asks git, rather than diffing against a
temp file: process substitution hands diff a /dev/fd path that some sandboxes refuse,
and a fixed temp path collides between concurrent runs. Both fail in ways that read
as a broken script rather than as drift.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gate regenerated docs/schema.sql in place and then asked git whether anything
moved. That cannot see a hand-edited file at all: the regeneration overwrote the
edit before git was consulted, so the check reported "current" on a file it had
just silently repaired. It caught a stale COMMIT and nothing else.

--check now derives to a sibling file and diffs, leaving the target untouched, so
CI no longer mutates its own checkout either.

The comparison file sits beside the target rather than in /tmp or behind `<(...)`.
Process substitution hands diff a /dev/fd path that some sandboxes and container
runtimes refuse, and a fixed temp path collides between concurrent runs — both
surface as "Operation not permitted" on the diff, which reads like a broken script
rather than like drift. Measured while writing the test suite; all three variants
were tried.

The test's repair assertion was also wrong and is corrected: the generator reports
against HEAD, not against whatever the working tree held a moment earlier, so
undoing an uncommitted edit correctly reports nothing to commit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…id SQL

Two faults, both found by CI rather than locally, and both from the same cause —
the script was verified on a machine where `dotnet ef` happened to be installed
GLOBALLY, so its real prerequisites were invisible.

1. dotnet-ef was not in dotnet-tools.json at all. On a clean runner the script
   failed with "dotnet-ef does not exist", which reads as a broken script rather
   than a missing prerequisite. It is now a pinned manifest tool, `dotnet tool
   restore` moved ahead of the steps that need it, and the script checks for the
   tool up front and says what to run.

2. `dotnet ef` writes its tools-version-mismatch notice to STDOUT, not stderr, so
   `2>/dev/null` did not keep it out of the generated file. The first committed
   docs/schema.sql therefore began with

     The Entity Framework tools version '10.0.7' is older than that of the runtime...

   and was not valid SQL. Pinning the tool to the runtime version removes the
   notice at source; a guard now rejects any first line that is not SQL, so a
   future mismatch is loud instead of silently corrupting the file.

⚠️ The generated output is tool-version-dependent, which is why the pin matters
beyond tidiness: an unpinned dotnet-ef would make the drift gate fail for whoever
happened to have a different version installed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The guard added in the previous commit read its first line with
`grep -v '^\s*$' | head -1`. `head` closes the pipe after one line, grep takes
SIGPIPE, and under `set -o pipefail` that fails the whole function — so the
generator exited 2 with "grep: write error: Broken pipe" and said nothing about
the schema.

Whether it fires depends on whether grep finished writing before head exited, so
it PASSED locally on macOS and failed on the CI runner against identical input.
A read loop has no pipeline and therefore no race.

Also fixes a latent false positive in the same guard: `CREATE*` does not match
"  CREATE TABLE", so an indented first statement would have been rejected as not
being SQL. dotnet ef does not indent it today, which is what kept that invisible.
Leading whitespace is now trimmed before the match.

Both faults are the same shape as the ones this branch already carries comments
about: a pipeline whose exit status is not what it appears to be, and a check
verified only against the one input that happened to be at hand.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…lobal install

The job installed dotnet-ef with `--global`, then failed:

  Tool 'dotnet-ef' (version '10.0.12') was successfully installed.
  Run "dotnet tool restore" to make the "dotnet-ef" command available.

Inside a repo carrying dotnet-tools.json the MANIFEST wins, so a global install is
ignored outright. The job was written before dotnet-ef was added to the manifest
two commits ago and was never revisited — the earlier fix created this one.

`dotnet tool restore` also keeps the version pinned in one place instead of two
that can disagree.

Verified against the upstream release while fixing this: the asset
tbls_v1.96.0_linux_amd64.tar.gz exists and carries `tbls` at the archive root, so
the download and extraction steps are right. The job still cannot be exercised
locally — it needs a live PostgreSQL — and remains continue-on-error so it cannot
block a merge.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…gram as mermaid

Third and final prerequisite the schema-diagram job did not inherit. It runs on a
clean checkout in its own job, so nothing has restored the solution, and
`dotnet ef database update` failed before reaching the database:

  error NETSDK1004: Assets file '.../Anichron.Core/project.assets.json' not found.
  Unable to retrieve project metadata. Ensure it's an SDK-style project.

Rather than discover the next missing step on the next round-trip, the remaining
chain was audited statically instead:

- `--connection` DOES override the design-time factory's hardcoded connection
  string. Verified by pointing it at a nonexistent host and confirming the error
  named that host, not the factory's `anichron_design`. Worth recording: the
  factory ignores its `args` parameter, which makes the opposite look true from
  reading the code.
- The POSTGRES_CONNECTION__* variables on that step were read by nobody — the
  factory never touches IConfiguration. Removed; they existed only to mislead.
- tbls' command form is `doc [DSN] [DOC_PATH]`, `--rm-dist` is a real flag, and
  the postgres:// DSN shape is right — all checked against upstream's README.

ER diagrams now render as mermaid rather than the default svg: no image-renderer
dependency, and the diagram stays readable in the artefact without downloading
it. The COMMENTs added via .HasComment(...) surface as table and column
descriptions, which is the point of having put the rationale in the schema.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The status line still read "implemented, except the ER-diagram job", written
while that job was unverifiable locally. It now runs green: tbls migrates a
throwaway Postgres and generates the mermaid ER diagram as a CI artefact.

A stale status line on the ADR that argues documentation should not drift is
not a small thing to leave behind.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@bitbiter-dev
bitbiter-dev merged commit fa7a54c into master Sep 28, 2026
7 checks passed
@bitbiter-dev
bitbiter-dev deleted the chore/schema-documentation branch September 28, 2026 17:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant