[bitbiter-dev]

Closes #337. Design: note, as amended by the decision.

Summary

At most one work item per storage config is Running, across all Workers. Unscoped items are not serialised.

 ClaimAsync
   oldest Queued item of a kind this Worker knows, in its scope
+    whose config has no Running item (NOT EXISTS)
   FOR UPDATE SKIP LOCKED → Running
+  racing second claim of one config → unique violation on
+    ix_workitems_storageconfig_running → return null, retry next poll

 RunAsync
+  selfStop = CTS(lease / 2, timeProvider), linked into the handler token
   heartbeat every interval
+    each call limited to one interval (CTS on timeProvider + WaitAsync)
+    answered → selfStop restarts from when the beat was sent
   outcome
     Lost → write nothing
+    selfStop fired → write nothing (log SelfStopped); lease recovery requeues and counts a stall
     CancelRequested / succeed / release / fail as before

• Migration only creates the index (never deployed). The plain FK index IX_WorkItems_StorageConfigId is now declared explicitly, because EF otherwise drops it in favour of the partial index.
• WorkerSettings: WorkLeaseSeconds / 2 must exceed 2 × WorkHeartbeatSeconds (was: lease > heartbeat). With each beat limited to one interval, two answered beats can be two intervals apart; a shorter half-lease would self-stop against a slow but healthy database. Defaults (60 / 2) pass.
• ADR 0008 amended (rule + self-stop), ADR 0005 row updated, docs/architecture/work-queue.md and schema.sql updated.
• Separate chore commits refresh test projects' packages.lock.json left stale by Bump the nuget group with 17 updates #353.

Evidence

• Before: ClaimAsync_AnotherClaimOfTheSameConfigCommitsFirst_… (another transaction's uncommitted claim of config C; the claim must block on the index, verified through pg_stat_activity) failed with NOT EXISTS alone: the claim took C's second item without blocking.
  After: blocks on the index and returns null after the commit. Integration suite 80/80.
• Before: heartbeats failing indefinitely left the handler running past lease recovery; a handler cancelled without shutdown fell through to FailAsync.
  After: WorkExecutorTests (Timeline): handler cancelled at half a lease and not before; a successful (or slow) beat restarts it from its send time; a heartbeat that never answers still self-stops and is given up after one interval; a self-stopped attempt writes nothing, even if its handler finishes. 33/33, stable over 20 runs.
• scripts/mutation-changed.sh: Anichron.Core 100.00 %, Anichron.Worker 100.00 %.

Merge Danger

Door: two-way

The index can be dropped in a later migration; the executor change reverts cleanly.

Blast Radius: Worker

A database outage longer than half a lease now restarts running work (a long first crawl included) and counts a stall; the WorkMaxStallsth such outage fails the item (accepted in decision 4). Workers configured with a lease ≤ 4 heartbeats now fail at startup.

🤖 Generated with Claude Code