Document the worker-function enable fence for production deployments

[jumski]

Problem

The production Update Deployed Flows guide currently says to:

1. deprecate old workers;
2. deploy the new function;
3. wait for cron to start it.

It does not disable the corresponding pgflow.worker_functions row before deprecation. While that row remains enabled, ensure_workers() may start another instance of the old deployed function between deprecation and deployment.

The existing proposed fix records only rows that are already enabled and later re-enables that set. It cannot enable a new worker function because no row existed to record. Pre-inserting the new row disabled would also leave it disabled under that rule.

Private per-step queues add several worker functions to one concrete flow. Versioned private queues create a different case: old-version workers must remain alive while old runs drain. One fence cannot safely treat in-place replacement and new-version rollout as the same operation.

Deployment manifest

Require an explicit manifest for every deployment. Each entry states:

function_name
concrete flow or queue role
action: replace | add | keep-draining | retire
desired final enabled state

The manifest includes every old and new function affected by the deployment. It is the source for pre-insertion, disabling, waiting, final enablement, and health checks.

Before code deployment, upsert every new manifest function with enabled = false. track_worker_function() preserves that disabled state on conflict. Do not infer final state only from rows that existed before the deployment.

Sequence A — replace compatible code on the same concrete slug and queue

Use this sequence for handler or worker-runtime changes that preserve the concrete flow's structural shape, queue mode, route map, and semantic input/output contract. Shape, route, or contract changes require a new concrete slug and Sequence B.

1. Upsert every manifest function disabled.
2. Set every function marked replace to enabled = false.
3. Deprecate its live worker rows.
4. Wait until each affected worker has stopped polling and finished in-flight callbacks.
5. Deploy every replacement while its row remains disabled.
6. Set each manifest function to its declared final enabled state.
7. Confirm every expected queue has the required live workers and current heartbeats.

No old worker may remain able to poll when the replacement becomes enabled.

Sequence B — deploy a new concrete version

Generated private queues use concrete slugs, so old and new versions are isolated. Do not stop old-version workers merely because V2 is deployed.

1. Keep every old-version function enabled while its runs can still reach its private queues.
2. Upsert all new-version manifest functions disabled.
3. Deploy the complete new-version worker set.
4. Enable the new functions and confirm coverage for every new queue.
5. Switch new starts through the caller or the alias activation operation from Add stable flow aliases through DeployedFlow startup deployment #648.
6. Monitor old runs until no executable or recoverable old-version work remains.
7. Disable and deprecate the old functions only after the drain check passes.

A missing new worker blocks the switch but does not require stopping healthy old workers.

Complete affected set

For in-place replacement, fence every function that can consume the same physical queues or depends on the changed handler/runtime contract.

For a private per-step flow, an isolated compatible handler change may fence only that step's worker function. Fence the complete concrete-flow worker set when the deployment changes shared runtime code or a contract imported by several step workers.

Flow shape, queue mode, route map, or semantic input/output contract changes use a new concrete slug and Sequence B. Do not stop old-version workers while their private queues drain.

For a shared queue, deploy registries containing both old and new concrete versions until the old version drains.

Fatal worker pause

When a worker stops because it read unsupported work, it must persistently disable or pause its HTTP worker-function row before shutdown. Otherwise ensure_workers() creates a restart loop.

Document the operator checks and explicit re-enable step after the queue or registry mismatch is repaired.

Durable waiting and checks

A flow definition may commit before every replacement worker starts. An alias switch must wait for new queue coverage, but tasks already routed to a missing worker remain durable.

Provide copyable SQL driven by the manifest for:

• upserting new functions disabled;
• disabling replacement functions;
• deprecating and waiting for old workers;
• applying declared final enabled states;
• checking expected queue coverage and current heartbeats;
• checking executable or recoverable work before old-version retirement;
• inspecting functions paused after fatal unsupported work.

Do not add a second activation protocol.

Acceptance criteria

• The guide requires an explicit old/new function manifest with desired final states.
• New functions are inserted disabled before deployment and can become enabled afterward.
• In-place replacement disables, drains, deploys, and re-enables the complete affected set.
• New concrete-version rollout leaves old private-queue workers alive until old work drains.
• Alias or caller switching happens only after every new queue has healthy coverage.
• Per-step deployments account for every worker function for the concrete flow.
• Shared-queue deployments keep old and new concrete versions in registries during drain.
• Unsupported-work shutdown persists a pause that prevents automatic restart loops.
• All manifest, wait, coverage, drain, and remediation queries are copyable.
• Worker-management, versioning, private per-step queue, and production-update pages link to these two sequences.

Out of scope

• Cross-worker activation transactions.
• Automatic deployment orchestration.
• Automatic old-version removal.