Skip to content

fix(cloudflare): wake-ups from alarm callbacks run on time - #2916

Merged
sam-goodwin merged 12 commits into
mainfrom
fix/do-engine-wake-backoff
Nov 2, 2026
Merged

sam-goodwin merged 12 commits into
mainfrom
fix/do-engine-wake-backoff

Conversation

@sam-goodwin

@sam-goodwin sam-goodwin commented Nov 1, 2026 •

Copy link
Copy Markdown
Member

Wake-ups scheduled from inside a Durable Object alarm callback inherited that callback's retry backoff, with nothing failing. Workflow sleeps and Domain projection batches came in late, and each hop's delay doubled.

// four one-second sleeps: previously still running after 45s, now ~4s
for (let nap = 0; nap < count; nap++) {
  yield* DurableClock.sleep({ name: `nap-${nap}`, duration: "1 second", inMemoryThreshold: 0 });
}

Bugs fixed

15 bugs, each pinned by a test that fails on main (except #2, found in review), plus two platform limits that used to hang and now fail fast. The conformance suite found 12 of the 15.

# Where Before
1 DurableObjectEngine A wake-up after a successful step waited out a retry backoff that doubled on every hop (#2915)
2 DurableObjectEngine A deferred's write and its wake-up weren't atomic; a second completion could hand a pending wake-up a backoff
3 Callback scheduler A schedule carrying a lower cursor than the running callback's was silently dropped
4 Domain A drain started with waitUntil inside a callback lost its wake-up; batches waited for the backstop (35s, not 5s)
5 Domain A batch wait scheduled inside a running drain inherited a 30s backoff (47s, not ~19s)
6 NativeEngine A signal or interrupt to a finished run retried for a minute, then died
7 NativeEngine Interrupting a sleeping run did nothing
8 NativeEngine A defect the workflow doesn't capture ended the run instead of being retried
9 NativeEngine A run suspended on failure died, and resume did nothing
10 DurableFunctionEngine A repeated or late signal hung for about 3 minutes, then died
11 DurableFunctionEngine An interrupt sent right after start was silently dropped
12 DurableFunctionEngine An interrupt sent before the run reached its wait stopped it without running compensations
13 DurableFunctionEngine Activity.retry saw attempt 1 on every retry
14 DurableFunctionEngine A sleep or signal inside an activity killed the run
15 DurableFunctionEngine A run suspended on failure died, and resume did nothing

Activity.raceAll hung forever on NativeEngine and DurableFunctionEngine. Neither platform can read a step's result before running it, so a race can't replay its winner there. Both now die at once with a message that points to the Durable Object engine.

DurableObjectEngine

The engine keeps a journal cursor and passes it as the callback progress. The cursor update commits in the same transaction as the journal write and the wake-up:

// a newly completed deferred wakes the run with progress
if (inserted.length > 0) return yield* wake(yield* advance);
// a new clock is scheduled with progress; a pending one is left alone on replay
yield* clocks.schedule(name, { at: due, payload: { deferred }, progress: yield* advance });

A deferred completed a second time reports no progress and leaves a pending wake-up alone, so the limit on self-rearms without progress (#1908) still holds.

One conformance suite for every engine

test/Workflows/conformance/ holds the workflows, the routes, and 28 cases, drawn from Effect's own engine tests and from bugs found here. Each engine's suite deploys a fixture and calls it:

conformance({
  test,
  url: Effect.map(stack, ({ url }) => url),
  napsWithinMillis: 20_000,
  recoveryWithinMillis: 30_000, // unset where the platform owns retry timing
  timeout: 90_000,
  unsupported: { raceAll: "aren't supported on Cloudflare Workflows" }, // checked, never skipped
});

It runs on the Durable Object engine (also with the EU jurisdiction) and the native engine, each locally and live, and on Lambda.WorkflowFunction. It found #6–#15 and the raceAll limit.

Callback scheduler

A cursor from a different callback no longer hides a schedule. It used to drop it silently:

// inside callback "source" (progress 100)
yield* target.schedule("a", { after: 0, payload, progress: 1 });
// before: skipped, because 1 < 100. after: scheduled, inheriting the source's budget

Domain

  • A drain handed to waitUntil from inside a callback lost its wake-up once that invocation ended. A batched projection's first input waited for the backstop, about 35s instead of the 5s batch wait. These drains now run outside the alarm pass.
  • The projection and outbox backstops report progress. A batch that starts while the drain's callback is running waits its batch wait, not a 30s backoff.

Closes #2915

@alchemy-version-bot

alchemy-version-bot Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Install the packages built from this commit:

alchemy

pnpm install https://pkg.alchemy.run/alchemy/pr:2916:dac138d
@alchemy.run (17)
pnpm install https://pkg.alchemy.run/@alchemy.run/better-auth/pr:2916:dac138d
pnpm install https://pkg.alchemy.run/@alchemy.run/git/pr:2916:dac138d
pnpm install https://pkg.alchemy.run/@alchemy.run/drizzle/pr:2916:dac138d
pnpm install https://pkg.alchemy.run/@alchemy.run/acme/pr:2916:dac138d
pnpm install https://pkg.alchemy.run/@alchemy.run/planetscale/pr:2916:dac138d
pnpm install https://pkg.alchemy.run/@alchemy.run/neon/pr:2916:dac138d
pnpm install https://pkg.alchemy.run/@alchemy.run/cloudflare/pr:2916:dac138d
pnpm install https://pkg.alchemy.run/@alchemy.run/aws/pr:2916:dac138d
pnpm install https://pkg.alchemy.run/@alchemy.run/domain/pr:2916:dac138d
pnpm install https://pkg.alchemy.run/@alchemy.run/kubernetes/pr:2916:dac138d
pnpm install https://pkg.alchemy.run/@alchemy.run/docker/pr:2916:dac138d
pnpm install https://pkg.alchemy.run/@alchemy.run/redis/pr:2916:dac138d
pnpm install https://pkg.alchemy.run/@alchemy.run/sql/pr:2916:dac138d
pnpm install https://pkg.alchemy.run/@alchemy.run/cloudflare-runtime/pr:2916:dac138d
pnpm install https://pkg.alchemy.run/@alchemy.run/frontend-frameworks/pr:2916:dac138d
pnpm install https://pkg.alchemy.run/@alchemy.run/node-utils/pr:2916:dac138d
pnpm install https://pkg.alchemy.run/@alchemy.run/pkg/pr:2916:dac138d
Transitive Dependencies (9)
pnpm install https://pkg.alchemy.run/@distilled.cloud/aws/pr:2916:dac138d
pnpm install https://pkg.alchemy.run/@distilled.cloud/acme/pr:2916:dac138d
pnpm install https://pkg.alchemy.run/@distilled.cloud/core/pr:2916:dac138d
pnpm install https://pkg.alchemy.run/@distilled.cloud/planetscale/pr:2916:dac138d
pnpm install https://pkg.alchemy.run/@distilled.cloud/neon/pr:2916:dac138d
pnpm install https://pkg.alchemy.run/@distilled.cloud/github/pr:2916:dac138d
pnpm install https://pkg.alchemy.run/@distilled.cloud/aws4signer/pr:2916:dac138d
pnpm install https://pkg.alchemy.run/@distilled.cloud/cloudflare/pr:2916:dac138d
pnpm install https://pkg.alchemy.run/@distilled.cloud/iceberg/pr:2916:dac138d

Published Nov 2, 2026, 2:52 AM UTC. Expires Nov 9, 2026, 2:52 AM UTC, extended while this pull request is open.

- background drains run apart from the callback invocation that started them,
  so their wake-ups aren't rejected once it ends
- projection and outbox backstops report progress
@sam-goodwin sam-goodwin changed the title fix(cloudflare): DurableObjectEngine wake-ups after a successful step run on time fix(cloudflare): wake-ups from alarm callbacks run on time Nov 1, 2026
@sam-goodwin
sam-goodwin merged commit d08dc60 into main Nov 2, 2026
4 checks passed
@sam-goodwin
sam-goodwin deleted the fix/do-engine-wake-backoff branch November 2, 2026 03:15
sam-goodwin added a commit that referenced this pull request Nov 2, 2026
Port #2916 (alarm-callback progress, workflow progress cursor) into the
shared alchemy/Workers runtime.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Workflows: a wake-up scheduled from a callback pass inherits that pass's attempts and retry backoff

1 participant