STOR-5615: Add configurable Durable Object retries by apeacock1991 · Pull Request #7383 · cloudflare/workerd

apeacock1991 · 2026-09-16T08:59:56Z

Summary

Add an optional retryPolicy to Durable Object namespace bindings and carry it into minted stubs and jurisdictional subnamespaces.
Support a configurable retry count and retry timeout for eligible fetch() calls.
Make the runtime's default retry timeout cancel a retry that is still running when the timeout expires.
Add the off-by-default DURABLE_OBJECT_RETRIES_USERLAND autogate.
Cap full-jitter retry backoff at two seconds.

Behavior

A retry policy has two fields:

maxAttempts is the number of retries after the initial attempt, not the total number of attempts. It defaults to four, matching the runtime default of five total attempts, and is capped at ten. Zero disables retries.
timeoutMs is the retry timeout in milliseconds. It defaults to 10,000, matching the runtime default, and must be between 500 and 60,000. The first retry's backoff can be up to 500ms, so a shorter timeout would usually mean no retry at all. The minimum also rules out zero, which people would read as "no retries". maxAttempts = 0 does that.

The timeout limits retrying, not the whole call. It works the same way whether it comes from the runtime or the binding. It runs from the start of the call, after the output-gate wait. A retry only starts if its backoff ends inside the timeout, and a retry still running when the timeout expires is cancelled. The caller then gets the disconnect that caused the first retry. The initial request always runs to completion. A slow request that eventually succeeds isn't a failure, so the timeout never cuts it off.

A redirect starts a new actor call with a fresh retry count, retry token, isRetry = false, and subrequest accounting. Its first request isn't a retry, so the timeout never cancels it either. It does share the original call's timeout, so a redirect chain can't keep resetting it.

The binding policy applies only when DURABLE_OBJECT_RETRIES_USERLAND and both existing fetch-retry gates are enabled. If any gate is off, or the binding has no policy, the call uses the runtime defaults.

Default change

The first commit changes the default for every call with fetch retries enabled, whether or not the new gate is on. Previously the 10-second window only decided whether a retry could start, so a retry started at 9.9 seconds could run for as long as the actor took to answer. That retry is now cancelled at 10 seconds. The window also used to reset on each redirect, and now covers the whole chain.

Callers see this in two cases: the error arrives sooner, or a retry that would have succeeded after the 10-second mark now fails with the original disconnect.

This PR does not enable the new gate.

maxmcd

lgtm, just a few nits and a concern about the autogate-enabled part of this feature

Cap the randomized backoff between actor call retries at two seconds. Without a cap the doubling window reaches 800ms by the fifth attempt and grows past the retry budget once attempt counts become configurable, which would waste most of a caller's budget waiting. Add fetch tests for redirects through Durable Object stubs. Each redirect hop is already treated as a new actor call with a fresh retry token and first-attempt subrequest accounting, and a redirected streaming body becomes replayable and so retryable. Nothing today asserts any of that. Configurable retry counts and durations change how limits are measured across hops, so the current per-hop behavior needs to be pinned down first.

Add an optional `retryPolicy` group to Durable Object namespace bindings with a `maxAttempts` field, and carry it into minted stubs and jurisdictional subnamespaces as a UserDefinedRetryPolicy. It counts retries after the initial attempt: zero disables retries, and one allows a single retry. Reject values above ten at config load. Resolve the policy once per fetch into an ActorRetryPolicy. A binding without a retry policy always runs under the runtime's five-attempt default. A binding with one uses it when the new off-by-default DURABLE_OBJECT_RETRIES_USERLAND autogate and both fetch retry gates are on, and falls back to the default otherwise. The gates are checked before the stub's I/O objects are touched so a disabled rollout never reaches them. `maxAttempts` defaults to four so a policy that sets only other fields still yields five attempts in total, and enabling the gate changes nothing for bindings that have not opted in. ActorCallRetryState reads the resolved policy and never asks where it came from. Consolidate the two retries-exhausted exits into one helper that records the outcome and returns the original disconnect. JSRPC retries share ActorCallRetryState and use the runtime's default policy.

The 10-second retry timeout used to decide only whether a retry could start. A retry that started just inside it could then run for as long as the actor took to answer, so the timeout said little about how long a failing call could take. The timeout now runs from the start of the call, after any output-gate wait. No retry starts unless its backoff ends inside the timeout, and a retry still running when the timeout expires is cancelled. The caller gets the disconnect that caused the first retry, as it already does when retries run out. The first attempt is never cancelled, whatever the timeout. A slow but healthy request is not a failure, and cutting it off would turn a working call into an error. A redirect is not a retry either, so the first attempt after a redirect is also left alone. It does share the original call's timeout, so a chain of redirects cannot keep resetting it. JSRPC retries are unchanged. Their timeout starts when their retry state is created, as before, and still only decides whether a retry can start. Add TimerChannel::atLimitTimeout() so the cutoff can be expressed as an absolute point on the limit-timeout clock.

Add `timeoutMs` to the Durable Object namespace binding's `retryPolicy`. When set, it replaces the runtime's 10-second retry timeout for calls through that binding. The timeout means the same thing either way: it runs from the start of the call, no retry starts after it, and a retry still running when it expires is cancelled. The field defaults to 10,000 rather than using zero to mean "unset". Zero reads as "no retries", which is what `maxAttempts = 0` is for. Values must be between 500ms and 60 seconds. The first retry's backoff can be up to 500ms, so a shorter timeout would usually stop the call from retrying at all. ActorRetryPolicy now carries the timeout alongside the attempt count, so ActorCallRetryState reads both from one place. A call that cannot retry is unaffected by any timeout, because the first attempt is never cancelled. The timeout starts after the output-gate wait. Covering the wait would mean resolving the target stub's policy before the Request exists. Not worth it. The wait is bounded by storage, not by this feature.

Forward namespace retry policy through ctx.exports loopback bindings so stubs minted from them retain the binding's retry limits. The parameter has no default. Standalone workerd passes none, because its ctx.exports namespaces come from the class rather than a configured binding.

Wrangler dev maps `retry` to the new `retryMaxAttempts` and `retryTimeoutMs` Durable Object options. Miniflare writes them to workerd's `DurableObjectNamespaceDesignator.retryPolicy`, and keeps them when the dev registry proxy reroutes a binding to a Worker in another process. The experimental config API accepts `retry: { maxAttempts, timeoutMs }` and converts it to Wrangler's shape. The generated Cap'n Proto types come from workerd.capnp with the `RetryPolicy` struct from cloudflare/workerd#7383, which no workerd release contains yet. Current workerd ignores the field, so local dev still starts. Regenerating after the next workerd bump gives the same file.

Add a `retry` option to DurableObjectNamespace's get(), getByName() and getExisting(): env.MY_DO.get(id, { retry: { maxAttempts: 5, timeoutMs: 10_000 } }) The policy belongs to the returned stub, so each call site chooses its own, and ctx.exports namespaces accept it the same way as env bindings. maxAttempts must be at most 10 and timeoutMs between 500 and 60000; values outside those limits throw a RangeError. Omitted fields use the runtime's defaults. The policy only takes effect while the DURABLE_OBJECT_RETRIES_USERLAND autogate is on. This replaces the binding-level retryPolicy from #7383, so that field and the namespace constructor parameter that carried it are removed. Cap'n Proto needs the ordinal kept, so the field becomes obsolete2.

Add a `retry` option to DurableObjectNamespace's get(), getByName() and getExisting(): env.MY_DO.get(id, { retry: { maxAttempts: 5, timeoutMs: 10_000 } }) The policy belongs to the returned stub, so each call site chooses its own, and ctx.exports namespaces accept it the same way as env bindings. maxAttempts must be an integer from 0 to 10 and timeoutMs an integer from 500 to 60000; anything else, including fractions, throws a RangeError. Omitted fields use the runtime's defaults. The option is only validated and applied while the DURABLE_OBJECT_RETRIES_USERLAND autogate is on, and is ignored otherwise. This replaces the binding-level retryPolicy from #7383, so that field and the namespace constructor parameter that carried it are removed. Cap'n Proto needs the ordinal kept, so the field becomes obsolete2.

Add a `retry` option to DurableObjectNamespace's get(), getByName() and getExisting(): env.MY_DO.get(id, { retry: { maxRetries: 5, timeoutMs: 10_000 } }) The policy belongs to the returned stub, so each call site chooses its own, and ctx.exports namespaces accept it the same way as env bindings. maxRetries must be an integer from 0 to 10 and timeoutMs an integer from 500 to 60000; anything else, including fractions, throws a RangeError. Omitted fields use the runtime's defaults. The option is only validated and applied while the DURABLE_OBJECT_RETRIES_USERLAND autogate is on, and is ignored otherwise. maxRetries counts retries after the initial attempt, so the UserDefinedRetryPolicy field and its constants are renamed from maxAttempts to match. ActorRetryPolicy::maxAttempts() keeps its name, since it counts every attempt. This replaces the binding-level retryPolicy from #7383, so that field and the namespace constructor parameter that carried it are removed. Cap'n Proto needs the ordinal kept, so the field becomes obsolete2.

Add a `retry` option to DurableObjectNamespace's get(), getByName() and getExisting(): env.MY_DO.get(id, { retry: { maxRetries: 5, timeoutMs: 10_000 } }) The policy belongs to the returned stub, so each call site chooses its own, and ctx.exports namespaces accept it the same way as env bindings. maxRetries must be an integer from 0 to 10 and timeoutMs an integer from 500 to 60000; anything else, including fractions, throws a RangeError. Omitted fields use the runtime's defaults. The option is always validated, but the policy applies only while the DURABLE_OBJECT_RETRIES_USERLAND autogate is on, so turning the gate on changes retry behaviour without changing which inputs throw. maxRetries counts retries after the initial attempt, so the UserDefinedRetryPolicy field and its constants are renamed from maxAttempts to match. ActorRetryPolicy::maxAttempts() keeps its name, since it counts every attempt. This replaces the binding-level retryPolicy from #7383, so that field and the namespace constructor parameter that carried it are removed. Cap'n Proto needs the ordinal kept, so the field becomes obsolete2.

Add a `retry` option to DurableObjectNamespace's get(), getByName() and getExisting(): env.MY_DO.get(id, { retry: { maxRetries: 5, timeoutMs: 10_000 } }) The policy belongs to the returned stub, so each call site chooses its own, and ctx.exports namespaces accept it the same way as env bindings. maxRetries must be an integer from 0 to 10 and timeoutMs an integer from 500 to 60000; anything else, including fractions, throws a RangeError. Omitted fields use the runtime's defaults. The option is experimental: without the `experimental` compatibility flag it is ignored and absent from the public types. With the flag, it is always validated, but the policy applies only while the DURABLE_OBJECT_RETRIES_USERLAND autogate is on, so turning the gate on changes retry behaviour without changing which inputs throw. maxRetries counts retries after the initial attempt, so the UserDefinedRetryPolicy field and its constants are renamed from maxAttempts to match. ActorRetryPolicy::maxAttempts() keeps its name, since it counts every attempt. This replaces the binding-level retryPolicy from #7383, so that field and the namespace constructor parameter that carried it are removed. Cap'n Proto needs the ordinal kept, so the field becomes obsolete2.

Add a `retry` option to DurableObjectNamespace's get(), getByName() and getExisting(): env.MY_DO.get(id, { retry: { maxRetries: 5, timeoutMs: 10_000 } }) The policy belongs to the returned stub, so each call site chooses its own, and ctx.exports namespaces accept it the same way as env bindings. maxRetries must be an integer from 0 to 10 and timeoutMs an integer from 500 to 60000. Non-numbers, including null, throw a TypeError rather than being coerced; other numbers outside those limits, including fractions, throw a RangeError. Omitted fields use the runtime's defaults. The option is experimental: without the `experimental` compatibility flag it is ignored and absent from the public types. With the flag, it is always validated, but the policy applies only while the DURABLE_OBJECT_RETRIES_USERLAND autogate is on, so turning the gate on changes retry behaviour without changing which inputs throw. maxRetries counts retries after the initial attempt, so the UserDefinedRetryPolicy field and its constants are renamed from maxAttempts to match. ActorRetryPolicy::maxAttempts() keeps its name, since it counts every attempt. This replaces the binding-level retryPolicy from #7383, so that field and the namespace constructor parameter that carried it are removed. Cap'n Proto needs the ordinal kept, so the field becomes obsolete2.

Add a `retry` option to DurableObjectNamespace's get(), getByName() and getExisting(): env.MY_DO.get(id, { retry: { maxRetries: 5, timeoutMs: 10_000 } }) The policy belongs to the returned stub, so each call site chooses its own, and ctx.exports namespaces accept it the same way as env bindings. maxRetries must be an integer from 0 to 10 and timeoutMs an integer from 500 to 60000. Non-numbers, including null, throw a TypeError rather than being coerced; other numbers outside those limits, including fractions, throw a RangeError. Omitted fields use the runtime's defaults. The option is experimental: without the `experimental` compatibility flag it is ignored and absent from the public types. With the flag, it is always validated, but the policy applies only while the DURABLE_OBJECT_RETRIES_USERLAND autogate is on, so turning the gate on changes retry behaviour without changing which inputs throw. maxRetries counts retries after the initial attempt, so the UserDefinedRetryPolicy field and its constants are renamed from maxAttempts to match. ActorRetryPolicy::maxAttempts() keeps its name, since it counts every attempt. This replaces the binding-level retryPolicy from #7383, so that field, its copy in compiled-bindings.capnp and the namespace constructor parameter that carried it are removed. Cap'n Proto needs the ordinals kept, so the fields become obsolete2 and obsolete28.

Add a `retry` option to DurableObjectNamespace's get(), getByName() and getExisting(): env.MY_DO.get(id, { retry: { maxRetries: 5, timeoutMs: 10_000 } }) The policy belongs to the returned stub, so each call site chooses its own, and ctx.exports namespaces accept it the same way as env bindings. maxRetries must be an integer from 0 to 10 and timeoutMs an integer from 500 to 60000. Non-numbers, including null, throw a TypeError rather than being coerced; other numbers outside those limits, including fractions, throw a RangeError. Omitted fields use the runtime's defaults. The option is experimental: without the `experimental` compatibility flag it is ignored and absent from the public types. The options dictionary keeps `retry` as a raw value and get() converts it only with the flag, so a worker without the flag never gets a conversion error, whatever it passes. With the flag, the option is always validated, but the policy applies only while the DURABLE_OBJECT_RETRIES_USERLAND autogate is on, so turning the gate on changes retry behaviour without changing which inputs throw. maxRetries counts retries after the initial attempt, so the UserDefinedRetryPolicy field and its constants are renamed from maxAttempts to match. ActorRetryPolicy::maxAttempts() keeps its name, since it counts every attempt. This replaces the binding-level retryPolicy from #7383, so that field, its copy in compiled-bindings.capnp and the namespace constructor parameter that carried it are removed. Cap'n Proto needs the ordinals kept, so the fields become obsolete2 and obsolete28.

Add a `retry` option to DurableObjectNamespace's get(), getByName() and getExisting(): env.MY_DO.get(id, { retry: { maxRetries: 5, timeoutMs: 10_000 } }) The policy belongs to the returned stub, so each call site chooses its own, and ctx.exports namespaces accept it the same way as env bindings. maxRetries must be an integer from 0 to 10 and timeoutMs an integer from 500 to 60000. Non-numbers, including null, throw a TypeError rather than being coerced; other numbers outside those limits, including fractions, throw a RangeError. Omitted fields use the runtime's defaults. The option is experimental: without the `experimental` compatibility flag it is absent from the public types and never read. Workers with the flag get separate get(), getByName() and getExisting() registrations whose options dictionary has the `retry` field; other workers keep the dictionary without it, so neither a bad value nor a throwing getter can affect them. With the flag, the option is always validated, but the policy applies only while the DURABLE_OBJECT_RETRIES_USERLAND autogate is on, so turning the gate on changes retry behaviour without changing which inputs throw. maxRetries counts retries after the initial attempt, so the UserDefinedRetryPolicy field and its constants are renamed from maxAttempts to match. ActorRetryPolicy::maxAttempts() keeps its name, since it counts every attempt. This replaces the binding-level retryPolicy from #7383, so that field, its copy in compiled-bindings.capnp and the namespace constructor parameter that carried it are removed. Cap'n Proto needs the ordinals kept, so the fields become obsolete2 and obsolete28.

Add a `retry` option to DurableObjectNamespace's get(), getByName() and getExisting(): env.MY_DO.get(id, { retry: { maxRetries: 5, timeoutMs: 10_000 } }) The policy belongs to the returned stub, so each call site chooses its own, and ctx.exports namespaces accept it the same way as env bindings. maxRetries must be an integer from 0 to 10 and timeoutMs an integer from 500 to 60000. Both fields reject non-number values, including null, with a TypeError rather than coercing them. Other numbers outside those limits, including fractions, throw a RangeError. Omitted fields use the runtime's defaults. Passing `retry: null` is equivalent to `retry: {}` and uses the runtime's defaults. The option is experimental: without the `experimental` compatibility flag it is absent from the public types and never read. Workers with the flag get separate get(), getByName() and getExisting() registrations whose options dictionary has the `retry` field; other workers keep the dictionary without it, so neither a bad value nor a throwing getter can affect them. With the flag, the option is always validated, but the policy applies only while the DURABLE_OBJECT_RETRIES_USERLAND autogate is on, so turning the gate on changes retry behaviour without changing which inputs throw. maxRetries counts retries after the initial attempt, so the UserDefinedRetryPolicy field and its constants are renamed from maxAttempts to match. ActorRetryPolicy::maxAttempts() keeps its name, since it counts every attempt. This replaces the binding-level retryPolicy from #7383, so that field, its copy in compiled-bindings.capnp and the namespace constructor parameter that carried it are removed. Cap'n Proto needs the ordinals kept, so the fields become obsolete2 and obsolete28.

apeacock1991 requested review from a team as code owners September 16, 2026 08:59

apeacock1991 force-pushed the apeacock/STOR-5615-do-retry-policy-config branch 11 times, most recently from 7b33183 to c2a7334 Compare September 17, 2026 13:57

apeacock1991 requested a review from maxmcd September 17, 2026 13:59

apeacock1991 force-pushed the apeacock/STOR-5615-do-retry-policy-config branch 2 times, most recently from 652c1fd to ff80881 Compare September 18, 2026 11:41

maxmcd reviewed Sep 23, 2026

View reviewed changes

Comment thread src/workerd/api/http.c++ Outdated

Comment thread src/workerd/api/actor-call-retry.h Outdated

Comment thread src/workerd/api/actor-call-retry.c++ Outdated

maxmcd reviewed Sep 23, 2026

View reviewed changes

Comment thread src/workerd/api/http.c++ Outdated

Comment thread src/workerd/util/autogate.h

apeacock1991 force-pushed the apeacock/STOR-5615-do-retry-policy-config branch 7 times, most recently from 743419f to c996417 Compare September 24, 2026 08:24

maxmcd approved these changes Sep 24, 2026

View reviewed changes

Comment thread src/workerd/server/workerd.capnp

Comment thread src/workerd/server/workerd-api.c++ Outdated

Comment thread src/workerd/server/server.c++

Comment thread src/workerd/api/http.c++ Outdated

Comment thread src/workerd/api/actor-call-retry.h

Comment thread src/workerd/api/http.c++

apeacock1991 force-pushed the apeacock/STOR-5615-do-retry-policy-config branch 3 times, most recently from 57e51b9 to 6e19966 Compare September 24, 2026 20:12

apeacock1991 mentioned this pull request Sep 24, 2026

STOR-5615: Apply retry policy to JSRPC calls #7513

Merged

apeacock1991 added 5 commits September 24, 2026 22:30

apeacock1991 force-pushed the apeacock/STOR-5615-do-retry-policy-config branch from 6e19966 to badef8b Compare September 24, 2026 21:41

apeacock1991 merged commit 8cb7f9a into main Sep 25, 2026
23 checks passed

apeacock1991 deleted the apeacock/STOR-5615-do-retry-policy-config branch September 25, 2026 07:52

apeacock1991 mentioned this pull request Sep 25, 2026

[wrangler,miniflare,vite-plugin] Add Durable Object retry policies cloudflare/workers-sdk#15872

Closed

2 of 5 tasks

apeacock1991 mentioned this pull request Oct 9, 2026

STOR-5615: Set the Durable Object retry policy per stub #7672

Merged

Navigation Menu

Repository navigation

STOR-5615: Add configurable Durable Object retries - #7383

STOR-5615: Add configurable Durable Object retries#7383
apeacock1991 merged 5 commits into
mainfrom
apeacock/STOR-5615-do-retry-policy-config

apeacock1991 commented Sep 16, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

maxmcd left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

apeacock1991 commented Sep 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Behavior

Default change

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

maxmcd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

apeacock1991 commented Sep 16, 2026 •

edited

Loading