Actors: a call cancelled while it runs frees the turn in daprd while the method is still executing, so the next call overlaps (port of dapr/dotnet-sdk#1937) #353

Summary

When a caller cancels an actor call while it's running, daprd releases that actor's turn and starts the next queued call, but this SDK keeps running the cancelled call's actor method. Two turns then run on the same actor ID at once, which breaks turn-based concurrency. It happens with reentrancy off as well as on.

The runtime alone can't prevent this. The .NET SDK fixed it on its side in dapr/dotnet-sdk#1937 (released in Dapr.Actors 1.19.0) by holding the turn in the app. This SDK has no equivalent, as far as we can see.

How it happens

• daprd takes the actor's lock, calls the app with the caller's context, and releases the lock when that call returns (app.InvokeMethod).
• If the caller cancels, the call to the app returns at once and the lock is released, even though the app is still running the method:
  • Over HTTP, daprd aborts the request.
  • Over the gRPC actor callback stream, daprd stops waiting for the response and sends the app nothing (callbackstream.go).
• daprd then sends the next queued call for that actor ID. Neither an aborted HTTP request nor an abandoned stream call stops the actor method, so it overlaps with the new call.

dapr/dapr#10835 reported the overlap. dapr/dapr#10842, in runtime 1.19.0, fixed a related case: a call cancelled while still queued no longer evicts the running turn. A cancelled running call still releases the turn early, because daprd can't know when the app finishes once it stops waiting.

Evidence

From the standalone reproduction. Each trial sends 20 calls × 2s to one actor ID, cancels them all at a fixed time, and counts in-progress turns inside the actor. Kubernetes (kind), .NET actor host:

We measured this with the .NET SDK only and haven't run the reproduction against this SDK. The code below has no lock that would prevent it, though.

Where (rust-sdk)

The extractor generated by #[actor] (dapr-macros) calls ActorRuntime::get_or_create_actor and hands the handler a shared reference to the actor. The RwLock around active_actors guards the map only, so concurrent axum requests for the same actor ID run their handlers at the same time. Reminders and timers (invoke_reminder, invoke_timer) take the same route.

Suggested fix

Port the .NET approach: a per-actor turn lock in the SDK (ActorTurnLock, acquired in ActorManager).

• Acquire it per actor ID before activation and the method call.
• Hold it until the actor code has finished, including the post-method state save, whether or not the request was cancelled.
• When reentrancy is enabled, let a call with the same non-empty reentrancy ID as the running turn enter.
• Apply it to reminders, timers and deactivation as well as method calls.

Its own comment states the rule: "The sidecar can stop waiting before actor code finishes. Keep local ownership until every invocation in the turn has completed, not until its request is cancelled."