Skip to content

Fix race condition on concurrent modification of alive nodes in NodePool - #301

Open
5k4nd wants to merge 2 commits into
elastic:mainfrom
5k4nd:fix-node-pool-get-concurrent-modification
Open

5k4nd wants to merge 2 commits into
elastic:mainfrom
5k4nd:fix-node-pool-get-concurrent-modification

Conversation

@5k4nd

@5k4nd 5k4nd commented Sep 23, 2026 •

Copy link
Copy Markdown

Fix RuntimeError: dictionary changed size during iteration in NodePool.get()

Fixes #302

NodePool.get() iterates over _alive_nodes while other threads sharing the client may mutate it through
mark_dead(), mark_live() or resurrect(), which raises:

  File "elastic_transport/_transport.py", line 338, in perform_request
    node = self.node_pool.get()
  File "elastic_transport/_node_pool.py", line 361, in get
    nodes = [
RuntimeError: dictionary changed size during iteration

We hit it in production with a client shared by the threads of a multi-threaded WSGI server.

The fix iterates over a copy of _alive_nodes. dict.copy() is atomic under the GIL, so no lock is needed.
The added test is deterministic: it mutates _alive_nodes during the iteration instead of relying on threads.

@cla-checker-service

cla-checker-service Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

💚 CLA has been signed

@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown

This pull request is stale because it has been open 7 days with no activity. Remove stale label or comment or this will be closed in 2 days.

@github-actions github-actions Bot added the Stale label Oct 1, 2026
@margaretjgu

Copy link
Copy Markdown
Member

Thanks for the contribution. Please open up an issue first so I can validate the context of the problem and for our tracking.

@5k4nd

5k4nd commented Oct 1, 2026

Copy link
Copy Markdown
Author

@margaretjgu thanks for your reply, I just opened an issue #302
I also signed the CLA a week ago.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Race condition on concurrent modification of alive nodes in NodePool

2 participants