Release GIL in blocking AIO bindings

[tohtana]

The blocking parallel AIO bindings can call the handle's internal wait() while retaining the Python GIL. An AIO worker may need that GIL while completing PyTorch tensor cleanup. This causes a deadlock: the Python caller waits for the worker, while the worker waits for the GIL held by the caller.

This PR releases the GIL at the pread, pwrite, sync_pread, and sync_pwrite pybind entrypoints to avoid the deadlock.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bab900afc3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[sfc-gh-truwase]

@tohtana thanks for this PR.
Is it possible to check perf impact using any of the following tuning resources:

1. https://www.deepspeed.ai/tutorials/deepnvme/#performance-tuning
2. https://github.com/deepspeedai/DeepSpeedExamples/blob/master/deepnvme/file_access/README.md
3. https://github.com/deepspeedai/DeepSpeedExamples/tree/master/deepnvme/ds_io
4. https://github.com/deepspeedai/DeepSpeedExamples/tree/master/deepnvme/model_checkpoint

[sfc-gh-truwase]

@tohtana on further thoughts, I think my perf concerns are invalid because the changes swap GIL for per-handle mutex, and so it should actually improve perf. Do you agree?

[sfc-gh-truwase]

@tohtana do you think similar GIL concerns also apply to the GDS variant:
https://github.com/tohtana/DeepSpeed/blob/master/csrc/gds/py_lib/py_ds_gds.cpp

[tohtana]

Hi @sfc-gh-truwase,

the changes swap GIL for per-handle mutex, and so it should actually improve perf. Do you agree?

Yes, basically it won't hurt performance. We have much smaller critical sections now. When I briefly ran the benchmark, read showed a stable performance (10.066 → 10.029 GB/s). But the write showed a lot of fluctuation in my env and I currently don't have reliable numbers.

@tohtana do you think similar GIL concerns also apply to the GDS variant:
https://github.com/tohtana/DeepSpeed/blob/master/csrc/gds/py_lib/py_ds_gds.cpp

I think the concern is valid, and the exact same approach as this PR is not sufficient. We would need additional work to protect resources specific to GDS. It should be a follow-up PR.