Repository navigation
Crash when concurrently writing with print and concurrently modifying sys.stdout #130163
Description
Crash report
What happened?
Playing with the code from #130148, I stumbled on a segfault in a free-threaded debug build with the following (seems to trigger faster/more consistently when pasted in the REPL):
from contextlib import redirect_stdout
from io import StringIO
from threading import Thread
import time
def test_redirect():
text = StringIO()
with redirect_stdout(text):
print("hello1", file=text)
time.sleep(0.1)
print("hello2", file=text)
print(text.getvalue())
assert text.getvalue() == "hello1\nhello2\n"
for x in range(100):
Thread(target=test_redirect, args=()).start()I didn't have time to check whether JIT, debug or no-gil are strictly necessary for this to crash. JIT is not needed for this to crash.
Cause
The problem is that in the print() implementation, _PySys_GetAttr returns a borrowed reference.
Lines 2173 to 2175 in 655fc8a
So if sys.stdout changes concurrently with the print(), the program may crash because file will point to a deallocated Python object.
This affects the GIL-enabled build as well. See this reproducer: https://gist.github.com/colesbury/c48f50e95d5d68e24814a56e2664e587
Suggested fix
Introduce a _PySys_GetAttrRef that returns a new reference instead of a borrowed reference. Use that instead.
We should audit all the uses of _PySys_GetAttr and PySys_GetObject. I expect most of them will need to be replaced with functions that return new references, but it doesn't all have to be in a single PR.
CPython versions tested on:
CPython main branch
Operating systems tested on:
Linux
Output from running 'python -VV' on the command line:
Python 3.14.0a5+ experimental free-threading build (heads/main:359c7dde3bb, Feb 15 2025, 17:54:11) [GCC 11.4.0]
Linked PRs
- gh-130163: Fix usage of borrow references from _PySys_GetAttr #130282
- gh-130163: Fix possible crashes related to PySys_GetObject() #130503
- [3.13] gh-130163: Fix crashes related to PySys_GetObject() (GH-130503) #130556
- [3.13] gh-130163: Fix a leak in _pickle.c after backporting #130568
- [3.12] gh-130163: Fix crashes related to PySys_GetObject() (GH-130503) (GH-130556) #130576
Activity
[-]Segfault in a freethreaded JIT build from using `contextlib.redirect_stdout` in threads[/-][+]Segfault in a freethreaded build from using `contextlib.redirect_stdout` in threads[/+]picnixz commentedon Feb 15, 2025
The issue could be with
StringIO()orsys.stdout/sys.stderror with context managers in general. I don't know where to start to investigate this =/tomasr8 commentedon Feb 15, 2025
Here's the backtrace:
The immediate cause seems to be related to the printing itself, but I didn't dig into it too much
devdanzin commentedon Feb 15, 2025
Substituting
StringIOwith the following stops crashing, would it point to it being the culprit?picnixz commentedon Feb 15, 2025
I suspect it has something to do with the
PyFile_*interface.tomasr8 commentedon Feb 15, 2025
Might be something to do with
redirect_stdout, I'm getting a segfault even with this simplified reproducer:edit:
Both crashes seem to be caused by
Py_TYPEreturningNULL. Here's the backtrace for this repro:picnixz commentedon Feb 15, 2025
Have you tried removing the usage to
StringIO()? because it appears that we crash inPy_DECREFof a file object.sergey-miryanov commentedon Feb 16, 2025
Yes, it is caused by
redirect_stdout.PyObject_GetAttr(cpython/Objects/fileobject.c
Line 113 in 798f8d3
Py_DECREF(cpython/Objects/fileobject.c
Line 127 in 798f8d3
_Py_DeallocPy_DECREFfrom T1 decrements local ref_cnt, got 0 and go to_Py_MergeZeroLocalRefcount, see that shared ref_cnt is 0 and go to fast path and do_Py_Dealloc.The easiest way to fix this is probably to protect
PyFile_WriteObjectwith a critical section. However, this might slightly affect the performance of single-threaded apps.The best solution might be to update the documentation and point out this potential issue with
redirect_*.devdanzin commentedon Feb 16, 2025
Using
open("/dev/null", "w")instead ofStringIOstill segfaults.But any program that does
redirect_stdoutdoes will also segfault:Would making the presence of the critical section dependent on whether the build is freethreaded (and/or the GIL is disabled) avoid the performance impact for gilfull builds? Or would it be a bad idea to have this conditional difference in behavior?
ISTM that fixing the segfault is necessary, even if a note is added to docs, because users may hit the segfault without using
redirect_*.30 remaining items
sergey-miryanov commentedon Feb 17, 2025
Thanks, I will try.
sergey-miryanov commentedon Feb 23, 2025
I've looked at how
_PySys_GetAttris used. I can reproduce thesegfaulterror in some places where it's used. I've written tests for these places.@colesbury @ZeroIntensity, @serhiy-storchaka please take a look.
Here's a copy from my PR:
_pickle.c/whichmodule_threadmodule.c/thread_excepthookfaulthandler.c/faulthandler_get_filenobltinmodule.c/builtin_print_implbltinmodule.c/builtin_input_implerrors.c/write_unraisable_excerrors.c/format_unraisable_v_warnings.c/show_warningintrinsics.c/print_exprpylifecycle.c/flush_std_filespythonrun.c/_PyRun_InteractiveLoopObjectpythonrun.c/pyrun_one_parse_astpythonrun.c/_Py_HandleSystemExitAndKeyboardInterruptpythonrun.c/_PyErr_PrintExpythonrun.c/PyErr_Displaypythonrun.c/flush_io_streamsysmodule.c/sys_displayhooksysmodule.c/get_warnoptionssysmodule.c/PySys_ResetWarnOptionssysmodule.c/PySys_HasWarnOptionssysmodule.c/_PySys_AddXOptionWithErrorsysmodule.c/PySys_SetArgvExsysmodule.c/sys_writesysmodule.c/sys_formattraceback.c/_Py_FindSourceFilesys_displayhook, but not where it's used.sys_write,sys_format- I have checked all the places where they called, and most of the locations are safe. However, a few places use%sspecifier, in such places it is very tricky to getsegfault.get_warnoptions,PySys_ResetWarnOptions,_PySys_AddXOptionWithError- are deprecated, but it is a very tricky to getsegfaulttoo.flush_io_stream,PyErr_Display,pyrun_one_parse_ast,print_expr,write_unraisable_exc,thread_excepthook- very tight - They use a borrowed reference right after getting. It might be possible to reproduce it under heavy contention, but I wasn't able to do so._PyRun_InteractiveLoopObjectonly checks presence ofps1andps2.flush_std_filesused from_Py_Finalizeand fromfatal_error. For_Py_FinalizeI can repro, forfatal_error- no.whichmodule,_Py_FindSourceFileiterate borrowed reference, but I wasn't able to reproducesegfault._PySys_GetAttrcalled without an exception set.Also, can you point me out what to do next? I can replace
_PySys_GetAttrwith_PySys_GetAttrRef, but in the most places it is interleave with @serhiy-storchaka PRs. I'm not sure what the correct way to deal with this situation is.serhiy-storchaka commentedon Feb 25, 2025
#130503 is an adaptation of #111035 or #129736 that does not introduce new public API and does not change the current behavior (except fixing crashes, mainly). It will make minimal difference with #111035 or #129736, whatever will be accepted for future Python version.
I added also tests for
print(),input()andsys.displayhook()(they are relatively simple). Reproducers for other crashes are harder, so I do not think that this is worth hassle.Thank you for your tests, @sergey-miryanov, but sorry, I have not used them. They are high quality, but too large and expensive. It is not worth to keep them forever after fixing this issue.
sergey-miryanov commentedon Feb 25, 2025
@serhiy-storchaka I simplified it a bit yesterday, not sure if you saw the latest changes. Maybe it will be useful, but it is all up to you. Anyway, it was very interesting.
gh-130163: Fix crashes related to PySys_GetObject() (GH-130503)
[3.13] pythongh-130163: Fix crashes related to PySys_GetObject() (pyt…
[3.13] gh-130163: Fix crashes related to PySys_GetObject() (GH-130503) (
[3.13] pythongh-130163: Fix a leak in _pickle.c after backporting
[3.13] gh-130163: Fix a leak in _pickle.c after backporting (GH-130568)
[3.12] pythongh-130163: Fix crashes related to PySys_GetObject() (pyt…
[3.13] pythongh-130163: Fix a leak in _pickle.c after backporting (py…
[3.12] gh-130163: Fix crashes related to PySys_GetObject() (GH-130503) (
vstinner commentedon Feb 27, 2025
I cannot reproduce the crash using #130163 (comment) reproducer. Can we close the issue?
serhiy-storchaka commentedon Feb 27, 2025
It has already been fixed.