Skip to content

Race in func_get_annotations under free-threading #128714

Closed
Closed
Race in func_get_annotations under free-threading#128714
Labels
3.13only security fixes3.14bugs and security fixesinterpreter-core(Objects, Python, Grammar, and Parser dirs)topic-free-threadingtype-bugAn unexpected behavior, bug, or error

Description

@hawkinsp
hawkinsp
opened on Jan 10, 2025

Bug report

Bug description:

With CPython 3.13 from commit 65da5db TSAN reports the following race, which looks right: the initialization of p->func_annotations is racy.

Reproducer:

import concurrent.futures
import functools
import threading

num_threads = 8

def closure(b, f):
  b.wait()
  f.__annotations__

with concurrent.futures.ThreadPoolExecutor(max_workers=num_threads) as executor:
  for _ in range(100):
    b = threading.Barrier(num_threads)
    def f(x: int) -> int:
      return x + 1
    for _ in range(num_threads):
      executor.submit(functools.partial(closure, b, f))

TSAN report (first race):

WARNING: ThreadSanitizer: data race (pid=3380651)
  Read of size 8 at 0x7f02728d6840 by thread T5:
    #0 func_get_annotations /usr/local/google/home/phawkins/p/cpython/Objects/funcobject.c:769:13 (python3.13+0x232542) (BuildId: 19c569fa942016d8ac49d19fd40ccb1ddded939b)
    #1 getset_get /usr/local/google/home/phawkins/p/cpython/Objects/descrobject.c:193:16 (python3.13+0x1ff888) (BuildId: 19c569fa942016d8ac49d19fd40ccb1ddded939b)
    #2 _PyObject_GenericGetAttrWithDict /usr/local/google/home/phawkins/p/cpython/Objects/object.c:1665:19 (python3.13+0x295e11) (BuildId: 19c569fa942016d8ac49d19fd40ccb1ddded939b)
    #3 PyObject_GenericGetAttr /usr/local/google/home/phawkins/p/cpython/Objects/object.c:1747:12 (python3.13+0x295c62) (BuildId: 19c569fa942016d8ac49d19fd40ccb1ddded939b)
    #4 PyObject_GetAttr /usr/local/google/home/phawkins/p/cpython/Objects/object.c:1261:18 (python3.13+0x2945c7) (BuildId: 19c569fa942016d8ac49d19fd40ccb1ddded939b)
    #5 _PyEval_EvalFrameDefault /usr/local/google/home/phawkins/p/cpython/Python/generated_cases.c.h:3766:28 (python3.13+0x3ee2b0) (BuildId: 19c569fa942016d8ac49d19fd40ccb1ddded939b)
    #6 _PyEval_EvalFrame /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dea3a) (BuildId: 19c569fa942016d8ac49d19fd40ccb1ddded939b)
    #7 _PyEval_Vector /usr/local/google/home/phawkins/p/cpython/Python/ceval.c:1807:12 (python3.13+0x3dea3a)
    #8 _PyFunction_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c (python3.13+0x1eb65f) (BuildId: 19c569fa942016d8ac49d19fd40ccb1ddded939b)
    #9 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x572352) (BuildId: 19c569fa942016d8ac49d19fd40ccb1ddded939b)
    #10 partial_vectorcall /usr/local/google/home/phawkins/p/cpython/./Modules/_functoolsmodule.c:252:16 (python3.13+0x572352)
    #11 _PyVectorcall_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:273:16 (python3.13+0x1eb2d3) (BuildId: 19c569fa942016d8ac49d19fd40ccb1ddded939b)
    #12 _PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:348:16 (python3.13+0x1eb2d3)
    #13 PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:373:12 (python3.13+0x1eb355) (BuildId: 19c569fa942016d8ac49d19fd40ccb1ddded939b)
    #14 _PyEval_EvalFrameDefault /usr/local/google/home/phawkins/p/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e4af2) (BuildId: 19c569fa942016d8ac49d19fd40ccb1ddded939b)
    #15 _PyEval_EvalFrame /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dea3a) (BuildId: 19c569fa942016d8ac49d19fd40ccb1ddded939b)
    #16 _PyEval_Vector /usr/local/google/home/phawkins/p/cpython/Python/ceval.c:1807:12 (python3.13+0x3dea3a)
    #17 _PyFunction_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c (python3.13+0x1eb65f) (BuildId: 19c569fa942016d8ac49d19fd40ccb1ddded939b)
    #18 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef62f) (BuildId: 19c569fa942016d8ac49d19fd40ccb1ddded939b)
    #19 method_vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/classobject.c:70:20 (python3.13+0x1ef62f)
    #20 _PyVectorcall_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:273:16 (python3.13+0x1eb2d3) (BuildId: 19c569fa942016d8ac49d19fd40ccb1ddded939b)
    #21 _PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:348:16 (python3.13+0x1eb2d3)
    #22 PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:373:12 (python3.13+0x1eb355) (BuildId: 19c569fa942016d8ac49d19fd40ccb1ddded939b)
    #23 thread_run /usr/local/google/home/phawkins/p/cpython/./Modules/_threadmodule.c:337:21 (python3.13+0x564a32) (BuildId: 19c569fa942016d8ac49d19fd40ccb1ddded939b)
    #24 pythread_wrapper /usr/local/google/home/phawkins/p/cpython/Python/thread_pthread.h:243:5 (python3.13+0x4bddb7) (BuildId: 19c569fa942016d8ac49d19fd40ccb1ddded939b)

  Previous write of size 8 at 0x7f02728d6840 by thread T7:
    #0 func_get_annotation_dict /usr/local/google/home/phawkins/p/cpython/Objects/funcobject.c:535:9 (python3.13+0x22f46f) (BuildId: 19c569fa942016d8ac49d19fd40ccb1ddded939b)
    #1 func_get_annotations /usr/local/google/home/phawkins/p/cpython/Objects/funcobject.c:774:19 (python3.13+0x23256c) (BuildId: 19c569fa942016d8ac49d19fd40ccb1ddded939b)
    #2 getset_get /usr/local/google/home/phawkins/p/cpython/Objects/descrobject.c:193:16 (python3.13+0x1ff888) (BuildId: 19c569fa942016d8ac49d19fd40ccb1ddded939b)
    #3 _PyObject_GenericGetAttrWithDict /usr/local/google/home/phawkins/p/cpython/Objects/object.c:1665:19 (python3.13+0x295e11) (BuildId: 19c569fa942016d8ac49d19fd40ccb1ddded939b)
    #4 PyObject_GenericGetAttr /usr/local/google/home/phawkins/p/cpython/Objects/object.c:1747:12 (python3.13+0x295c62) (BuildId: 19c569fa942016d8ac49d19fd40ccb1ddded939b)
    #5 PyObject_GetAttr /usr/local/google/home/phawkins/p/cpython/Objects/object.c:1261:18 (python3.13+0x2945c7) (BuildId: 19c569fa942016d8ac49d19fd40ccb1ddded939b)
    #6 _PyEval_EvalFrameDefault /usr/local/google/home/phawkins/p/cpython/Python/generated_cases.c.h:3766:28 (python3.13+0x3ee2b0) (BuildId: 19c569fa942016d8ac49d19fd40ccb1ddded939b)
    #7 _PyEval_EvalFrame /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dea3a) (BuildId: 19c569fa942016d8ac49d19fd40ccb1ddded939b)
    #8 _PyEval_Vector /usr/local/google/home/phawkins/p/cpython/Python/ceval.c:1807:12 (python3.13+0x3dea3a)
    #9 _PyFunction_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c (python3.13+0x1eb65f) (BuildId: 19c569fa942016d8ac49d19fd40ccb1ddded939b)
    #10 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x572352) (BuildId: 19c569fa942016d8ac49d19fd40ccb1ddded939b)
    #11 partial_vectorcall /usr/local/google/home/phawkins/p/cpython/./Modules/_functoolsmodule.c:252:16 (python3.13+0x572352)
    #12 _PyVectorcall_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:273:16 (python3.13+0x1eb2d3) (BuildId: 19c569fa942016d8ac49d19fd40ccb1ddded939b)
    #13 _PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:348:16 (python3.13+0x1eb2d3)
    #14 PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:373:12 (python3.13+0x1eb355) (BuildId: 19c569fa942016d8ac49d19fd40ccb1ddded939b)
    #15 _PyEval_EvalFrameDefault /usr/local/google/home/phawkins/p/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e4af2) (BuildId: 19c569fa942016d8ac49d19fd40ccb1ddded939b)
    #16 _PyEval_EvalFrame /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3dea3a) (BuildId: 19c569fa942016d8ac49d19fd40ccb1ddded939b)
    #17 _PyEval_Vector /usr/local/google/home/phawkins/p/cpython/Python/ceval.c:1807:12 (python3.13+0x3dea3a)
    #18 _PyFunction_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c (python3.13+0x1eb65f) (BuildId: 19c569fa942016d8ac49d19fd40ccb1ddded939b)
    #19 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef62f) (BuildId: 19c569fa942016d8ac49d19fd40ccb1ddded939b)
    #20 method_vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/classobject.c:70:20 (python3.13+0x1ef62f)
    #21 _PyVectorcall_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:273:16 (python3.13+0x1eb2d3) (BuildId: 19c569fa942016d8ac49d19fd40ccb1ddded939b)
    #22 _PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:348:16 (python3.13+0x1eb2d3)
    #23 PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:373:12 (python3.13+0x1eb355) (BuildId: 19c569fa942016d8ac49d19fd40ccb1ddded939b)
    #24 thread_run /usr/local/google/home/phawkins/p/cpython/./Modules/_threadmodule.c:337:21 (python3.13+0x564a32) (BuildId: 19c569fa942016d8ac49d19fd40ccb1ddded939b)
    #25 pythread_wrapper /usr/local/google/home/phawkins/p/cpython/Python/thread_pthread.h:243:5 (python3.13+0x4bddb7) (BuildId: 19c569fa942016d8ac49d19fd40ccb1ddded939b)

CPython versions tested on:

3.13

Operating systems tested on:

Linux

Linked PRs

Activity

  1. hawkinsp
    added
    type-bugAn unexpected behavior, bug, or error
    on Jan 10, 2025
  2. mpage
    added
    topic-free-threading
    on Jan 10, 2025
  3. ZeroIntensity
    added
    interpreter-core(Objects, Python, Grammar, and Parser dirs)
    3.13only security fixes
    3.14bugs and security fixes
    on Jan 10, 2025

  4. ZeroIntensity commented on Jan 10, 2025

    @ZeroIntensity
    ZeroIntensity
    on Jan 10, 2025
    Member

    cc @JelleZijlstra

  5. xuantengh commented on Jan 19, 2025

    @xuantengh
    xuantengh
    on Jan 19, 2025
    Contributor

    Hi, I've submittedd a PR, and it seems funcobject.c lacks the support for free-threaded builds. Maybe we should consider audit this just like #127945, and #128002 ?

  6. xuantengh commented on Jan 19, 2025

    @xuantengh
    xuantengh
    on Jan 19, 2025
    Contributor

    Besides, it seems the the range object also has data races when I run the test suites test_free_threading under FT build:

    WARNING: ThreadSanitizer: data race (pid=2516929)
  Read of size 8 at 0x7f9cbfaa66a0 by thread T392:
    #0 rangeiter_next ../Objects/rangeobject.c:819 (python+0x21713e) (BuildId: 3d91448656453eabf7067b9e5af5b37997c21fc8)
    #1 zip_next ../Python/bltinmodule.c:3111 (python+0x313553) (BuildId: 3d91448656453eabf7067b9e5af5b37997c21fc8)
    #2 builtin_next ../Python/bltinmodule.c:1651 (python+0x3155bb) (BuildId: 3d91448656453eabf7067b9e5af5b37997c21fc8)
    #3 _PyEval_EvalFrameDefault ../Python/generated_cases.c.h:1510 (python+0x32d5df) (BuildId: 3d91448656453eabf7067b9e5af5b37997c21fc8)
    #4 _PyEval_EvalFrame ../Include/internal/pycore_ceval.h:116 (python+0x358df3) (BuildId: 3d91448656453eabf7067b9e5af5b37997c21fc8)
    #5 _PyEval_Vector ../Python/ceval.c:1913 (python+0x35904f) (BuildId: 3d91448656453eabf7067b9e5af5b37997c21fc8)
    #6 _PyFunction_Vectorcall ../Objects/call.c:413 (python+0x165e43) (BuildId: 3d91448656453eabf7067b9e5af5b37997c21fc8)
    #7 _PyObject_VectorcallTstate ../Include/internal/pycore_call.h:167 (python+0x16a189) (BuildId: 3d91448656453eabf7067b9e5af5b37997c21fc8)
    #8 method_vectorcall ../Objects/classobject.c:72 (python+0x16a477) (BuildId: 3d91448656453eabf7067b9e5af5b37997c21fc8)
    #9 _PyVectorcall_Call ../Objects/call.c:273 (python+0x167f38) (BuildId: 3d91448656453eabf7067b9e5af5b37997c21fc8)
    #10 _PyObject_Call ../Objects/call.c:348 (python+0x168362) (BuildId: 3d91448656453eabf7067b9e5af5b37997c21fc8)
    #11 PyObject_Call ../Objects/call.c:373 (python+0x1683c2) (BuildId: 3d91448656453eabf7067b9e5af5b37997c21fc8)
    #12 thread_run ../Modules/_threadmodule.c:354 (python+0x4bcd17) (BuildId: 3d91448656453eabf7067b9e5af5b37997c21fc8)
    #13 pythread_wrapper ../Python/thread_pthread.h:242 (python+0x41af14) (BuildId: 3d91448656453eabf7067b9e5af5b37997c21fc8)

  Previous write of size 8 at 0x7f9cbfaa66a0 by thread T394:
    #0 rangeiter_next ../Objects/rangeobject.c:822 (python+0x217198) (BuildId: 3d91448656453eabf7067b9e5af5b37997c21fc8)
    #1 zip_next ../Python/bltinmodule.c:3111 (python+0x313553) (BuildId: 3d91448656453eabf7067b9e5af5b37997c21fc8)
    #2 builtin_next ../Python/bltinmodule.c:1651 (python+0x3155bb) (BuildId: 3d91448656453eabf7067b9e5af5b37997c21fc8)
    #3 _PyEval_EvalFrameDefault ../Python/generated_cases.c.h:1510 (python+0x32d5df) (BuildId: 3d91448656453eabf7067b9e5af5b37997c21fc8)
    #4 _PyEval_EvalFrame ../Include/internal/pycore_ceval.h:116 (python+0x358df3) (BuildId: 3d91448656453eabf7067b9e5af5b37997c21fc8)
    #5 _PyEval_Vector ../Python/ceval.c:1913 (python+0x35904f) (BuildId: 3d91448656453eabf7067b9e5af5b37997c21fc8)
    #6 _PyFunction_Vectorcall ../Objects/call.c:413 (python+0x165e43) (BuildId: 3d91448656453eabf7067b9e5af5b37997c21fc8)
    #7 _PyObject_VectorcallTstate ../Include/internal/pycore_call.h:167 (python+0x16a189) (BuildId: 3d91448656453eabf7067b9e5af5b37997c21fc8)
    #8 method_vectorcall ../Objects/classobject.c:72 (python+0x16a477) (BuildId: 3d91448656453eabf7067b9e5af5b37997c21fc8)
    #9 _PyVectorcall_Call ../Objects/call.c:273 (python+0x167f38) (BuildId: 3d91448656453eabf7067b9e5af5b37997c21fc8)
    #10 _PyObject_Call ../Objects/call.c:348 (python+0x168362) (BuildId: 3d91448656453eabf7067b9e5af5b37997c21fc8)
    #11 PyObject_Call ../Objects/call.c:373 (python+0x1683c2) (BuildId: 3d91448656453eabf7067b9e5af5b37997c21fc8)
    #12 thread_run ../Modules/_threadmodule.c:354 (python+0x4bcd17) (BuildId: 3d91448656453eabf7067b9e5af5b37997c21fc8)
    #13 pythread_wrapper ../Python/thread_pthread.h:242 (python+0x41af14) (BuildId: 3d91448656453eabf7067b9e5af5b37997c21fc8)

  7. hawkinsp commented on Jan 31, 2025

    @hawkinsp
    hawkinsp
    on Jan 31, 2025
    ContributorAuthor

    Here's a somewhat similar looking race on type_get_annotations:

    WARNING: ThreadSanitizer: data race (pid=355399)
  Read of size 4 at 0x7e07ad0c2190 by thread T65 (mutexes: read M0):
    #0 PyType_Modified /__w/jax/jax/cpython/Objects/typeobject.c:1059:15 (python3.13+0x305823) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #1 type_get_annotations /__w/jax/jax/cpython/Objects/typeobject.c:1790:17 (python3.13+0x305823)
    #2 getset_get /__w/jax/jax/cpython/Objects/descrobject.c:193:16 (python3.13+0x1ff5e8) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #3 _Py_type_getattro_impl /__w/jax/jax/cpython/Objects/typeobject.c:5411:19 (python3.13+0x2e92cd) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #4 PyObject_GetOptionalAttr /__w/jax/jax/cpython/Objects/object.c:1307:19 (python3.13+0x292eb1) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #5 builtin_getattr /__w/jax/jax/cpython/Python/bltinmodule.c:1200:13 (python3.13+0x3d8d79) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #6 cfunction_vectorcall_FASTCALL /__w/jax/jax/cpython/Objects/methodobject.c:425:24 (python3.13+0x289b7b) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #7 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ead4a) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)

...

Previous atomic write of size 4 at 0x7e07ad0c2190 by thread T67 (mutexes: read M0):
    #0 _Py_atomic_store_uint32_relaxed /__w/jax/jax/cpython/./Include/cpython/pyatomic_gcc.h:461:3 (python3.13+0x2e5090) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #1 set_version_unlocked /__w/jax/jax/cpython/Objects/typeobject.c:978:5 (python3.13+0x2e5090)
    #2 type_modified_unlocked /__w/jax/jax/cpython/Objects/typeobject.c:1047:5 (python3.13+0x2e5090)
    #3 PyType_Modified /__w/jax/jax/cpython/Objects/typeobject.c:1064:5 (python3.13+0x3058d4) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #4 type_get_annotations /__w/jax/jax/cpython/Objects/typeobject.c:1790:17 (python3.13+0x3058d4)
    #5 getset_get /__w/jax/jax/cpython/Objects/descrobject.c:193:16 (python3.13+0x1ff5e8) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #6 _Py_type_getattro_impl /__w/jax/jax/cpython/Objects/typeobject.c:5411:19 (python3.13+0x2e92cd) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
    #7 PyObject_GetOptionalAttr /__w/jax/jax/cpython/Objects/object.c:1307:19 (python3.13+0x292eb1) (BuildId: c9937216e103905f871b62bf50b66fc5a8e96f80)
 
...

    I figured it might be simplest just to add this here, but I can open a new issue if you like!

  8. hawkinsp
    added 2 commits that reference this issue on Jan 31, 2025

    Add additional tsan suppressions for races.

    ac2cffd

    Add additional tsan suppressions for races.

    d4c6657

  9. kumaraditya303 commented on Feb 1, 2025

    @kumaraditya303
    kumaraditya303
    on Feb 1, 2025
    Contributor

    I figured it might be simplest just to add this here, but I can open a new issue if you like!

    Please create a new issue for it, the getters and setters of typeobject are missing critical section which would have to be added independent of function object fixes.

  10. kumaraditya303
    added a commit that references this issue on Feb 6, 2025

    gh-128714: Fix function object races in `__annotate__`, `__annotation…

    55f17b7
  11. xuantengh
    added a commit that references this issue on Feb 6, 2025

    pythongh-128714: Fix function object races in `__annotate__`, `__anno…

    0e09c6f
  12. srinivasreddy
    added a commit that references this issue on Feb 7, 2025

    pythongh-128714: Fix function object races in `__annotate__`, `__anno…

    31f6cd1
  13. cmaloney
    added a commit that references this issue on Feb 8, 2025

    pythongh-128714: Fix function object races in `__annotate__`, `__anno…

    0e02a24
  14. kumaraditya303
    closed this as completedon Feb 10, 2025
  15. terryjreedy
    added a commit that references this issue on Feb 24, 2025

    [3.13] gh-128714: Fix function object races in `__annotate__`, `__ann…

    bb6f3c3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.13only security fixes3.14bugs and security fixesinterpreter-core(Objects, Python, Grammar, and Parser dirs)topic-free-threadingtype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions