data race when resizing unicode objects #132070

kumaraditya303 · 2025-04-04T11:49:00Z

When running ctypes tests in parallel, the following data race is detected:

env TSAN_OPTIONS="suppressions=Tools/tsan/suppressions_free_threading.txt halt_on_error=1" ./python -m test test_ctypes --parallel-threads 4

==================
WARNING: ThreadSanitizer: data race (pid=19492)
  Atomic read of size 8 at 0x7f1b0a230680 by thread T1323:
    #0 _Py_atomic_load_uintptr_relaxed /home/realkumaraditya/cpython/./Include/cpython/pyatomic_gcc.h:375:10 (python+0x271055) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #1 _Py_IsOwnedByCurrentThread /home/realkumaraditya/cpython/./Include/object.h:254:12 (python+0x271055)
    #2 _Py_TryIncrefFast /home/realkumaraditya/cpython/./Include/internal/pycore_object.h:558:9 (python+0x271055)
    #3 _Py_TryIncrefCompare /home/realkumaraditya/cpython/./Include/internal/pycore_object.h:597:9 (python+0x271055)
    #4 compare_unicode_unicode_threadsafe /home/realkumaraditya/cpython/Objects/dictobject.c:1432:18 (python+0x271055)
    #5 do_lookup /home/realkumaraditya/cpython/Objects/dictobject.c:1013:23 (python+0x271055)
    #6 unicodekeys_lookup_unicode_threadsafe /home/realkumaraditya/cpython/Objects/dictobject.c:1449:12 (python+0x271055)
    #7 _Py_dict_lookup_threadsafe /home/realkumaraditya/cpython/Objects/dictobject.c:1504:18 (python+0x270757) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #8 _PyDict_GetItemRef_KnownHash /home/realkumaraditya/cpython/Objects/dictobject.c:2353:21 (python+0x272c0e) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #9 PyDict_GetItemRef /home/realkumaraditya/cpython/Objects/dictobject.c:2389:12 (python+0x272c0e)
    #10 cache_struct_converter /home/realkumaraditya/cpython/./Modules/_struct.c:2524:9 (_struct.cpython-314t-x86_64-linux-gnu.so+0x5338) (BuildId: cd844f32564aba7a2b18bdb1e258a78c397c8ebc)
    #11 calcsize /home/realkumaraditya/cpython/./Modules/clinic/_struct.c.h:266:10 (_struct.cpython-314t-x86_64-linux-gnu.so+0x4ac7) (BuildId: cd844f32564aba7a2b18bdb1e258a78c397c8ebc)
    #12 _PyEval_EvalFrameDefault /home/realkumaraditya/cpython/Python/generated_cases.c.h:2225:35 (python+0x3fef50) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #13 _PyEval_EvalFrame /home/realkumaraditya/cpython/./Include/internal/pycore_ceval.h:119:16 (python+0x3f7150) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #14 _PyEval_Vector /home/realkumaraditya/cpython/Python/ceval.c:1908:12 (python+0x3f7150)
    #15 _PyFunction_Vectorcall /home/realkumaraditya/cpython/Objects/call.c (python+0x1f1cdf) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #16 _PyObject_VectorcallTstate /home/realkumaraditya/cpython/./Include/internal/pycore_call.h:169:11 (python+0x1f66b0) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #17 method_vectorcall /home/realkumaraditya/cpython/Objects/classobject.c:94:18 (python+0x1f66b0)
    #18 _PyVectorcall_Call /home/realkumaraditya/cpython/Objects/call.c:273:16 (python+0x1f196f) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #19 _PyObject_Call /home/realkumaraditya/cpython/Objects/call.c:348:16 (python+0x1f196f)
    #20 PyObject_Call /home/realkumaraditya/cpython/Objects/call.c:373:12 (python+0x1f19d5) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #21 _PyEval_EvalFrameDefault /home/realkumaraditya/cpython/Python/generated_cases.c.h:2411:32 (python+0x3ff399) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #22 _PyEval_EvalFrame /home/realkumaraditya/cpython/./Include/internal/pycore_ceval.h:119:16 (python+0x3f7150) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #23 _PyEval_Vector /home/realkumaraditya/cpython/Python/ceval.c:1908:12 (python+0x3f7150)
    #24 _PyFunction_Vectorcall /home/realkumaraditya/cpython/Objects/call.c (python+0x1f1cdf) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #25 _PyObject_VectorcallTstate /home/realkumaraditya/cpython/./Include/internal/pycore_call.h:169:11 (python+0x1f65ff) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #26 method_vectorcall /home/realkumaraditya/cpython/Objects/classobject.c:72:20 (python+0x1f65ff)
    #27 _PyVectorcall_Call /home/realkumaraditya/cpython/Objects/call.c:273:16 (python+0x1f196f) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #28 _PyObject_Call /home/realkumaraditya/cpython/Objects/call.c:348:16 (python+0x1f196f)
    #29 PyObject_Call /home/realkumaraditya/cpython/Objects/call.c:373:12 (python+0x1f19d5) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #30 thread_run /home/realkumaraditya/cpython/./Modules/_threadmodule.c:353:21 (python+0x5998d2) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #31 pythread_wrapper /home/realkumaraditya/cpython/Python/thread_pthread.h:242:5 (python+0x4f4157) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)

  Previous write of size 8 at 0x7f1b0a230680 by thread T1321:
    #0 __tsan_memcpy <null> (python+0xdd10f) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #1 _mi_memcpy /home/realkumaraditya/cpython/./Include/internal/mimalloc/mimalloc/internal.h:929:3 (python+0x2af9bf) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #2 _mi_heap_realloc_zero /home/realkumaraditya/cpython/Objects/mimalloc/alloc.c:745:7 (python+0x2af9bf)
    #3 mi_heap_realloc /home/realkumaraditya/cpython/Objects/mimalloc/alloc.c:753:10 (python+0x2cce30) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #4 _PyObject_MiRealloc /home/realkumaraditya/cpython/Objects/obmalloc.c:276:12 (python+0x2cce30)
    #5 PyObject_Realloc /home/realkumaraditya/cpython/Objects/obmalloc.c:1416:12 (python+0x2cfc12) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #6 resize_compact /home/realkumaraditya/cpython/Objects/unicodeobject.c:1178:31 (python+0x359915) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #7 _PyUnicodeWriter_Finish /home/realkumaraditya/cpython/Objects/unicodeobject.c:14157:16 (python+0x33162a) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #8 PyUnicode_Format /home/realkumaraditya/cpython/Objects/unicodeobject.c:15361:12 (python+0x35e728) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #9 unicode_mod /home/realkumaraditya/cpython/Objects/unicodeobject.c:14332:12 (python+0x36df71) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #10 binary_op1 /home/realkumaraditya/cpython/Objects/abstract.c:964:13 (python+0x1bdb83) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #11 binary_op /home/realkumaraditya/cpython/Objects/abstract.c:1003:24 (python+0x1be2a8) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #12 PyNumber_Remainder /home/realkumaraditya/cpython/Objects/abstract.c:1187:1 (python+0x1be2a8)
    #13 _PyEval_EvalFrameDefault /home/realkumaraditya/cpython/Python/generated_cases.c.h:62:35 (python+0x3f7ae5) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #14 _PyEval_EvalFrame /home/realkumaraditya/cpython/./Include/internal/pycore_ceval.h:119:16 (python+0x3f7150) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #15 _PyEval_Vector /home/realkumaraditya/cpython/Python/ceval.c:1908:12 (python+0x3f7150)
    #16 _PyFunction_Vectorcall /home/realkumaraditya/cpython/Objects/call.c (python+0x1f1cdf) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #17 _PyObject_VectorcallTstate /home/realkumaraditya/cpython/./Include/internal/pycore_call.h:169:11 (python+0x1f66b0) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #18 method_vectorcall /home/realkumaraditya/cpython/Objects/classobject.c:94:18 (python+0x1f66b0)
    #19 _PyVectorcall_Call /home/realkumaraditya/cpython/Objects/call.c:273:16 (python+0x1f196f) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #20 _PyObject_Call /home/realkumaraditya/cpython/Objects/call.c:348:16 (python+0x1f196f)
    #21 PyObject_Call /home/realkumaraditya/cpython/Objects/call.c:373:12 (python+0x1f19d5) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #22 _PyEval_EvalFrameDefault /home/realkumaraditya/cpython/Python/generated_cases.c.h:2411:32 (python+0x3ff399) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #23 _PyEval_EvalFrame /home/realkumaraditya/cpython/./Include/internal/pycore_ceval.h:119:16 (python+0x3f7150) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #24 _PyEval_Vector /home/realkumaraditya/cpython/Python/ceval.c:1908:12 (python+0x3f7150)
    #25 _PyFunction_Vectorcall /home/realkumaraditya/cpython/Objects/call.c (python+0x1f1cdf) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #26 _PyObject_VectorcallTstate /home/realkumaraditya/cpython/./Include/internal/pycore_call.h:169:11 (python+0x1f65ff) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #27 method_vectorcall /home/realkumaraditya/cpython/Objects/classobject.c:72:20 (python+0x1f65ff)
    #28 _PyVectorcall_Call /home/realkumaraditya/cpython/Objects/call.c:273:16 (python+0x1f196f) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #29 _PyObject_Call /home/realkumaraditya/cpython/Objects/call.c:348:16 (python+0x1f196f)
    #30 PyObject_Call /home/realkumaraditya/cpython/Objects/call.c:373:12 (python+0x1f19d5) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #31 thread_run /home/realkumaraditya/cpython/./Modules/_threadmodule.c:353:21 (python+0x5998d2) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)
    #32 pythread_wrapper /home/realkumaraditya/cpython/Python/thread_pthread.h:242:5 (python+0x4f4157) (BuildId: 27b79e16bbd11b196a3f4a8232b4b09ecf4e18ee)

Linked PRs

The text was updated successfully, but these errors were encountered:

kumaraditya303 · 2025-04-04T13:39:23Z

I think in free-threading Py_REFCNT(op)==1 should be replaced by PyObject_IsUniquelyReferenced in such functions in unicodeobject.c:

cpython/Objects/unicodeobject.c

Lines 1197 to 1202 in a126cef

    
           static int 
        
           resize_inplace(PyObject *unicode, Py_ssize_t length) 
        
           { 
        
               assert(!PyUnicode_IS_COMPACT(unicode)); 
        
               assert(Py_REFCNT(unicode) == 1);

colesbury · 2025-04-04T13:48:39Z

We should leave the assertions as is. I don't think that's related to this.

The race is because PyObject_Realloc() calls memcpy to fill in the new allocation, if the existing allocation can't be reused. The memcpy doesn't use relaxed atomic writes, so it can race with an one of the optimistic, non-locking reads.

For now, we should just add a suppression.

colesbury · 2025-04-04T13:49:33Z

We should change Py_REFCNT(op)==1 to PyObject_IsUniquelyReferenced for places where it controls behaviors, such as in if-statements, but the uses in assertions don't really matter.

)

godlygeek · 2025-04-27T23:17:52Z

Would it make sense to change the behavior of Py_REFCNT in free-threading so that Py_REFCNT(op)==1 remains a valid check? I think that due to TOCTOU the only thing an extension module can usefully do with Py_REFCNT even in GIL builds is check whether the recount of an object is or isn't 1. If you're no longer able to use Py_REFCNT to check that in FT builds, it seems like Py_REFCNT becomes totally useless, and it would be much more useful to do

#define Py_REFCNT(x) (PyObject_IsUniquelyReferenced((x)) ? 1 : INT_MAX)

or some other arbitrary large number.

serhiy-storchaka · 2025-04-28T08:49:37Z

There are some places where Py_REFCNT is compared with 2 or 0.

godlygeek · 2025-04-28T15:09:42Z

Sure, but those don't work in free threaded builds today, right? So my question is whether we can make it so that Py_REFCNT in an FT build returns something wrong but usually useful instead of something wrong and never useful.

ZeroIntensity · 2025-04-28T15:55:34Z

Why couldn't it just be _PyObject_IsUniquelyReferenced(op) ? 1 : actual_refcnt(op)?

colesbury · 2025-04-28T15:58:31Z

@ZeroIntensity - that's identical to actual_refcnt(op). _PyObject_IsUniquelyReferenced(1) == 1 implies Py_REFCNT(op) == 1. The inverse is not necessarily true.

I agree with @serhiy-storchaka here. Py_REFCNT() is used for lots of things. Breaking it for a wide variety of use cases in order to try to make the _PyObject_IsUniquelyReferenced() use case more robust is not worth it.

ZeroIntensity · 2025-04-28T16:06:42Z

The inverse is not necessarily true.

Oh, that makes sense.

Is this documented anywhere? The docs do say that you can check for a reference count of 1. If that has problems with FT, we need to update it.
We should probably expose _PyObject_IsUniquelyReferenced as an unstable API for this then. Is there any current plan for that?

…3039) --------- Co-authored-by: Kumar Aditya <[email protected]> Co-authored-by: Serhiy Storchaka <[email protected]>

…ythongh-133039) --------- (cherry picked from commit 75cbb8d) Co-authored-by: Donghee Na <[email protected]> Co-authored-by: Kumar Aditya <[email protected]> Co-authored-by: Serhiy Storchaka <[email protected]>

…gh-133039) (gh-133126) * gh-132070: Use _PyObject_IsUniquelyReferenced in unicodeobject (gh-133039) --------- (cherry picked from commit 75cbb8d) Co-authored-by: Donghee Na <[email protected]> Co-authored-by: Kumar Aditya <[email protected]> Co-authored-by: Serhiy Storchaka <[email protected]> * Add _PyObject_IsUniquelyReferenced --------- Co-authored-by: Kumar Aditya <[email protected]> Co-authored-by: Serhiy Storchaka <[email protected]>

corona10 · 2025-04-29T07:38:34Z

@ZeroIntensity Would you like to open a new issue to track adding unstable C API?

kumaraditya303 added the topic-free-threading label Apr 4, 2025

picnixz added type-bug An unexpected behavior, bug, or error interpreter-core (Objects, Python, Grammar, and Parser dirs) topic-unicode labels Apr 4, 2025

bedevere-app bot mentioned this issue Apr 4, 2025

gh-132070: use _PyObject_IsUniquelyReferenced in unicodeobject #132091

Closed

bedevere-app bot mentioned this issue Apr 13, 2025

gh-132070: add PyObject_Realloc suppression in free-threading #132468

Merged

kumaraditya303 added a commit that referenced this issue Apr 16, 2025

gh-132070: add PyObject_Realloc suppression in free-threading (#132468

4b15d10

)

bedevere-app bot mentioned this issue Apr 27, 2025

gh-132070: Use _PyObject_IsUniquelyReferenced in unicodeobject #133039

Merged

JelleZijlstra mentioned this issue Apr 28, 2025

gh-133009: fix UAF in xml.etree.ElementTree.Element.__deepcopy__ #133010

Merged

corona10 added a commit that referenced this issue Apr 29, 2025

gh-132070: Use _PyObject_IsUniquelyReferenced in unicodeobject (gh-13…

75cbb8d

…3039) --------- Co-authored-by: Kumar Aditya <[email protected]> Co-authored-by: Serhiy Storchaka <[email protected]>

This was referenced Apr 29, 2025

[3.13] gh-132070: Use _PyObject_IsUniquelyReferenced in unicodeobject (gh-133039) #133121

Closed

[3.13] gh-132070: Use _PyObject_IsUniquelyReferenced in unicodeobject (gh-133039) #133126

Merged

ZeroIntensity mentioned this issue Apr 29, 2025

Adding an unstable C API for unique references #133140

Closed

corona10 closed this as completed Apr 29, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

data race when resizing unicode objects #132070

data race when resizing unicode objects #132070

kumaraditya303 commented Apr 4, 2025 •

edited by bedevere-app bot

Loading

kumaraditya303 commented Apr 4, 2025

Uh oh!

colesbury commented Apr 4, 2025

Uh oh!

colesbury commented Apr 4, 2025

Uh oh!

godlygeek commented Apr 27, 2025 •

edited

Loading

Uh oh!

serhiy-storchaka commented Apr 28, 2025

Uh oh!

godlygeek commented Apr 28, 2025

Uh oh!

ZeroIntensity commented Apr 28, 2025 •

edited

Loading

Uh oh!

colesbury commented Apr 28, 2025

Uh oh!

ZeroIntensity commented Apr 28, 2025 •

edited

Loading

Uh oh!

corona10 commented Apr 29, 2025

Uh oh!

Uh oh!

data race when resizing unicode objects #132070

data race when resizing unicode objects #132070

Comments

kumaraditya303 commented Apr 4, 2025 • edited by bedevere-app bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Linked PRs

kumaraditya303 commented Apr 4, 2025

Uh oh!

colesbury commented Apr 4, 2025

Uh oh!

colesbury commented Apr 4, 2025

Uh oh!

godlygeek commented Apr 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

serhiy-storchaka commented Apr 28, 2025

Uh oh!

godlygeek commented Apr 28, 2025

Uh oh!

ZeroIntensity commented Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

colesbury commented Apr 28, 2025

Uh oh!

ZeroIntensity commented Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

corona10 commented Apr 29, 2025

Uh oh!

kumaraditya303 commented Apr 4, 2025 •

edited by bedevere-app bot

Loading

godlygeek commented Apr 27, 2025 •

edited

Loading

ZeroIntensity commented Apr 28, 2025 •

edited

Loading

ZeroIntensity commented Apr 28, 2025 •

edited

Loading