BUG: segfault from random_raw from instance initialized from invalid value

@vstinner

Describe the issue:

It's possible to segfault the interpreter by (re)initializing a random generator with invalid data, then calling random_raw.

Reproduce the code example:

import numpy.random

p = numpy.random.PCG64DXSM()

try:
    p.__init__(("",))
except ValueError:
    pass
p.random_raw()

Error message:

Thread 1 "python" received signal SIGSEGV, Segmentation fault.
0x00007ffff4137f7a in __pyx_f_5numpy_6random_6_pcg64_pcg64_cm_uint64 ()
   from /home/danzin/venvs/3.13_upstream_fusil_venv/lib/python3.13t/site-packages/numpy/random/_pcg64.cpython-313t-x86_64-linux-gnu.so

#0  0x00007ffff4137f7a in __pyx_f_5numpy_6random_6_pcg64_pcg64_cm_uint64 ()
   from /home/danzin/venvs/3.13_upstream_fusil_venv/lib/python3.13t/site-packages/numpy/random/_pcg64.cpython-313t-x86_64-linux-gnu.so
#1  0x00007ffff4dd45d6 in __pyx_f_5numpy_6random_7_common_random_raw ()
   from /home/danzin/venvs/3.13_upstream_fusil_venv/lib/python3.13t/site-packages/numpy/random/_common.cpython-313t-x86_64-linux-gnu.so
#2  0x00007ffff53af54b in __pyx_pw_5numpy_6random_13bit_generator_12BitGenerator_11random_raw ()
   from /home/danzin/venvs/3.13_upstream_fusil_venv/lib/python3.13t/site-packages/numpy/random/bit_generator.cpython-313t-x86_64-linux-gnu.so
#3  0x0000555555a484db in _PyObject_VectorcallTstate (tstate=0x5555566c6300 <_PyRuntime+326528>,
    callable=0x7fffb438ce90, args=0x7fffffffcc38, nargsf=15750249268501108917,
    kwnames=0x5555564cdf0c <PyCFunction_Type+12>) at ./Include/internal/pycore_call.h:168
#4  0x0000555555dbd570 in _PyEval_EvalFrameDefault (tstate=<optimized out>,
    frame=<optimized out>, throwflag=<optimized out>) at Python/generated_cases.c.h:813
#5  0x0000555555dac3eb in PyEval_EvalCode (co=co@entry=0x7fffb4844990,
    globals=globals@entry=0x7fffb4755770, locals=locals@entry=0x7fffb4755770)
    at Python/ceval.c:604
#6  0x0000555555f5d469 in run_eval_code_obj (
    tstate=tstate@entry=0x5555566c6300 <_PyRuntime+326528>, co=co@entry=0x7fffb4844990,
    globals=globals@entry=0x7fffb4755770, locals=locals@entry=0x7fffb4755770)
    at Python/pythonrun.c:1381
#7  0x0000555555f5cb3f in run_mod (mod=mod@entry=0x7fffb45a8758,
    filename=filename@entry=0x7fffb423d210, globals=0x7fffb4755770, locals=0x7fffb4755770,
    flags=<optimized out>, arena=arena@entry=0x7fffb4051870, interactive_src=0x0,
    generate_new_source=0) at Python/pythonrun.c:1466
#8  0x0000555555f54b3a in pyrun_file (fp=<optimized out>, filename=0x7fffb423d210, start=257,
    globals=<optimized out>, locals=<optimized out>, closeit=<optimized out>,
    flags=<optimized out>) at Python/pythonrun.c:1295
#9  _PyRun_SimpleFileObject (fp=<optimized out>, fp@entry=0x515000000f80,
    filename=filename@entry=0x7fffb423d210, closeit=<optimized out>, closeit@entry=1,
    flags=<optimized out>, flags@entry=0x7fffffffd6d0) at Python/pythonrun.c:517
#10 0x0000555555f53f36 in _PyRun_AnyFileObject (fp=fp@entry=0x515000000f80,
    filename=filename@entry=0x7fffb423d210, closeit=closeit@entry=1,
    flags=flags@entry=0x7fffffffd6d0) at Python/pythonrun.c:77
#11 0x0000555555fbcb55 in pymain_run_file_obj (program_name=0x7fffb4da3030,
    filename=0x7fffb423d210, skip_source_first_line=0) at Modules/main.c:410
#12 pymain_run_file (config=config@entry=0x555556697d88 <_PyRuntime+136712>) at Modules/main.c:429
#13 0x0000555555fba882 in pymain_run_python (exitcode=0x7fffffffd784) at Modules/main.c:696
#14 Py_RunMain () at Modules/main.c:775
#15 0x0000555555fbb73e in pymain_main (args=<optimized out>) at Modules/main.c:805
#16 0x0000555555fbb8a4 in Py_BytesMain (argc=2, argv=<optimized out>) at Modules/main.c:829
#17 0x00007ffff7c2a3b8 in __libc_start_call_main (main=main@entry=0x55555588a630 <main>,
    argc=argc@entry=2, argv=argv@entry=0x7fffffffdb68)
    at ../sysdeps/nptl/libc_start_call_main.h:58
#18 0x00007ffff7c2a47b in __libc_start_main_impl (main=0x55555588a630 <main>, argc=2,
    argv=0x7fffffffdb68, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>,
    stack_end=0x7fffffffdb58) at ../csu/libc-start.c:360
#19 0x00005555557a9105 in _start ()

Python and NumPy Versions:

2.3.0.dev0+git20250415.e151f0d
3.13.3+ experimental free-threading build (heads/3.13:83cb89b941b, Apr 18 2025, 20:59:43) [Clang 19.1.7 (++20250114103253+cd708029e0b2-1exp120250114103309.40)]

Runtime Environment:

[{'numpy_version': '2.3.0.dev0+git20250415.e151f0d',
'python': '3.13.3+ experimental free-threading build '
'(heads/3.13:83cb89b941b, Apr 18 2025, 20:59:43) [Clang 19.1.7 '
'(++20250114103253+cd708029e0b2-1exp120250114103309.40)]',
'uname': uname_result(system='Linux', node='beesknees', release='6.11.0-24-generic', version='https://github.com/numpy/numpy/pull/24-Ubuntu SMP PREEMPT_DYNAMIC Fri Mar 14 18:13:56 UTC 2025', machine='x86_64')},
{'simd_extensions': {'baseline': ['SSE', 'SSE2', 'SSE3'],
'found': ['SSSE3',
'SSE41',
'POPCNT',
'SSE42',
'AVX',
'F16C',
'FMA3',
'AVX2',
'AVX512F',
'AVX512CD',
'AVX512_SKX',
'AVX512_CLX',
'AVX512_CNL',
'AVX512_ICL'],
'not_found': ['AVX512_KNL', 'AVX512_KNM', 'AVX512_SPR']}},
{'architecture': 'SkylakeX',
'filepath': '/home/danzin/venvs/3.13_upstream_fusil_venv/lib/python3.13t/site-packages/numpy.libs/libscipy_openblas64_-56d6093b.so',
'internal_api': 'openblas',
'num_threads': 16,
'prefix': 'libscipy_openblas',
'threading_layer': 'pthreads',
'user_api': 'blas',
'version': '0.3.29'}]

Context for the issue:

I have been fuzzing Numpy using fusil by @vstinner. I realize these crashes are unlikely to be triggered in normal usage and therefore might be of low priority.

The fuzzing was done with an ASAN free-threading clang build and not confirmed on a GILfull non-sanitizer GCC build yet.

@rkern

I can reproduce this in an ASAN build of Python:

=================================================================
==36838==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x0001083b7294 bp 0x00016ddd2810 sp 0x00016ddd27f0 T0)
==36838==The signal is caused by a READ memory access.
==36838==Hint: address points to the zero page.
    #0 0x0001083b7294 in pcg64_cm_next64 pcg64.h:399
    #1 0x0001083b71cc in __pyx_f_5numpy_6random_6_pcg64_pcg64_cm_uint64 _pcg64.pyx.c:5584
    #2 0x000107f07270 in __pyx_f_5numpy_6random_7_common_random_raw _common.pyx.c:6857
    #3 0x000107d751fc in __pyx_pf_5numpy_6random_13bit_generator_12BitGenerator_10random_raw bit_generator.pyx.c:13164
    #4 0x000107d6c3d4 in __pyx_pw_5numpy_6random_13bit_generator_12BitGenerator_11random_raw bit_generator.pyx.c:13131
    #5 0x000107d9f284 in __Pyx_CyFunction_Vectorcall_FASTCALL_KEYWORDS bit_generator.pyx.c:22069
    #6 0x000103259fb0 in PyObject_Vectorcall call.c:327
    #7 0x00010354c690 in _PyEval_EvalFrameDefault generated_cases.c.h:813
    #8 0x00010353ab6c in PyEval_EvalCode ceval.c:604
    #9 0x000103533258 in builtin_exec bltinmodule.c.h:556
    #10 0x000103541764 in _PyEval_EvalFrameDefault generated_cases.c.h:1217
    #11 0x00010329b1dc in gen_send_ex2 genobject.c:229
    #12 0x00010329ac9c in gen_send_ex genobject.c:270
    #13 0x000103553ba4 in _PyEval_EvalFrameDefault generated_cases.c.h:1796
    #14 0x0001032602fc in method_vectorcall classobject.c:62
    #15 0x000103259d44 in _PyVectorcall_Call call.c:285
    #16 0x00010354dc6c in _PyEval_EvalFrameDefault generated_cases.c.h:1355
    #17 0x00010353ab6c in PyEval_EvalCode ceval.c:604
    #18 0x000103658d34 in run_eval_code_obj pythonrun.c:1381
    #19 0x0001036585f8 in run_mod pythonrun.c:1466
    #20 0x000103653864 in _PyRun_SimpleFileObject pythonrun.c:517
    #21 0x000103652e20 in _PyRun_AnyFileObject pythonrun.c:77
    #22 0x0001036aadf4 in pymain_run_file main.c:429
    #23 0x0001036a980c in Py_RunMain main.c:775
    #24 0x0001036a9f84 in pymain_main main.c:805
    #25 0x0001036aa288 in Py_BytesMain main.c:829
    #26 0x000182a2ab48  (<unknown module>)

==36838==Register values:
 x[0] = 0x0000000000000000   x[1] = 0x000000016ddd28a0   x[2] = 0x000000016ddd28c0   x[3] = 0x000000016ddd28e0
 x[4] = 0x0000000000000048   x[5] = 0x0000000000000048   x[6] = 0x000000016cdd8000   x[7] = 0x0000000000000001
 x[8] = 0x0000000000000000   x[9] = 0x0000000000000000  x[10] = 0x000000000000c09b  x[11] = 0x0000000000000000
x[12] = 0xffffffffffffffff  x[13] = 0x0000000000000000  x[14] = 0x00007fffffffffff  x[15] = 0x000010700001ffff
x[16] = 0x0000000000000018  x[17] = 0x0000603000005320  x[18] = 0x0000000000000000  x[19] = 0x000000016ddd2a20
x[20] = 0x0000000103bb0690  x[21] = 0x0000007000020000  x[22] = 0x0000611000fdfc50  x[23] = 0x00006290000057c8
x[24] = 0x00006290000057d0  x[25] = 0x00006290000057d0  x[26] = 0x0000611000fdfc58  x[27] = 0x0000000000000035
x[28] = 0x0000007000020000     fp = 0x000000016ddd2810     lr = 0x00000001083b71d0     sp = 0x000000016ddd27f0
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV pcg64.h:399 in pcg64_cm_next64
==36838==ABORTING

Ping @rkern because this is happening inside the Cython RNG code.

BitGenerator.__init__() (re)sets the state to a null pointer before trying to process the arguments. When you instantiate the object and then (re)run the __init__() with an invalid argument, the state gets (re)set but the exception gets in the way. In typical usage, this exception won't be a problem because one typically instantiates and runs the __init__() in the same expression before the user gets a hold of the instance, so they never get a chance to call methods on the invalid instance. But in this circumstance of rerunning __init__(), you already have a reference to the instance.

https://github.com/numpy/numpy/blob/main/numpy/random/bit_generator.pyx#L525

Another, slightly more realistic, reproducer would be to call __new__() then __init__() separately.

>>> import numpy as np
>>> bg = np.random.PCG64.__new__(np.random.PCG64)
>>> bg.random_raw()
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[3], line 1
----> 1 bg.random_raw()
File bit_generator.pyx:663, in numpy.random.bit_generator.BitGenerator.random_raw()
File _common.pyx:95, in numpy.random._common.random_raw()
AttributeError: __exit__
>>> bg.lock is None
True
>>> bg.__init__(("",))
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[7], line 1
----> 1 bg.__init__(("",))
File _pcg64.pyx:123, in numpy.random._pcg64.PCG64.__init__()
File bit_generator.pyx:535, in numpy.random.bit_generator.BitGenerator.__init__()
File bit_generator.pyx:315, in numpy.random.bit_generator.SeedSequence.__init__()
File bit_generator.pyx:389, in numpy.random.bit_generator.SeedSequence.get_assembled_entropy()
File bit_generator.pyx:148, in numpy.random.bit_generator._coerce_to_uint32_array()
File bit_generator.pyx:138, in numpy.random.bit_generator._coerce_to_uint32_array()
ValueError: unrecognized seed string
>>> bg.lock is None
False
>>> bg.random_raw()
[1]    2766765 segmentation fault (core dumped)  ipython --profile doctest

From the first example, we can see that we can prevent most of the badness if we also make sure to reset the lock attribute back to None if the rest of the __init__() fails (e.g. with a try: except Exception: self.lock = None; raise. Access to the state should be protected by the lock everywhere, so keeping it None when we think it is likely to be invalid will cause graceful(ish) exceptions instead of segmentation faults. This probably only needs to be done in BitGenerator.__init__(), but Philox.__init__() might also need a similar treatment in case its handling of its key argument fails.

BUG: preserve BitGenerator state after failed reinitialization

Navigation Menu

Repository navigation

BUG: segfault from `random_raw` from instance initialized from invalid value #28784

Describe the issue:

Reproduce the code example:

Error message:

Python and NumPy Versions:

Runtime Environment:

Context for the issue:

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

BUG: segfault from random_raw from instance initialized from invalid value #28784

Description

Describe the issue:

Reproduce the code example:

Error message:

Python and NumPy Versions:

Runtime Environment:

Context for the issue:

Activity

ngoldbaum commented on Apr 23, 2025

rkern commented on Apr 23, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

BUG: segfault from `random_raw` from instance initialized from invalid value #28784