Skip to content

BUG: Running test suite in parallel with AddressSanitizer results in errors and crashes #29890

Open

Description

@bwhitt7

Describe the issue:

When running the NumPy test suite with pytest-run-parallel and AddressSanitizer, some tests result in ASan errors and a few even result in fatal terminations that crash the entire terminal.

Affected tests

  • numpy/_core/tests/test_arrayprint.py::test_multithreaded_array_printing
    • Fatal termination, no output or specific error
  • numpy/_core/tests/test_shape_base.py::TestConcatenate::test_huge_list_error
    • Fatal termination, no output or specific error
  • numpy/_core/tests/test_array_coercion.py::TestArrayLikes::test_too_large_array_error_paths
    • No crash, failed to allocate bytes ASan error
  • numpy/_core/tests/test_multiarray.py::TestCreation::test_malloc_fails
    • No crash, failed to allocate bytes ASan error
  • numpy/lib/tests/test_io.py::TestFromTxt::test_utf8_file_nodtype_unicode
    • Fatal termination after printing heap-use-after-free on address ASan error

Steps to reproduce

  • Use Python 3.14t-dev with AddressSanitizer enabled.
    • Steps on how to do this can be found here or in this PR for the docs.
    • Currently using these ASAN options export ASAN_OPTIONS=detect_leaks=0:symbolize=1:strict_init_order=true:allocator_may_return_null=1:halt_on_error=0
  • Build NumPy using clang, AddressSanitizer and without openBLAS
    • spin build --clean -- -Db_sanitize=address -Dbuildtype=debugoptimized -Dallow-noblas=true
  • Run test suite under pytest-run-parallel
    • spin test -- --parallel-threads=auto -sv

Reproduce the code example:

N/A

Error message:

============
numpy/_core/tests/test_arrayprint.py::test_multithreaded_array_printing
============
No specific error output, just prints "Terminated" after test


============
numpy/_core/tests/test_shape_base.py::TestConcatenate::test_huge_list_error
============
No specific error output, just prints "Terminated" after test


============
numpy/_core/tests/test_array_coercion.py::TestArrayLikes::test_too_large_array_error_paths
============
numpy/_core/tests/test_array_coercion.py::TestArrayLikes::test_too_large_array_error_paths ==12708==WARNING: AddressSanitizer failed to allocate 0x4000000000000000 bytes
[... prints multiple times]
==12708==WARNING: AddressSanitizer failed to allocate 0x4000000000000000 bytes
PARALLEL PASSED


============
numpy/_core/tests/test_multiarray.py::TestCreation::test_malloc_fails
============
numpy/_core/tests/test_multiarray.py::TestCreation::test_malloc_fails ==123475==WARNING: AddressSanitizer failed to allocate 0x7fffffffffffffff bytes
==123475==WARNING: AddressSanitizer failed to allocate 0x7fffffffffffffff bytes
[... prints multiple times]
==123475==WARNING: AddressSanitizer failed to allocate 0x7fffffffffffffff bytes
PARALLEL PASSED


============
numpy/lib/tests/test_io.py::TestFromTxt::test_utf8_file_nodtype_unicode
============
Full traceback for test_io.py file https://gist.github.com/bwhitt7/29d89710dc46a4feb52b8045c0c8bd7c

Python and NumPy Versions:

NumPy: 2.4.0.dev0+git20251003.da5c632
Python: 3.14.0rc3+ free-threading build (heads/3.14:460ad1aa1ba, Sep 22 2025, 23:51:20) [Clang 18.1.3 (1ubuntu1)]

Runtime Environment:

[{'numpy_version': '2.4.0.dev0+git20251003.da5c632',
  'python': '3.14.0rc3+ free-threading build (heads/3.14:460ad1aa1ba, Sep 22 '
            '2025, 23:51:20) [Clang 18.1.3 (1ubuntu1)]',
  'uname': uname_result(system='Linux', node='SPACE', release='6.6.87.2-microsoft-standard-WSL2', version='#1 SMP PREEMPT_DYNAMIC Thu Jun  5 18:30:46 UTC 2025', machine='x86_64')},
 {'simd_extensions': {'baseline': ['SSE', 'SSE2', 'SSE3'],
                      'found': ['SSSE3',
                                'SSE41',
                                'POPCNT',
                                'SSE42',
                                'AVX',
                                'F16C',
                                'FMA3',
                                'AVX2'],
                      'not_found': ['AVX512F',
                                    'AVX512CD',
                                    'AVX512_KNL',
                                    'AVX512_SKX',
                                    'AVX512_CLX',
                                    'AVX512_CNL',
                                    'AVX512_ICL']}},
 {'architecture': 'Haswell',
  'filepath': '/usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so',
  'internal_api': 'openblas',
  'num_threads': 24,
  'prefix': 'libopenblas',
  'threading_layer': 'pthreads',
  'user_api': 'blas',
  'version': '0.3.26'}]

Context for the issue:

ASan helps detect memory errors. Looks like running these tests in threads causes some sort of memory issue, so taking a look at these would help improve NumPy's thread safety.

Activity

  1. added
    39 - free-threadingPRs and issues related to support for free-threading CPython (a.k.a. no-GIL, PEP 703)
    on Oct 11, 2025
  2. rgommers commented on Oct 11, 2025

    @rgommers
    Member

    The failures in the 2nd, 3rd and 4th bullets are not relevant I think, those are all tests for very large arrays. Given that ASan requires a lot more memory than a regular build/test, it's unsurprising that those large memory tests then fail. They should be marked with the thread_unsafe marker, there isn't really a point to run those tests in parallel and they'll be quite slow on a regular (non-ASan) build as well.

    The 1st and 5th errors may need investigation.

  3. bwhitt7 commented on Oct 13, 2025

    @bwhitt7
    ContributorAuthor

    @rgommers Makes sense. I believe test_huge_list_error is already marked as thread-unsafe, which actually I should've mentioned in the bug report since that means I ran that test in a single thread. The others were all ran in parallel.

  4. kumaraditya303 commented on Dec 11, 2025

    @kumaraditya303
    Contributor

    The 1st and 5th errors may need investigation.

    The 5th issue is because the test calls locale.getpreferredencoding which internally calls setlocale which is thread unsafe https://www.man7.org/linux/man-pages/man3/setlocale.3.html. If multiple threads call setlocale then glibc would free the previous locale string and if some thread is using it while it cause use-after-free issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    00 - Bug39 - free-threadingPRs and issues related to support for free-threading CPython (a.k.a. no-GIL, PEP 703)

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions