Skip to content

ChaCha20's AVX-512(VL) assembly do not respect Microsoft x64 calling convention #15314

@jpboivin

Description

@jpboivin

Hello,

Some AVX-512 and AVX-512VL assembly code for ChaCha20 is zeroing registers that should be preserved by the callee on Win64 platforms (as per Microsoft doc). More precisely, ChaCha20_avx512 and ChaCha20_avx512vl save the nonvolatile xmm6 and xmm7 registers properly, but they do not for xmm8-xmm15. In their teardown (the done part), they clear all YMM registers with vzeroall, zeroing the remaining nonvolatile XMM registers.

This issue has been seen with OpenSSL 1.1.1k, with a corrupted caller state after an OpenSSL call.

Metadata

Metadata

Assignees

No one assigned

    Labels

    branch: 1.1.1Applies to OpenSSL_1_1_1-stable branch (EOL)branch: masterApplies to master branchtriaged: bugThe issue/pr is/fixes a bug

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions