secp256k1: fix two kernel bugs, add a fixed-iteration inverse, add an opt-in noinline switch

Four fixes and improvements to the secp256k1 kernel code (1–4), plus two changes that are not secp256k1-specific: an opt-in noinline switch (5) and the small helper it shares with three existing call sites (6). Each is described separately so any single part can be dropped.

1. Bug: out-of-bounds read in point_mul_xy() when the scalar is zero

OpenCL/inc_ecc_secp256k1.cl

const u32 multiplier = (naf[loop_start >> 3] >> ((loop_start & 7) << 2)) & 0x0f;
const u32 odd        = multiplier & 1;
const u32 x_pos      = ((multiplier - 1 + odd) >> 1) * 24;

With multiplier == 0: odd = 0, (0 - 1) >> 1 = 0x7FFFFFFF, * 24 puts x_pos about 3.2e9
words past tmps->xy[], and the next 16 statements read from there. CPU OpenCL devices fault, GPUs
read whatever is there.

Reachable: -m 28501/28502/28505/28506 and -m 30901/30902/30905/30906 derive prv_key straight
from the candidate, so a wordlist entry encoding the key 0 with a valid base58 checksum reaches it.
For -m 21700/21800 it is not realistically reachable.

Fixed by clamping a zero first-window digit to 1. k == 0 is not a valid private key, so the value
does not matter, only that the index stays in bounds.

Reproduced with AddressSanitizer against the unpatched code (details under Verification).

2. Bug: inv_mod() never terminates on a zero input

OpenCL/inc_ecc_secp256k1.cl

The file already carried the check, commented out:

// How often does this really happen? it should "almost" never happen (but would be safer)
// if ((a[0] | a[1] | ... | a[7]) == 0) return;

It is not almost-never, and the consequence is worse than a wrong result. t0 = 0 is always even,
so the shift keeps it 0 and the exit condition t0 == p is never reached. On a CPU OpenCL device
the work-item spins forever and clFinish hangs; GPUs happen to mask it. z == 0 is reachable
from point_add of P + (-P) and from a zero scalar. The guard is 7 ORs and a compare, uniform
across a warp.

3. inv_mod() now uses a fixed-iteration safegcd inverse

OpenCL/inc_ecc_secp256k1.cl, OpenCL/inc_ecc_secp256k1.h

The existing binary extended GCD has an input-dependent iteration count. Under SIMT every lane
in a warp runs to the slowest lane's count. inv_mod_safegcd() is a port of libsecp256k1's
modinv32 (bitcoin-core/secp256k1, src/modinv32_impl.h, MIT) and runs a fixed 20 x 30 = 600
divsteps for every input, so a warp finishes in lock-step. Same result for every input.

The file is now structured as two implementations plus a selector:

inv_mod_binary_gcd()   the existing code, unchanged apart from the guard above
inv_mod_safegcd()      the port
inv_mod()              picks one

Build with -D USE_LEGACY_BINARY_GCD_INV_MOD to go back to the binary GCD, either to A/B them or
for a device whose signed right shift is not arithmetic (safegcd assumes it is; the binary GCD does
not).

Two notes for review:

• safegcd_i64. hashcat has no signed counterpart to u64, and plain long is not portable
  here: under CUDA it follows the host ABI and is 32-bit on Windows, which is exactly why
  inc_types.h spells ullong as unsigned long long. This file is additionally compiled on the
  host by src/emu_inc_ecc_secp256k1.c. The typedef therefore keys off the same backend macros
  inc_vendor.h selects.
• Formatting. The ported block keeps libsecp256k1's denser layout rather than hashcat house
  style, so it can be diffed against the upstream reference. Happy to reformat if preferred.

No throughput measurement on hashcat's kernels is included. The motivation is the divergence
argument plus measurements from a downstream project with a different kernel shape, which do not
transfer directly. safegcd also uses four 9-element int arrays; on register-tight kernels that
could cost occupancy. If you would rather have the code in but the default unchanged, inverting the
#ifdef in inv_mod() is a one-line change.

4. point_to_affine()

OpenCL/inc_ecc_secp256k1.cl

The Jacobian to affine conversion (inv_mod + three mul_mod) appeared four times verbatim: three
times in point_get_coords() and once at the tail of point_mul_xy(). Now one function, four call
sites, 28 lines fewer. Behavior identical; the only difference is that the three sites in
point_get_coords() no longer clobber neg as scratch, which they overwrote immediately anyway.

Not exported in the header, matching the file's existing split: the header declares what module
kernels consume, not the field and point primitives.

5. FORCE_NO_INLINE (opt-in, off by default)

OpenCL/inc_vendor.h, src/backend.c

NO_INLINE only drops the inline static hint. LLVM-based OpenCL back-ends still inline every
DECLSPEC helper at -O3 and merge a kernel built from many large helpers into a single huge
function; greedy register allocation and SelectionDAG scheduling scale roughly super-linearly per
function, so compile time explodes. -cl-opt-disable is not an alternative — it fails to link the
static/DECLSPEC helpers with ld.lld: undefined hidden symbol.

FORCE_NO_INLINE emits the hard noinline attribute. It is deliberately separate from
NO_INLINE, which already has a meaning -m 33000 relies on.

Off by default and byte-identical to today when off: HC_NOINLINE expands to nothing, so all four
DECLSPEC branches are unchanged. Turned on with HASHCAT_FORCE_NO_INLINE, following the existing
HASHCAT_PIPE / HASHCAT_MEMORY convention. Out-of-line calls cost runtime throughput, so this
must never become a default.

Honest scope note: the multi-minute AMD compile was measured on a downstream project's larger
kernel, not on hashcat's. I have no evidence hashcat's kernels hit it today. This adds the escape
hatch and documents the trap; it does not claim to fix an observed hashcat problem. Easy to drop —
it is confined to these two files.

6. hc_env_flag()

src/shared.c, include/shared.h, src/backend.c, src/pwpipe.c

Item 5 would have been a fourth copy of this:

static int on = -1;
if (on == -1) on = (getenv ("NAME") != NULL) ? 1 : 0;
const bool result = (on == 1) ? true : false;
return result;

Extracted instead. The cache slot stays with the caller, because one static inside a shared function
would be a single slot shared by every variable. pipe_enabled, memory_debug_enabled and
pw_pipe_sync were converted; each drops from 7 lines to 3. HASHCAT_SALT_INNER was left alone —
it parses a value rather than testing presence and has an extra condition.

This is the one part that touches code unrelated to secp256k1. Split out on request.

Verification

Built and run in Docker (Ubuntu 24.04, gcc, pocl 3.0 CPU device). Clean build, no new warnings under
-W -Wall -Wextra.

Self-test plus a real crack against each module's own ST_HASH/ST_PASS, -D 1 on pocl:

FORCE_NO_INLINE is confirmed to reach the compiler and not silently no-op: the kernel cache hash
changes from m28501_a0-pure.6bd5126c to 2de7abca.

Native harness over the kernel sources (host IS_NATIVE path), ASan + UBSan:

sizeof(safegcd_i64) = 8
[1] safegcd vs binary GCD, 20000 random values: OK (0 failures)
[2] inv_mod_binary_gcd(0) terminates, result 0: OK
[3] inv_mod_safegcd(0)    terminates, result 0: OK
[4] point_mul_xy(k=0) returned without fault: OK
[5] point_mul_xy(k=1) == G: OK

[1] checks both inverses against each other and a * a^-1 == 1 mod p.

Negative controls — the same harness with each guard removed, to show the tests are not vacuous:

• without the inv_mod zero guard: hangs after [1], killed at 20 s
• without the point_mul_xy clamp:

ERROR: AddressSanitizer: SEGV on unknown address 0x79e760e00300
The signal is caused by a READ memory access.
    #0 point_mul_xy  OpenCL/inc_ecc_secp256k1.cl:2150

Not verified

• CUDA, HIP and Metal. No toolkit available. The signed long long branch of safegcd_i64 is
  the one that matters for Windows + CUDA and it has not been executed — only constructed to mirror
  inc_types.h's own ullong handling. Worth a check on your side.
• Throughput. No before/after numbers on hashcat kernels, for either safegcd or the
  register-pressure question (see item 3).
• AMD compile time, the premise of item 5 (see the note there).