Skip to content

Instantly share code, notes, and snippets.

@prathamhole14
Created August 4, 2026 10:20
Show Gist options
  • Select an option

  • Save prathamhole14/17d187d84837de39d190d7a7db0e0d03 to your computer and use it in GitHub Desktop.

Select an option

Save prathamhole14/17d187d84837de39d190d7a7db0e0d03 to your computer and use it in GitHub Desktop.

Created with help of AI agnet. Ran this on isolated machine therefore these results can be trusted with respect with noise.

_rational_tests Limited API (abi3) — benchmark results

python bench_rational_heap_types.py --build
python bench_rational_heap_types.py --core 2 --trials 15

Three builds of one standalone .so, same installed numpy behind all three:

variant source flag size
static 899d0c20b (main), static PyTypeObject — 65632
heap b0debb3a0 (branch), heap types — 64848
abi3 b0debb3a0 (branch) -DPy_LIMITED_API=0x030d0000 60696

heap/static is what ships (allow_limited_api is false by default); abi3/heap is the opt-in flag cost.

15 subprocesses per variant, order rotated and reversed each trial, ~50 ms batches, min-of-7, median across trials, bootstrap 95% CI. Pinned to core 2, governor=performance, boost=0. CPython 3.13.14, GCC 14.3.0, Ryzen 7 4800H. r = rational(3,4), a = np.array([r]*8), big = np.array([r]*100_000).

Per-call

Call static heap abi3 heap/static 95% CI abi3/heap
r + r 0.058 us 0.073 us 0.065 us 1.255x [1.247, 1.266] 0.891x
1k construct+free 132.05 us 149.87 us 149.93 us 1.135x [1.133, 1.137] 1.000x
-r 0.047 us 0.052 us 0.052 us 1.118x [1.110, 1.135] 0.995x
r * r 0.068 us 0.075 us 0.075 us 1.099x [1.095, 1.105] 0.992x
R(3, 4) 0.107 us 0.114 us 0.119 us 1.068x [1.037, 1.068] 1.041x
isinstance(r, R) 0.028 us 0.030 us 0.030 us 1.056x [1.046, 1.058] 1.001x
np.array([r]*8) 0.897 us 0.936 us 0.944 us 1.044x [1.022, 1.065] 1.008x
R(r) (fast path) 0.051 us 0.051 us 0.052 us 1.004x [1.000, 1.007] 1.021x
a[0] = r 0.177 us 0.177 us 0.173 us 1.000x [0.996, 1.004] 0.980x
a[0] 0.109 us 0.110 us 0.109 us 1.000x [0.998, 1.001] 0.997x
hash(r) 0.043 us 0.043 us 0.043 us 1.000x [0.999, 1.004] 1.000x
float(r) 0.038 us 0.038 us 0.038 us 1.000x [0.995, 1.004] 0.998x
r.n 0.033 us 0.033 us 0.033 us 0.999x [0.998, 1.001] 1.001x
r == r 0.035 us 0.035 us 0.035 us 0.999x [0.998, 1.001] 1.004x

Bulk (control — ufunc loops, untouched by the port)

Call static heap abi3 heap/static 95% CI
matrix_multiply(mm, mm) 322441 us 322421 us 322401 us 1.000x [0.999, 1.000]
big.sum() 2996.6 us 3001.3 us 2997.7 us 1.002x [1.001, 1.002]
big + big 1942.5 us 1942.4 us 1942.5 us 1.000x [0.999, 1.001]
big.astype(np.float64) 88.40 us 88.43 us 88.37 us 1.000x [0.999, 1.001]

Findings

  • The cost tracks allocation. R(r) takes the early-return branch, allocates nothing, costs nothing (1.004x); R(3,4) allocates and pays 1.068x. Same function, one branch apart. Roughly 5-18 ns per heap-type allocate/free cycle — Py_INCREF(type) in PyType_GenericAlloc, Py_DECREF(tp) in the new tp_dealloc, plus its PyType_GetFlags() call.

  • Quote nanoseconds, not ratios. r + r (1.255x) and a[0] (1.000x) both allocate one rational. The difference is 58 ns of surrounding work versus 109 ns.

  • Bulk is unaffected: 1.000x on three of four controls, so the per-call numbers are trustworthy. big.sum() at 1.002x is 0.2% on a pure C loop with no mechanism to reach it — not quotable.

  • The abi3 flag is near-free here, unlike the Cython result in results.md (13-18%). Hand-written C emits far less refcounting per call than Cython, so it barely pays the out-of-lining tax.

Assessment

Not a blocker. _rational_tests is test-only, no user path allocates rationals, and array work is flat.

The result that matters is for row 13: this is the first measurement of what heap-type conversion costs NumPy's scalar types, and np.float64(1) + np.float64(2) is the same shape as r + r. Budget ~5-18 ns per scalar allocation, 0% bulk when planning _multiarray_umath.

If the per-call cost ever needs reducing, the PyType_GetFlags() call in pyrational_dealloc can be short-circuited by a pointer compare against the two known non-GC types. Left simple deliberately.

@prathamhole14

Copy link
Copy Markdown
Author
#!/usr/bin/env python3
"""
A/B/C benchmark for the `_rational_tests` Limited API (abi3) port.

Companion to `bench_limited_api.py`, which measured the numpy.random port.
Same noise controls (see that file's header); the differences are:

* **Three variants, not two.**  This port changes two things at once and they
  need separating:

      A  static   - main's `_rational_tests.c`, no Py_LIMITED_API
      B  heap     - the branch's source,        no Py_LIMITED_API
      C  abi3     - the branch's source,        -DPy_LIMITED_API=0x030d0000

  B/A  = cost of the source conversion alone (static types -> heap types,
         `PyTuple_GET_ITEM` -> `PyTuple_GetItem`, custom `tp_dealloc`, ...).
         **This is the number that matters**: it is what ships, because
         `python.allow_limited_api` is false by default.
  C/B  = cost of the Py_LIMITED_API flag alone (out-of-lined Py_INCREF).
  C/A  = total cost of a fully ported abi3 build.

* **Only one .so differs between variants.**  `_rational_tests` is a
  standalone extension, so instead of building and installing three numpy
  trees we build three copies of that single module and put each in its own
  directory on sys.path.  Everything else the child imports -- the numpy that
  actually does the work -- is byte-identical across all three.  That removes
  a whole class of "did I accidentally compare two different numpys" error.

Usage
-----
    python bench_rational_heap_types.py --build          # phase 1, once
    python bench_rational_heap_types.py --core 2         # phase 2, measure

Phase 1 briefly checks out the parent commit's `_rational_tests.c` to build
variant A, then restores it.  It refuses to run if that file has uncommitted
changes, and restores git state on any exit path.

Run `--help` for knobs.  Defaults (15 trials x 3 variants) take ~5 minutes.
"""

from __future__ import annotations

import argparse
import json
import os
import random
import shutil
import statistics
import subprocess
import sys
from pathlib import Path

HERE = Path(__file__).resolve().parent
DEFAULT_NUMPY = Path("/home/ptm/Desktop/numpyWorkspace/numpy")
REL_SRC = "numpy/_core/src/umath/_rational_tests.c"

VARIANTS = ["static", "heap", "abi3"]
VARIANT_DESC = {
    "static": "main, no Py_LIMITED_API   (baseline: what ships today)",
    "heap":   "branch, no Py_LIMITED_API (what ships after this PR)",
    "abi3":   "branch, -DPy_LIMITED_API  (opt-in build)",
}

# --------------------------------------------------------------------------
# Workloads.
#
# "per-call" exercises the code the port actually rewrote: type construction
# (tp_new -> PyType_GenericAlloc), the per-result allocation inside
# PyRational_FromRational, and the new tp_dealloc.  Every scalar-producing
# operation on this dtype allocates and frees one heap-type instance, so this
# is where a heap-type regression would surface.
#
# "bulk" exercises ufunc inner loops, which the port did not touch at all.
# They are the control: if a bulk case moves, the measurement is broken, not
# the code.
# --------------------------------------------------------------------------
CASES = {
    # name:                    (setup, statement, kind)
    "rational(3,4)":           ("", "R(3, 4)", "per-call"),
    "rational(r)  [fast path]": ("", "R(r)", "per-call"),
    "r + r":                   ("", "r + r", "per-call"),
    "r * r":                   ("", "r * r", "per-call"),
    "-r  [unary]":             ("", "-r", "per-call"),
    "r == r  [no alloc]":      ("", "r == r", "per-call"),
    "r.n  [getset]":           ("", "r.n", "per-call"),
    "float(r)":                ("", "float(r)", "per-call"),
    "hash(r)":                 ("", "hash(r)", "per-call"),
    "a[0]  [getitem]":         ("", "a[0]", "per-call"),
    "a[0] = r  [setitem]":     ("", "a.__setitem__(0, r)", "per-call"),
    "isinstance(r, rational)": ("", "isinstance(r, R)", "per-call"),
    "np.array([r]*8)":         ("", "np.array(lst)", "per-call"),
    "1k construct+free":       ("", "[R(i, 7) for i in rng1k]", "per-call"),
    "big + big  (100k)":       ("", "big + big", "bulk"),
    "big.sum()  (100k)":       ("", "big.sum()", "bulk"),
    "big.astype(f8)  (100k)":  ("", "big.astype(np.float64)", "bulk"),
    "matrix_multiply 200x200": ("", "M.matrix_multiply(mm, mm)", "bulk"),
}

# Body of the child process.  Kept as a string so the child never imports this
# module, and so the only thing on its sys.path is the one variant directory.
CHILD = r'''
import gc, json, os, sys, time, timeit

target_ns = float(os.environ["BENCH_TARGET_NS"])
inner = int(os.environ["BENCH_INNER"])
core = os.environ.get("BENCH_CORE")

if core and hasattr(os, "sched_setaffinity"):
    try:
        os.sched_setaffinity(0, {int(core)})
    except OSError:
        pass

sys.path.insert(0, os.environ["BENCH_VARIANT_DIR"])

import numpy as np
import _rational_tests as M

# Report which variant actually got imported.  The directory name is the
# authority: variants A and B share a filename and differ only in content.
tag = os.path.basename(os.path.dirname(os.path.abspath(M.__file__)))

R = M.rational
r = R(3, 4)
a = np.array([r] * 8)
lst = [r] * 8
rng1k = list(range(1, 1001))
big = np.array([r] * 100_000)
mm = np.array([[R(1, 3)] * 200] * 200)

cases = json.loads(os.environ["BENCH_CASES"])
out = {}
gc.disable()
try:
    for name, (setup, stmt) in cases.items():
        ns = {"np": np, "M": M, "R": R, "r": r, "a": a, "lst": lst,
              "rng1k": rng1k, "big": big, "mm": mm}
        t = timeit.Timer(stmt, setup=setup or "pass", globals=ns)
        # Calibrate: grow `number` until one batch clears the target duration,
        # so short cases are not measured against the clock's resolution.
        number = 1
        while True:
            dt = t.timeit(number)
            if dt >= target_ns / 1e9:
                break
            number = max(number * 2,
                         int(number * (target_ns / 1e9) / max(dt, 1e-9)) + 1)
            if number > 50_000_000:
                break
        t.timeit(number)          # warmup at the final `number`
        best = min(t.timeit(number) for _ in range(inner)) / number
        out[name] = best
finally:
    gc.enable()

json.dump({"tag": tag, "times": out, "so": os.path.abspath(M.__file__)},
          sys.stdout)
'''


# --------------------------------------------------------------------------
# Phase 1: build the three variants
# --------------------------------------------------------------------------

def git(repo: Path, *args: str, check=True) -> str:
    r = subprocess.run(["git", *args], cwd=repo, capture_output=True,
                       text=True, check=False)
    if check and r.returncode != 0:
        sys.exit(f"git {' '.join(args)} failed:\n{r.stderr}")
    return r.stdout.strip()


def meson_setup(repo: Path, builddir: Path, limited: bool) -> None:
    if (builddir / "build.ninja").exists():
        return
    cmd = [sys.executable, str(repo / "vendored-meson/meson/meson.py"),
           "setup", str(builddir), "-Dbuildtype=release"]
    if limited:
        cmd.append("-Dpython.allow_limited_api=true")
    print(f"  configuring {builddir.name} ...", flush=True)
    r = subprocess.run(cmd, cwd=repo, capture_output=True, text=True)
    if r.returncode != 0:
        sys.exit(f"meson setup failed:\n{r.stdout}\n{r.stderr}")


def ninja(repo: Path, builddir: Path, target: str) -> None:
    r = subprocess.run(["ninja", "-C", str(builddir), target],
                       cwd=repo, capture_output=True, text=True)
    if r.returncode != 0:
        sys.exit(f"ninja {target} failed:\n{r.stdout}\n{r.stderr}")


def build_variants(repo: Path, out: Path) -> None:
    src = repo / REL_SRC
    if not src.exists():
        sys.exit(f"no such file: {src}")

    # Refuse to clobber uncommitted work -- phase 1 rewrites this file.
    if git(repo, "status", "--porcelain", "--", REL_SRC):
        sys.exit(f"{REL_SRC} has uncommitted changes.\n"
                 "Commit or stash them first; --build rewrites this file.")

    head = git(repo, "rev-parse", "HEAD")
    parent = git(repo, "rev-parse", "HEAD~1")
    unchanged = subprocess.run(
        ["git", "diff", "--quiet", f"{parent}..{head}", "--", REL_SRC],
        cwd=repo).returncode == 0
    if unchanged:
        sys.exit(f"HEAD and HEAD~1 have the same {REL_SRC}.\n"
                 "Check out the branch whose tip commit is the port.")

    plain = repo / "build-bench-plain"
    limited = repo / "build-bench-abi3"
    meson_setup(repo, plain, limited=False)
    meson_setup(repo, limited, limited=True)

    ext = subprocess.run(
        [sys.executable, "-c",
         "import sysconfig;print(sysconfig.get_config_var('EXT_SUFFIX'))"],
        capture_output=True, text=True).stdout.strip()
    plain_so = f"numpy/_core/_rational_tests{ext}"
    abi3_so = "numpy/_core/_rational_tests.abi3.so"

    out.mkdir(parents=True, exist_ok=True)
    for v in VARIANTS:
        (out / v).mkdir(exist_ok=True)

    try:
        print(f"[1/3] static  <- {parent[:9]} (main), plain build")
        git(repo, "checkout", parent, "--", REL_SRC)
        ninja(repo, plain, plain_so)
        shutil.copy2(plain / plain_so, out / "static" / f"_rational_tests{ext}")

        print(f"[2/3] heap    <- {head[:9]} (branch), plain build")
        git(repo, "checkout", head, "--", REL_SRC)
        ninja(repo, plain, plain_so)
        shutil.copy2(plain / plain_so, out / "heap" / f"_rational_tests{ext}")

        print(f"[3/3] abi3    <- {head[:9]} (branch), -DPy_LIMITED_API")
        ninja(repo, limited, abi3_so)
        shutil.copy2(limited / abi3_so, out / "abi3" / "_rational_tests.abi3.so")
    finally:
        # Always put the working tree back, even on Ctrl-C or a build failure.
        git(repo, "checkout", head, "--", REL_SRC, check=False)

    # Sanity: the two plain builds must actually differ.
    sa = (out / "static" / f"_rational_tests{ext}").read_bytes()
    sb = (out / "heap" / f"_rational_tests{ext}").read_bytes()
    if sa == sb:
        sys.exit("static and heap .so are byte-identical -- the two commits "
                 "did not produce different code. Nothing to measure.")

    # Bring the editable install back up to date so that `import numpy` in the
    # measured children does not trigger a rebuild mid-run.
    for d in (repo / "build" / "cp313",):
        if (d / "build.ninja").exists():
            subprocess.run(["ninja", "-C", str(d)], cwd=repo,
                           capture_output=True, text=True)

    print(f"\nvariants written to {out}")
    for v in VARIANTS:
        f = next((out / v).glob("_rational_tests*.so"))
        print(f"  {v:7s} {f.stat().st_size:>7} bytes  {VARIANT_DESC[v]}")


# --------------------------------------------------------------------------
# Phase 2: measure
# --------------------------------------------------------------------------

def check_machine(force: bool) -> dict:
    gov = boost = "?"
    try:
        gov = Path("/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor"
                   ).read_text().strip()
    except OSError:
        pass
    try:
        boost = Path("/sys/devices/system/cpu/cpufreq/boost").read_text().strip()
    except OSError:
        boost = "n/a"

    ok = (gov == "performance") and (boost in ("0", "n/a"))
    if not ok:
        msg = (f"machine is not pinned for benchmarking "
               f"(governor={gov}, boost={boost}).\n"
               "Expected governor=performance, boost=0. Fix with:\n\n"
               "  sudo sh -c 'for g in /sys/devices/system/cpu/cpu*/cpufreq/"
               "scaling_governor; do echo performance > \"$g\"; done'\n"
               "  echo 0 | sudo tee /sys/devices/system/cpu/cpufreq/boost\n\n"
               "Re-run with --force to measure anyway (results not defensible).")
        if not force:
            sys.exit(msg)
        print("WARNING: " + msg.splitlines()[0], file=sys.stderr)
    return {"governor": gov, "boost": boost}


def run_child(variant_dir: Path, cases, target_ns, inner, core):
    env = dict(os.environ)
    env.pop("PYTHONPATH", None)
    env["PYTHONHASHSEED"] = "0"
    env["BENCH_VARIANT_DIR"] = str(variant_dir)
    env["BENCH_CASES"] = json.dumps({k: (v[0], v[1]) for k, v in cases.items()})
    env["BENCH_TARGET_NS"] = str(target_ns)
    env["BENCH_INNER"] = str(inner)
    if core is not None:
        env["BENCH_CORE"] = str(core)
    # cwd=/tmp so a stray ./numpy source dir cannot shadow the install
    r = subprocess.run([sys.executable, "-c", CHILD], env=env, cwd="/tmp",
                       capture_output=True, text=True)
    if r.returncode != 0:
        sys.exit(f"child failed ({variant_dir.name}):\n{r.stderr}")
    return json.loads(r.stdout)


def bootstrap_ci(a_samples, b_samples, n=10000, seed=0):
    """95% CI for median(b)/median(a) by resampling both sides."""
    rng = random.Random(seed)
    ratios = []
    for _ in range(n):
        ra = statistics.median(rng.choices(a_samples, k=len(a_samples)))
        rb = statistics.median(rng.choices(b_samples, k=len(b_samples)))
        ratios.append(rb / ra)
    ratios.sort()
    return ratios[int(0.025 * n)], ratios[int(0.975 * n)]


def main():
    p = argparse.ArgumentParser(
        description="A/B/C benchmark the _rational_tests Limited API port.")
    p.add_argument("--repo", type=Path, default=DEFAULT_NUMPY,
                   help=f"numpy checkout (default: {DEFAULT_NUMPY})")
    p.add_argument("--variants", type=Path, default=HERE / "rational_variants",
                   help="where the three .so live (default: ./rational_variants)")
    p.add_argument("--build", action="store_true",
                   help="phase 1: build the three variants, then exit")
    p.add_argument("--trials", type=int, default=15,
                   help="processes per variant (default: 15)")
    p.add_argument("--inner", type=int, default=7,
                   help="min-of-N repeats inside each process (default: 7)")
    p.add_argument("--target-ms", type=float, default=50.0,
                   help="calibrate each case to this batch duration "
                        "(default: 50ms)")
    p.add_argument("--core", type=int, default=None,
                   help="pin children to this CPU core, e.g. --core 2")
    p.add_argument("--only", choices=["per-call", "bulk"], default=None,
                   help="restrict to one class of workload")
    p.add_argument("--force", action="store_true",
                   help="measure even if the CPU is not pinned")
    p.add_argument("--json", type=Path, default=None,
                   help="also write raw samples here")
    args = p.parse_args()

    if args.build:
        build_variants(args.repo.resolve(), args.variants.resolve())
        return

    for v in VARIANTS:
        if not list((args.variants / v).glob("_rational_tests*.so")):
            sys.exit(f"no .so for variant '{v}' in {args.variants}. "
                     "Run with --build first.")

    machine = check_machine(args.force)

    cases = {k: v for k, v in CASES.items()
             if args.only is None or v[2] == args.only}
    samples = {v: {k: [] for k in cases} for v in VARIANTS}

    for trial in range(args.trials):
        # Rotate the order every trial and reverse on odd ones, so all six
        # permutations appear and thermal drift cannot favour one variant.
        k = trial % len(VARIANTS)
        order = VARIANTS[k:] + VARIANTS[:k]
        if trial % 2:
            order = list(reversed(order))
        for label in order:
            res = run_child(args.variants / label, cases,
                            args.target_ms * 1e6, args.inner, args.core)
            if res["tag"] != label:
                sys.exit(f"expected variant '{label}' but imported "
                         f"'{res['tag']}' ({res['so']}).")
            for kk, vv in res["times"].items():
                samples[label][kk].append(vv)
        if sys.stderr.isatty():
            print(f"\rtrial {trial + 1}/{args.trials}",
                  end="", file=sys.stderr, flush=True)
    if sys.stderr.isatty():
        print("\r" + " " * 30 + "\r", end="", file=sys.stderr, flush=True)

    print("variants:")
    for v in VARIANTS:
        print(f"  {v:7s} {VARIANT_DESC[v]}")
    print(f"\ngovernor={machine['governor']}  boost={machine['boost']}  "
          f"core={args.core}\n")

    hdr = (f"{'case':26s} {'kind':9s} {'static':>9s} {'heap':>9s} {'abi3':>9s}"
           f" {'heap/st':>8s}  {'95% CI':14s} {'abi3/hp':>8s}  95% CI")
    print(hdr)
    print("-" * len(hdr))
    for name in cases:
        A, B, C = (samples[v][name] for v in VARIANTS)
        mA, mB, mC = (statistics.median(x) for x in (A, B, C))
        lo1, hi1 = bootstrap_ci(A, B)
        lo2, hi2 = bootstrap_ci(B, C)
        f1 = "" if lo1 <= 1.0 <= hi1 else ("*" if lo1 > 1.0 else "-")
        f2 = "" if lo2 <= 1.0 <= hi2 else ("*" if lo2 > 1.0 else "-")
        print(f"{name:26s} {cases[name][2]:9s} "
              f"{mA * 1e6:7.3f}us {mB * 1e6:7.3f}us {mC * 1e6:7.3f}us "
              f"{mB / mA:7.3f}x [{lo1:.3f},{hi1:.3f}]{f1:1s} "
              f"{mC / mB:7.3f}x [{lo2:.3f},{hi2:.3f}]{f2}")

    print(f"\n{args.trials} trials/variant, min-of-{args.inner} per trial, "
          f"~{args.target_ms:.0f}ms batches, rotated order.")
    print("'*' = resolved slowdown, '-' = resolved speedup, blank = CI spans "
          "1.00 (not resolved; raise --trials).")
    print("heap/st is the number that ships. abi3/hp is the opt-in flag cost.")
    print("Any 'bulk' row that resolves is a red flag: those loops were not "
          "touched, so a real change there means the run is contaminated.")

    if args.json:
        args.json.write_text(json.dumps(
            {"machine": machine, "trials": args.trials, "inner": args.inner,
             "core": args.core, "samples": samples}, indent=1))
        print(f"\nraw samples -> {args.json}")


if __name__ == "__main__":
    main()

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment