Skip to content

Commit 1716714

Browse files
Merge 678ec3a into 2a09c45
2 parents 2a09c45 + 678ec3a commit 1716714

19 files changed

Lines changed: 2629 additions & 37 deletions

‎benchmarks/native-sdk/README.md‎

Lines changed: 30 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -29,17 +29,19 @@ uv venv --python "$BASE_PY" "$LAB/peer"
2929
STABLE_PY="$LAB/stable/bin/python"
3030
NATIVE_PY="$LAB/native/bin/python"
3131
PEER_PY="$LAB/peer/bin/python"
32-
uv pip install --python "$STABLE_PY" 'openai-agents==0.24.0' \
32+
uv pip install --compile-bytecode --python "$STABLE_PY" 'openai-agents==0.24.0' \
3333
'openai==3.29.0' 'pydantic==2.12.3' 'typing-extensions==4.16.0'
34-
uv pip install --python "$NATIVE_PY" "$WHEEL" \
34+
uv pip install --compile-bytecode --python "$NATIVE_PY" "$WHEEL" \
3535
'pydantic==2.12.3' 'typing-extensions==4.16.0' \
3636
'jsonschema==4.26.0' 'referencing==0.37.0'
37-
uv pip install --python "$PEER_PY" \
37+
uv pip install --compile-bytecode --python "$PEER_PY" \
3838
'aiohttp==3.14.3' 'openai==3.41.0' 'pydantic==2.12.3'
3939
```
4040

4141
The wheel hook builds the native extension and matching execution helpers. Install that exact wheel into the fresh environment; do not substitute an editable checkout or an unrelated native binary. The dependency versions above come from existing compatible environment records. They are not complete dependency locks. Keep the wheel hash, source commit, Python build and complete installed package inventories with results. Use `uv pip freeze --python <interpreter>` to record each environment. If registry access is unavailable, supply the exact stable release wheel locally.
4242

43+
Compile bytecode for every environment before measurement, as shown above. Workers and peers run with `-B` to prevent cache writes; they still read existing caches. Unequal cache coverage can change first-call CPU even when package versions match. Keep before/after inventories of installed Python sources and valid, non-optimized bytecode caches, including their interpreter cache tag and source validation. Worker rows record the effective cache settings, but those settings alone do not prove complete or equal cache coverage.
44+
4345
## Verify and run a small correctness check
4446

4547
Before execution, verify the eight source hashes and inventory against `benchmark-source-manifest.json`. Keep exactly eight `.py` files beside `compare.py`; it records every sibling Python file. Run ordinary Python without optimization or instrumentation. The controller and worker reject known inherited Python path/home, allocator, preload, profiling and scheduler overrides, including empty environment values. They also reject active Python trace/profile hooks and profiling `-X` options. Clear the listed controls rather than relying on a shell that happens to omit them. `PYTHONSAFEPATH` and the controller's fixed `PYTHONHASHSEED=0` remain supported. These checks cover known controls, not every possible external profiler or modified interpreter.
@@ -102,7 +104,7 @@ conversation beyond the observation buffer's retention and rejection of a genuin
102104
SDK gap. It reports correctness only:
103105

104106
```sh
105-
uv pip install --python "$NATIVE_PY" 'aiohttp==3.14.3'
107+
uv pip install --compile-bytecode --python "$NATIVE_PY" 'aiohttp==3.14.3'
106108
"$NATIVE_PY" "$BENCH/tests/native_stream_cursor.py" -v
107109
```
108110

@@ -117,3 +119,27 @@ SDK_BENCH_SERVER_PYTHON="$PEER_PY" "$NATIVE_PY" "$BENCH/worker.py" \
117119
A `--profile` row is explicitly diagnostic. The comparison rejects diagnostic/profiled/instrumented rows rather than mixing them into latency/CPU summaries. Profiling retains the existing exercise window, which includes setup/warmup/close; instrumented counts or times are not uninstrumented latency shares.
118120

119121
Generated metadata, errors and logs can contain local absolute paths. Review and separately sanitize a public export, preserving original hashes and recording the export's own digest. This directory contains no historical raw results, profiles, installed environments or credentials. Record the source-manifest hash with results; a manifest identifies source bytes, not a performance qualification.
122+
123+
## Optional native buffered Responses mode
124+
125+
The default command keeps the native provider's streaming transport and omits the new provider keyword, so it can still benchmark older native SDK wheels. A native wheel that supports `NativeProvider(response_mode="buffered")` can opt into final-result HTTP JSON delivery with `compare.py --v1-response-mode buffered`. Unsupported wheels fail explicitly; the benchmark never retries in another mode. The stable v0 adapter uses SSE for live workloads and JSON for final-result workloads. Its behavior does not change when selecting the native mode.
126+
127+
Select only final-result workloads for a buffered comparison, for example:
128+
129+
```sh
130+
"$PEER_PY" "$BENCH/compare.py" \
131+
--v0-python "$STABLE_PY" --v1-python "$NATIVE_PY" \
132+
--v0-version 0.24.0 --v1-version 0.24.0-alpha.6 \
133+
--server-python "$PEER_PY" --output-dir "$RESULTS/buffered" \
134+
--v1-response-mode buffered --samples 5 --iterations 20 \
135+
--cases oneshot_text oneshot_tool oneshot_output_64k_1_plain \
136+
oneshot_output_64k_256_plain oneshot_output_64k_4096_plain
137+
```
138+
139+
Omit `--v1-response-mode buffered` and use a fresh output directory for the default SSE control. Keep the same source, wheel, dependencies, interpreter, CA and host conditions. The three 64 KiB cases are final-only `Agent.run` calls in independent fresh conversations, with one prefill and 20 measured calls by default. Their chunk setting controls the SSE reference workload; a JSON response has zero SSE chunks. This removes peer framing work too, so wall-time differences do not isolate client processing. Client process CPU and peer CPU remain separate.
140+
141+
Rows record the native response mode and expected wire transport, checked against all actual request receipts, per-response chunk counts and total emitted chunks. Both modes retain the same final output, input/history and tool-result checks. Buffered comparisons reject live-observation workloads. They report no first-delta metric or token-live claim; buffered progress arrives after the complete body. Existing live streaming cursor, gap and exact chunk checks remain unchanged.
142+
143+
`elapsed_ms`, `cpu_ms` and latency p50/p95 still cover measured operations after prefill. For oneshot cases, these are warmed fresh-conversation measurements. New `full_owner_wall_ms` and `full_owner_cpu_ms` span Adapter construction through successful joined provider/telemetry exit, including prefill and ordinary client scheduling, prompt work and in-scope peer-stat requests. They exclude SDK import, peer startup and final correctness validation. These spans overlap the existing metrics and must not be added to them. Failed runs cannot qualify.
144+
145+
`charged_operations = concurrency * (prefill + iterations)` counts turns, including prefill. It counts fresh conversations only for oneshot cases; conversation cases reuse Sessions. Summaries apply the existing paired estimator to `full_owner_wall_ms_per_operation` and `full_owner_cpu_ms_per_operation`, retaining raw totals and matched vectors. Import-only rows have no owner/operation/transport metrics. RSS stays unqualified and does not enter comparisons. The existing confidence-interval limitations and engineering parity rule still apply.
Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
11
{
2-
"compare.py": "684a4dba0201b4d6fa23f5ff6decf64334af0596a532daeaa1891624c01ca007",
2+
"compare.py": "dd853b968679dd5b5e1d833516abf52ea9e139169dfbd74b375a59ef033b470f",
33
"owned_process.py": "9be3bd2a11154dec666a943172fd7645e040ef7279f0f53633d5c3f22d168e4e",
44
"protocol_peer.py": "912165f4e3d2abc8d45a0690e08e7b564fcc0562d1ea1a7725de92ba0f841bee",
55
"v0_adapter.py": "36d8e51ce6743c82bc73385c79241a2dddcd2b259ef7f3207f53d0721fccf6d2",
6-
"v1_adapter.py": "5649d0688172337c760af79db245258990be81b0ac6af19b31f74b480f11a350",
6+
"v1_adapter.py": "63e6a993a82c83ec6bfe3f76cb1fab601e25dd92f73e68c5f77423fe274f784b",
77
"validate_peer.py": "5d0829b7cc68310cbfc861bff8927b2e51051757a7d74e0a1b8faa48ab87b527",
8-
"worker.py": "266ee4533be914889fcd00e6fc748d1e8fb078fb54b1012223e99bd6f703d7df",
9-
"workloads.py": "ba5539876ecc2b0ef7d3c0911b89006b207c3fa92c79505a976107736bb82062"
8+
"worker.py": "7fcc4b120ccbb5f3e1c8da49b792a7e323b634454618f4abdc6d8c4b5e132167",
9+
"workloads.py": "a86d44a82b16c6fcb4097ce8a4487093ddc676029836020eebff46c6cf013546"
1010
}

‎benchmarks/native-sdk/compare.py‎

Lines changed: 87 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@
1818

1919
sys.path.insert(0, str(Path(__file__).resolve().parent))
2020
from owned_process import _note, defer_cleanup_signals, terminate_group
21-
from worker import reject_inherited_overrides
21+
from worker import reject_inherited_overrides, validate_transport
2222
from workloads import WORKLOADS, get_workload
2323

2424

@@ -78,7 +78,17 @@ def comparison(stable, native):
7878
}
7979

8080

81-
def validate_sample(row, *, side, case, sample, expected_versions, identities, iterations=None):
81+
def validate_sample(
82+
row,
83+
*,
84+
side,
85+
case,
86+
sample,
87+
expected_versions,
88+
identities,
89+
iterations=None,
90+
v1_response_mode="streaming",
91+
):
8292
"""Check scheduling, declared identities and existing worker count receipts."""
8393
if row.get("errors") != 0:
8494
raise ValueError("Worker validation failed")
@@ -105,17 +115,22 @@ def validate_sample(row, *, side, case, sample, expected_versions, identities, i
105115
"cache_tag": str,
106116
"pointer_bits": int,
107117
"abiflags": str,
118+
"dont_write_bytecode": bool,
119+
"pycache_prefix": type(None),
108120
}
109121
if (
110122
not isinstance(build, dict)
111-
or any(type(build.get(name)) is not kind for name, kind in build_types.items())
123+
or any(
124+
name not in build or type(build[name]) is not kind for name, kind in build_types.items()
125+
)
112126
or build["implementation"] != "cpython"
113127
or not build["cache_tag"]
128+
or not build["dont_write_bytecode"]
114129
or type(row.get("python")) is not str
115130
or not row["python"]
116131
or build["version"] != row["python"]
117132
):
118-
raise ValueError("Missing or inconsistent CPython build/ABI identity")
133+
raise ValueError("Missing or inconsistent CPython runtime identity")
119134
module = row.get("module_file")
120135
if not isinstance(module, str) or not Path(module).is_absolute():
121136
raise ValueError("Worker module path is missing or not absolute")
@@ -124,12 +139,57 @@ def validate_sample(row, *, side, case, sample, expected_versions, identities, i
124139
)
125140
if row.get("workload") != workload.dictionary():
126141
raise ValueError("Worker workload differs from the scheduled work")
142+
if v1_response_mode not in ("streaming", "buffered"):
143+
raise ValueError("Unknown scheduled native response mode")
144+
expected_mode = v1_response_mode if side == "v1" and case != "startup_import" else None
145+
if "response_mode" not in row or row["response_mode"] != expected_mode:
146+
raise ValueError("Native response mode differs from the scheduled mode")
147+
expected_transport = (
148+
None
149+
if case == "startup_import"
150+
else (
151+
"sse"
152+
if (workload.stream if side == "v0" else v1_response_mode == "streaming")
153+
else "json"
154+
)
155+
)
156+
if "wire_transport" not in row or row["wire_transport"] != expected_transport:
157+
raise ValueError("Wire transport metadata differs from the scheduled mode")
127158
if case == "startup_import":
128159
if any(
129-
name in row for name in ("operations", "server", "completed_operations", "tool_calls")
160+
name in row
161+
for name in (
162+
"operations",
163+
"server",
164+
"completed_operations",
165+
"tool_calls",
166+
"charged_operations",
167+
"full_owner_wall_ms",
168+
"full_owner_cpu_ms",
169+
)
130170
):
131171
raise ValueError("Import-only row contains operation/server work")
132172
else:
173+
if v1_response_mode == "buffered" and workload.stream:
174+
raise ValueError("Buffered benchmarks cannot measure token-live workloads")
175+
validate_transport(
176+
workload, row.get("server", {}), engine=side, response_mode=v1_response_mode
177+
)
178+
charged = workload.concurrency * (workload.prefill + workload.iterations)
179+
if type(row.get("charged_operations")) is not int or row["charged_operations"] != charged:
180+
raise ValueError("Charged operation count differs from the scheduled work")
181+
for clock in ("wall", "cpu"):
182+
total = row.get(f"full_owner_{clock}_ms")
183+
per_operation = row.get(f"full_owner_{clock}_ms_per_operation")
184+
if (
185+
type(total) not in (int, float)
186+
or not math.isfinite(total)
187+
or total <= 0
188+
or type(per_operation) not in (int, float)
189+
or not math.isfinite(per_operation)
190+
or not math.isclose(per_operation, total / charged, rel_tol=1e-12)
191+
):
192+
raise ValueError("Invalid full-owner metric or operation denominator")
133193
count = workload.concurrency * workload.iterations
134194
measured_tools = count * workload.tools_per_round * workload.tool_rounds
135195
all_tools = (
@@ -168,7 +228,7 @@ def validate_sample(row, *, side, case, sample, expected_versions, identities, i
168228
identities[side] = identity
169229

170230

171-
def summarize(records, *, expected_versions, iterations=None):
231+
def summarize(records, *, expected_versions, iterations=None, v1_response_mode="streaming"):
172232
summary, identities = {}, {}
173233
for case, sides in records.items():
174234
if set(sides) != {"v0", "v1"}:
@@ -195,6 +255,7 @@ def summarize(records, *, expected_versions, iterations=None):
195255
expected_versions=expected_versions,
196256
identities=identities,
197257
iterations=iterations,
258+
v1_response_mode=v1_response_mode,
198259
)
199260
fields = ["import_ms"]
200261
if case != "startup_import":
@@ -207,6 +268,8 @@ def summarize(records, *, expected_versions, iterations=None):
207268
"elapsed_ms",
208269
"cpu_ms",
209270
"final_close_ms",
271+
"full_owner_wall_ms_per_operation",
272+
"full_owner_cpu_ms_per_operation",
210273
]
211274
if case != "startup_import" and WORKLOADS[case].stream:
212275
fields += ["first_delta_p50_ms", "first_delta_p95_ms"]
@@ -241,6 +304,9 @@ def main():
241304
parser.add_argument("--v1-python", type=Path, required=True)
242305
parser.add_argument("--v0-version", required=True)
243306
parser.add_argument("--v1-version", required=True)
307+
parser.add_argument(
308+
"--v1-response-mode", choices=["streaming", "buffered"], default="streaming"
309+
)
244310
parser.add_argument("--server-python", type=Path, required=True)
245311
parser.add_argument("--output-dir", type=Path, required=True)
246312
parser.add_argument("--samples", type=int, default=5)
@@ -265,6 +331,12 @@ def main():
265331
or not args.v1_version.strip()
266332
):
267333
parser.error("Use positive counts/timeout, distinct cases and explicit expected versions")
334+
if args.v1_response_mode == "buffered" and any(
335+
case != "startup_import" and WORKLOADS[case].stream for case in args.cases
336+
):
337+
parser.error(
338+
"Buffered comparisons require final-result workloads; select --cases explicitly"
339+
)
268340
expected_versions = {"v0": args.v0_version, "v1": args.v1_version}
269341
identities = {}
270342

@@ -295,6 +367,7 @@ def stop_requested(signum, _frame):
295367
},
296368
"sample_order": [],
297369
"expected_sdk_versions": expected_versions,
370+
"v1_response_mode": args.v1_response_mode,
298371
"completed": False,
299372
"rss_scope": "Unqualified raw process ru_maxrss only; no RSS comparison or verdict",
300373
"statistics": (
@@ -333,6 +406,7 @@ def stop_requested(signum, _frame):
333406
executable = args.v0_python if side == "v0" else args.v1_python
334407
command = [
335408
str(executable.absolute()),
409+
"-B",
336410
str(directory / "worker.py"),
337411
"--engine",
338412
side,
@@ -341,6 +415,8 @@ def stop_requested(signum, _frame):
341415
"--sample-index",
342416
str(sample),
343417
]
418+
if side == "v1" and args.v1_response_mode == "buffered":
419+
command += ["--response-mode", "buffered"]
344420
if case == "startup_import":
345421
command.append("--import-only")
346422
else:
@@ -421,6 +497,7 @@ def stop_requested(signum, _frame):
421497
expected_versions=expected_versions,
422498
identities=identities,
423499
iterations=args.iterations,
500+
v1_response_mode=args.v1_response_mode,
424501
)
425502
records[case][side].append(data)
426503
value = data.get("latency_p50_ms", data["import_ms"])
@@ -430,7 +507,10 @@ def stop_requested(signum, _frame):
430507
flush=True,
431508
)
432509
summary = summarize(
433-
records, expected_versions=expected_versions, iterations=args.iterations
510+
records,
511+
expected_versions=expected_versions,
512+
iterations=args.iterations,
513+
v1_response_mode=args.v1_response_mode,
434514
)
435515
(args.output_dir / "summary.json").write_text(json.dumps(summary, indent=2) + "\n")
436516
primary = summary[case]["primary"]

0 commit comments

Comments
 (0)