Technical reports · Oliver Yu

Oliver Yu

AI Systems Engineer · System Architect

Enterprise AI platforms and LLM inference performance.

1Abstract: At work I build enterprise AI platforms: a unified AI gateway, on-premises model serving and a coding agent. I also measure LLM inference on AMD, NVIDIA and Apple Silicon and send the fixes upstream.
2Key result: 1.61×; vLLM QSA Indexer · Geometric-mean output throughput vs main across 12 end-to-end MI355X/TP2 cases, measured independently by an AMD engineer.
3More results: 228.38 → 79.06 sllm-inference-systems · Seven-turn session latency: sparse prefill alone vs rebuilding the reusable prefix in idle time; 1,474 tok/sdeepseek-v4-flash-mi300x · Aggregate output with 32 concurrent coding sub-agents, 2× MI300X at TP=2; 92.1 %qwen3.8-27b-5070ti-eval · HumanEval+ pass@1, 151 of 164 tasks (95 % CI 86.9–95.3 %), on a single 16 GB RTX 5070 TiRound 1 makes no comparison: the first competitor spilled out of VRAM mid-run, and that run was declared invalid.
4Links: GitHub
Hugging Face
LinkedIn
X
YouTube

Featured Research

Running MLX GPU and Apple Neural Engine requests side by side exposed host-side contention that raw accelerator benchmarks missed.

Measure the handoff · Isolate GIL contention · Ship adaptive GPU + ANE serving

GPU result return (P50), once synchronous Core ML stopped holding the GIL: 7.67 → 0.14 ms

AMD AITER
Merged MI325X FP8 fused-MoE tuning configurations: 3.7–13% lower kernel time on eight measured shapes.
vLLM
Submitted ROCm QSA indexer and sparse-attention optimizations, validated on MI325X; the indexer change was also tested independently on MI355X.
LiteLLM
Submitted fixes for slow failover and uneven concurrent least-busy routing; in a two-backend reproduction, failover dropped from 72–137 s to about 6 s.
MLX
Fixed lost rank output in the distributed launcher by draining both pipes after the process exits.
oMLX
Improved SpecPrefill cache reuse, deterministic prefill routing, and Responses API tool routing.
Apple coremltools
Submitted a fix that releases the Python GIL during native Core ML prediction, so other threads keep running.

Shouri / 收理
Independent product
Saves webpages, files, and media first, then uses AI to turn them into structured, searchable knowledge. The original is always kept and can be exported.
Visit Shouri
PiShip
Agent infrastructure
A toolchain that lets a company ship Pi as its own coding agent without forking it, adding OIDC login, short-lived gateway credentials, policy, and a sandbox.
GitHub

SignalForge
An event-centric intelligence pipeline that runs every morning.
Read Today GitHub
Claude Team Kit
A Claude Code plugin that caps native Agent Teams teammates and adds a read-only Mission Control pane for workers, task dependencies, and usage.
GitHub

Apple Silicon · Inference
When host-side contention hides accelerator performance
Tracing the GIL and latency costs of concurrent GPU and Neural Engine serving.
LLM Inference · State
Recovering reusable state between inference requests
Why saving one request is not enough when the following session pays the cost.
Chinese
vLLM · ROCm · Inference
Reducing wasted work in vLLM QSA indexing
An MI325X QSA scoring profile, visible-column skipping, 16×64 tiling, and the decode cases where the first revision regressed. Includes independent MI355X results.
Chinese

2026 · Journal of Information Security and Applications

Latest writing