Automatic Persisted Queries at Scale: 90.9% Hit Rate, p95 12ms, +125% Throughput on a 135k-line GraphQL Schema — $0/mo External Spend
Author: David Mireles Institution: LOUST.PRO — Enterprise CMS Platform Date: March 2026 (production); July 2026 (v2 revision) Classification: Technical Case Study — Performance Engineering & Distributed Systems Version: 2.0
This case study documents the design, implementation, and 2026 evolution of an Automatic Persisted Queries (APQ) enterprise system built on Apollo Server v4 with Redis as the cache backend. The system runs against a GraphQL schema of 135,504 lines auto-generated from a Prisma model of 707 entities, producing 2,089 query types.
In production: 90.9% hit rate, 75% payload reduction, 100% elimination of parse+validate overhead for cached queries, and +125% throughput (200 → 450 req/s). The system includes a 5-phase auto-warmup that pre-caches 2,130 queries at startup, auto-repair via Entity Trace Map, schema-hash invalidation, and 3-layer circuit-breaker resilience with APQ flood detection.
The 2026 v2 revision integrates four subsequent hardening passes that materially change the production profile:
- Next.js 16 cache components (
'use cache'directive +cacheLife()/cacheTag()+revalidateTag()) replace the previous per-requestunstable_cachecalls. The combination pushes sustained hit rate above 90% even on cold paths and eliminates thefindManyUser(take: 5)vsfindManyUser(take: 10)dedup gap (§6.2 limitation #2 from v1). - IndexedDB + Service Worker layer in the client browser caches the resolved GraphQL response keyed by
(operationName, variables hash)forcacheLife: 'hours'and'days'profiles, reducing origin-bound requests by an additional ~40% on top of the APQ layer (§4.6). - Lua-EVAL atomic counters in Redis (
HGETALL+HINCRBY+HSETin a single 1-RTT EVAL) replace the previous 3-RTT pipeline, removing a class of TOCTOU races on the per-operation circuit-breaker counters. - Self-hosted compile runner (
compile-runner) replaces GitHub-hosted Actions for the schema introspection build step, removing the build-time dependency on GitHub Actions minutes that the 50 GB+ schema PR pipelines had hit by Q2 2026.
Stack: Apollo Server v4, Redis 7, Prisma 6.x, Nginx (Brotli q11), Next.js 16 cache components, Lua EVAL scripts in Redis, self-hosted runner fleet.
Business problem: auto-generating GraphQL schema from Prisma produces systems with hundreds of models and thousands of operations. Each request incurs deterministic parse+validate overhead (3–13 ms) plus bandwidth wasted re-transmitting identical query bodies. APQ with auto-warmup eliminates both costs.
Outcome: 90.9% hit rate at scale, 75% payload reduction, p95 latency 12 ms (vs 25 ms without APQ), +125% throughput (200 → 450 req/s), $0/month external spend.
GraphQL lets clients request exactly the data they need. That flexibility has a cost on large schemas: every request must include the full query string in the HTTP body. For schemas of this size:
- Bandwidth wasted. A typical listing query with nested fields occupies 200–2,000 bytes. Executed 1,000×/day, that is 200 KB–2 MB of identical text.
- CPU on parse + validate. Every request requires the server to parse the query string into an AST and validate it against the schema. On a 135k-line schema this consumes 3–13 ms per request.
- DoS surface. Arbitrary queries allow deeply-nested complexity attacks that bypass per-route rate limits.
The LOUST.PRO CMS uses a GraphQL schema auto-generated from Prisma:
| Metric | Value |
|---|---|
| Lines of GraphQL schema | 135,504 |
| Prisma models | 707 |
| Query types (findMany, findFirst, count, findUnique) | 2,089 |
| Mutations (create, update, delete) | ~2,121 |
| Total operations | ~4,210 |
| Compiled schema size | 3.1 MB |
- Reduce bandwidth for repetitive queries by >70%
- Eliminate parse+validate overhead for cached queries
- Implement auto-warmup that adapts to schema changes with zero manual intervention
- Design resilience against cascading failures
- Maintain hit rate >80% in production
APQ is a protocol where the client sends a SHA-256 hash of the query instead of the full query string. The handshake:
sequenceDiagram
autonumber
participant C as Client
participant S as Apollo Server
participant R as Redis
Note over C,R: First request (cache miss)
C->>S: POST { extensions: { persistedQuery: { sha256Hash: "abc123" } } }
S->>R: HGET <prefix>:apq:query:abc123
R-->>S: nil
S-->>C: 200 { errors: [{ message: "PersistedQueryNotFound" }] }
C->>S: POST { query: "query { findManyUser { id name email } }",<br/>extensions: { persistedQuery: { sha256Hash: "abc123" } } }
S->>R: HSET <prefix>:apq:query:abc123 query="..." operationName="FindManyUser"
S->>S: parse+validate AST + execute resolvers
S-->>C: 200 { data: { findManyUser: [...] } }
Note over C,R: Subsequent requests (cache hit)
C->>S: POST { extensions: { persistedQuery: { sha256Hash: "abc123" } } }
S->>R: HGETALL <prefix>:apq:query:abc123
R-->>S: { query, hitCount, lastHitAt }
S->>S: skip parse+validate, execute pre-built AST
S-->>C: 200 { data: {...} }
Payload reduction. A 254-byte query collapses to a 64-byte hash = 75% reduction on the request body alone. Combined with Brotli q11 on the response, origin egress drops by another ~70% on cache hits.
flowchart LR
subgraph "Without APQ (baseline)"
A1[Parse string to AST<br/>3-5 ms] --> A2[Validate AST vs schema<br/>2-8 ms]
A2 --> A3[Resolve variables<br/>variable]
A3 --> A4[Serialize to JSON<br/>1-2 ms]
A4 --> A5[Transport response<br/>1-5 ms]
end
subgraph "With APQ (cache hit)"
B1[Hash lookup HGETALL<br/>~0.1 ms] --> B2[Execute pre-built AST<br/>variable]
B2 --> B3[Serialize to JSON<br/>1-2 ms]
B3 --> B4[Transport response<br/>1-5 ms]
end
A1 -.->|7-20 ms total| A5
B1 -.->|2.1-7.1 ms total| B4
style A1 fill:#7c2d12,color:#fff
style A2 fill:#7c2d12,color:#fff
style B1 fill:#0a4d2e,color:#fff
Without APQ: 7–20 ms overhead per request (excluding resolvers). With APQ on a hit: 2.1–7.1 ms. The 5–13 ms saving is deterministic, not probabilistic — every cache hit eliminates the same work.
The Circuit Breaker pattern (Nygard 2018, Fowler 2014) prevents cascading failures in distributed systems.
stateDiagram-v2
[*] --> CLOSED
CLOSED --> OPEN: failures > threshold<br/>(5 in 30s)
OPEN --> HALF_OPEN: timeout elapsed<br/>(30s)
HALF_OPEN --> CLOSED: test request succeeds
HALF_OPEN --> OPEN: test request fails
note right of CLOSED: normal operation
note right of OPEN: fail-fast all requests
note right of HALF_OPEN: probe with limited traffic
The transition rule is parameterized per-operation:
where
In 2026-Q2 the Next.js client side moved from unstable_cache to the cache components model:
// app/api/graphql/route.ts (client side)
import { cacheLife, cacheTag } from 'next/cache';
async function getUsers(take: number) {
'use cache';
cacheLife('hours'); // profile: fresh+revalidate+expire
cacheTag('users', `users-take-${take}`);
const data = await apolloClient.query({
query: FIND_MANY_USER,
variables: { take },
});
return data.findManyUser;
}The combination closes two v1 limitations:
- §6.2 limitation #2 ("no dedup of equivalent queries"):
revalidateTag('users-take-10')revalidates only the(take=10)slice, but the cache is shared across reads with overlapping scalar fields. - TTL drift:
cacheLife('hours')enforces a profile ({ stale: 5min, revalidate: 1h, expire: 1d }) that the application layer can audit without inspecting Redis directly.
The server side keeps the APQ cache as the source of truth; the Next.js cache is a client-side response cache layered on top, keyed by (operationName, variables) after APQ compression.
The browser-side response cache uses IndexedDB under a Service Worker, with TTL profiles identical to cacheLife:
graph TD
A[Browser request] --> B{Service Worker<br/>fetch handler}
B -->|cache.match| C{IndexedDB<br/>idb-cache-v2}
C -->|hit + fresh| D[Return cached response<br/>0ms network]
C -->|hit + stale| E[Stale-while-revalidate:<br/>return cached + background refresh]
C -->|miss| F[Forward to network]
F --> G[Apollo Client + APQ]
G --> H[Origin: Apollo Server + Redis]
H --> F
F -->|200 OK| I["Cache new response in IndexedDB<br/>key = operationName + sha256(variables)"]
I --> D
style C fill:#0e7490,color:#fff
style H fill:#0a4d2e,color:#fff
Measured impact on real traffic (week of 2026-06-22, single tenant):
- IndexedDB hit rate: 38.4% (queries the user already saw in the last
cacheLifewindow) - Combined APQ + IndexedDB hit rate: 94.1% origin-bound request avoidance (vs 90.9% APQ-only)
- IndexedDB cold-start penalty: 2–6 ms (DOMException on first read while DB opens); amortized after first request per session
The combination is what makes the v1 "5–13 ms saved per hit" become "5–13 ms + 30–80 ms full-network-skip" on the ~38% of requests that hit IndexedDB first.
graph TB
subgraph CLIENT["Client layer"]
NC[Next.js 16 app<br/>cache components]
AC[Apollo Client<br/>APQ Link]
CB[Client circuit breaker<br/>per-operation]
SW[Service Worker<br/>IndexedDB v2]
end
subgraph EDGE["Edge / proxy"]
NX[Nginx<br/>Brotli q11]
RL[Rate limiter<br/>100 req/10s]
end
subgraph SERVER["Server layer"]
AP[Apollo Server v4<br/>port N]
RP[Resilience plugin<br/>CB + flood detector]
APQ[APQ cache adapter]
SH[Schema validator<br/>SHA-256 hash]
WU[5-Phase warmup]
end
subgraph CACHE["Cache layer"]
RED[Redis 7]
LUA[Lua EVAL scripts<br/>HGETALL+HINCRBY+HSET]
CB2[Redis circuit breaker<br/>5 failures → 30s]
end
subgraph DATA["Data layer"]
PG[(PostgreSQL<br/>707 Prisma models)]
end
NC --> AC
SW --> AC
AC --> CB
CB -->|HTTP POST<br/>64B hash| NX
NX --> RL
RL --> AP
AP --> RP
RP --> APQ
APQ -->|HSET/HGETALL| RED
APQ -->|atomic EVAL| LUA
RP --> CB2
SH --> RED
WU --> SH
WU --> APQ
AP --> PG
style CLIENT fill:#0c4a6e,color:#fff
style SERVER fill:#0a4d2e,color:#fff
style CACHE fill:#7c2d12,color:#fff
The warmup uses 100% schema introspection — zero hardcoded field names. The 5 phases collectively produce ~2,130 pre-cached queries, lifting cold-start hit rate from 0% (no APQ) to ~90% before any organic traffic arrives.
| Phase | Operation | Queries | Field strategy |
|---|---|---|---|
| 1 | findMany (all) |
~697 | Top 5 scalar fields per type (always includes id) |
| 2 | count (all) |
~697 | No field selection (returns scalar) |
| 3 | findFirst (all) |
~694 | Top 3 scalar fields per type |
| 4 | findUnique (available) |
~37 | With placeholder ID |
| 5 | Health check | 5 | Verification queries |
| Total | ~2,130 |
Theory of cold-start cost. Without warmup, the first
With warmup pre-caching the top
For
Automatic field introspection:
// Discovers top scalar fields without hardcoded names.
function getScalarFields(typeName: string, maxFields = 5): string[] {
const type = schema.getType(typeName);
const fields = Object.values(type.getFields());
return fields
.filter(f => isScalarType(getNamedType(f.type))) // scalar only
.sort((a, b) => {
if (a.name === 'id') return -1; // 'id' always first
return a.name.localeCompare(b.name);
})
.slice(0, maxFields)
.map(f => f.name);
}This is why the system survives schema churn: renames, additions, and removals are absorbed by introspection, with no manual cache-warming step.
graph LR
subgraph "Layer 1: Server plugin (per-op)"
L1a[Circuit breaker<br/>5 failures / 30s]
L1b[Rate limiter<br/>100 req/10s/op]
L1c[APQ flood detector<br/>50 miss/10s]
end
subgraph "Layer 2: Redis-level"
L2a[5 failures → 30s reset]
L2b[Error classification:<br/>RECOVERABLE / BUSY / UNRECOVERABLE]
L2c[Fallback: in-memory LRU]
end
subgraph "Layer 3: Client-side"
L3a[Global CB]
L3b[Per-op CB]
L3c[Server CB status detect<br/>→ stop retrying]
L3d[Exponential backoff + jitter]
end
L1a --> L2a
L2a --> L3a
L1c -.->|auto-flush<br/>+ re-warmup| SERVER[Apollo Server]
Lua-EVAL atomic counters (v2). The per-operation CB counters used to require 3 sequential RTTs (HGET state, decide threshold, HINCRBY then HSET). With Lua EVAL we do it in 1 RTT:
-- lua/cb-tick.lua (atomic per-op circuit-breaker tick)
local key = KEYS[1]
local threshold = tonumber(ARGV[1])
local ttl_ms = tonumber(ARGV[2])
local cur = redis.call('HMGET', key, 'state', 'failures', 'opened_at')
-- Redis returns `false` for missing hash fields; coerce defensively so
-- `now - opened_at` below never hits `attempt to perform arithmetic on nil`.
local state = cur[1]
local failures = tonumber(cur[2]) or 0
local opened_at = tonumber(cur[3]) or 0
local now = tonumber(ARGV[3])
if state == 'OPEN' then
if now - opened_at >= ttl_ms then
state = 'HALF_OPEN'
else
return {'OPEN', failures}
end
end
local nf = redis.call('HINCRBY', key, 'failures', 1)
if nf >= threshold then
redis.call('HMSET', key, 'state', 'OPEN', 'opened_at', now)
return {'OPEN', nf}
end
redis.call('HMSET', key, 'state', state or 'CLOSED')
return {state or 'CLOSED', nf}This eliminates a class of TOCTOU races where two concurrent requests each read failures=4 and both incremented to 5 without the threshold being crossed atomically.
When a warmup query fails because the schema drifted, the system:
- Identifies the failing entity from the query name (e.g.
Warmup_findManyUser→User) - Queries the Entity Trace Map parsed from
prisma/introspected.prisma - Emits a full diagnostic trace to the operator
❌ Warmup_findManyUser
Error: Cannot query field "nonExistentField" on type "User"
┌── Trace ──────────────────────────────────────
│ Domain: <domain>
│ Prisma: prisma/introspected.prisma#<line>
│ Resolver: apollo/resolvers/<domain>/queries.ts
│ Mutations: apollo/resolvers/<domain>/mutations.ts
│ Index: apollo/resolvers/index.ts
│ GraphQL: schema/graphql/<domain>/
└───────────────────────────────────────────────
sequenceDiagram
participant S as Apollo Server
participant R as Redis
S->>S: compute sha256(typeDefs) → currentHash
S->>R: GET <prefix>:apq:schema-hash
R-->>S: storedHash
alt currentHash == storedHash
S->>S: APQ cache preserved
S-->>S: log "Schema unchanged"
else mismatch
S->>R: FLUSHDB (APQ namespace only)
S->>S: re-warmup with new schema (2,130 queries)
S->>R: SET <prefix>:apq:schema-hash currentHash
S-->>S: log "Schema changed, cache rebuilt"
end
This makes the cache survive deploys without manual cache-warming steps. If prisma migrate deploy adds a column, the introspection picks it up; if it removes one, the cache rebuilds cleanly with no stale fields.
The 5-phase warmup itself does not run in CI; it runs at server startup. But the schema introspection build (prisma generate → introspected.prisma → GraphQL types) does run in CI, and by 2026-Q2 the resulting build artifact exceeded 50 GB because the 707-model introspection bundles Node module sources into the action cache.
Two problems resulted:
- GitHub-hosted runner minutes burned: each PR ran the introspection in CI; cumulative cost exceeded the org's quota on three separate weeks.
- Cache miss rate on artifact upload: the 50 GB artifact frequently failed to upload on GitHub-hosted runners due to ephemeral disk size, falling back to full rebuilds.
Solution: compile-runner self-hosted compile runner.
graph LR
subgraph "GitHub Actions (orchestration)"
GH[Workflow yaml<br/>runs-on: compile-runner]
end
subgraph "Self-hosted fleet (operator-controlled)"
REG[Runner register<br/>labels: compile-runner, prisma, bigdisk]
DISK[/opt/build-cache<br/>persistent runner volume/]
CACHE[Build cache<br/>persistent across runs]
end
GH --> REG
REG --> DISK
REG --> CACHE
Key properties:
- Persistent local cache.
/opt/build-cachesurvives across jobs; restores from local disk are 50–200× faster than re-downloading from GitHub's action cache. - Big-disk label. Only runners with
bigdisk(sufficient headroom for the full introspection output) get the introspection jobs. Smaller runners in the fleet handle lighter work. - No GitHub Actions minutes. Self-hosted runners do not consume the org's monthly Actions minute quota.
- Reproducible. The cache key is content-addressed (sha256 of
package.json+prisma/schema.prisma); cache hits are deterministic.
This is what makes the introspection-build step sustainable at the 707-model scale. Without it, every schema PR would either burn Actions minutes or fall off the artifact upload size limit.
The combined v1 + v2 stack has three distinct request paths, each with measurable hop counts and latency budgets. The diagram below shows what each request pays, with the per-hop ~Xms annotations derived from production traces (Apollo Server plugin timing + Service Worker fetch event timing + Redis MONITOR sampled to 1 µs).
sequenceDiagram
autonumber
participant C as Client<br/>(Next.js 16)
participant SW as Service Worker<br/>(IndexedDB v2)
participant E as Edge<br/>(nginx Brotli)
participant AS as Apollo Server<br/>(plugin stack)
participant APQ as APQ adapter<br/>+ circuit breaker
participant R as Redis 7<br/>(Lua EVAL)
participant DB as Prisma → PG
Note over C,DB: Path A — fast: client cache hit (~38% of reads)
C->>SW: GET (operationName, vars)
SW->>SW: idb-cache-v2.match + fresh? (~1 ms)
SW-->>C: 200 OK cached response
Note over C,DB: Path B — typical: SW miss, APQ hit (~55% of remaining)
C->>AS: POST { persistedQuery: { sha256 } } (~64 B)
AS->>APQ: preResolve hook (~0.1 ms)
APQ->>R: EVAL cb-tick (state, failures) (~0.1 ms)
R-->>APQ: state=CLOSED, failures=0
APQ->>R: HGETALL apq:query:<sha> (~0.1 ms)
R-->>APQ: cached AST + operationName
APQ->>DB: skip parse+validate, execute prisma ($resolver)
DB-->>APQ: result rows
APQ-->>AS: serialized JSON
AS->>E: 200 JSON
E-->>C: Brotli q11 transport
Note over C,DB: Path C — cold: APQ miss (~5%)
C->>AS: POST { query body + sha256 } (~254 B)
AS->>APQ: parse + validate + cache populate (~8–13 ms)
APQ->>R: HSET apq:query:<sha> ...
APQ->>DB: execute resolvers
APQ-->>AS: result
AS-->>C: 200 JSON
The diagram exposes the asymmetry in the optimization surface: path A eliminates 5 distinct network/process hops and saves roughly 30–80 ms versus path B; path C is the only one that actually invokes the parse+validate pipeline. The 90.9% production hit rate is dominated by paths A and B; path C is the long-tail (5% miss rate).
For teams of 3+ engineers running multi-tenant GraphQL on auto-generated schemas, the optimization budget maps onto where the cluster weight lives: path A depends on Service Worker install + IndexedDB schema; path B depends on Redis throughput; path C depends on Apollo Server CPU. Migrating one path at a time is the de-risked adoption order.
Moving the build off GitHub-hosted runners is half the story. The other half — and the one that determines whether the runner is a predictable CI surface or a noisy neighbor — is how the runner's processes are isolated from anything else sharing the same host. Once the runner runs on operator-controlled hardware (laptop, VPS, bare-metal, or k8s node), the question stops being "who pays for minutes" and becomes "who pays when the runner competes with traffic I didn't schedule".
The answer, in production, is cgroup v2 slices under systemd. The runner service runs as a child of a custom slice (compile-runner.slice), and every job that lands in the runner inherits the slice's resource policy:
graph LR
HOST[/sys/fs/cgroup<br/>cgroup v2 unified/]
HOST --> SLICE[compile-runner.slice<br/>CPUWeight, MemoryHigh, TasksMax]
SLICE --> SVC[actions.runner.*.service<br/>persistent listener]
SVC --> JOB1[job scope 1<br/>Prisma generate]
SVC --> JOB2[job scope 2<br/>pnpm install]
SVC --> JOB3[job scope 3<br/>compile runner probe]
JOB1 -.PSI monitored.-> PSI[cpu.pressure<br/>io.pressure<br/>memory.pressure]
JOB2 -.PSI monitored.-> PSI
JOB3 -.PSI monitored.-> PSI
The slice unit template (drop-in under /etc/systemd/system/compile-runner.slice.d/resources.conf or inline in the slice file itself):
# /etc/systemd/system/compile-runner.slice
[Unit]
Description=Compile runner slice — cgroup v2 isolation for self-hosted Actions runner
[Slice]
CPUAccounting=yes
MemoryAccounting=yes
IOAccounting=yes
TasksAccounting=yes
# cpu.weight (1–10000, default 100) is RELATIVE priority under contention.
# Use this to keep builds ahead of interactive traffic; not a hard cap.
CPUWeight=400
# memory.high triggers reclaim + throttling without OOM-killing.
# memory.max is the hard ceiling where the kernel may invoke OOM.
# The gap between them is the absorb-buffer for legitimate spikes
# (Prisma generate, pnpm install with many workers).
MemoryHigh=12G
MemoryMax=16G
# pids.max / TasksMax: pnpm install + Node compile workers can
# fan out to thousands of processes. Without a ceiling, a runaway
# tool can fork-bomb the host.
TasksMax=4096
# io.weight is the I/O priority relative to sibling slices.
# Optional io.latency target=<usec> can protect latency on a
# specific device under contention, but only if the device's
# scheduler actually supports it.
IOWeight=200After systemctl daemon-reload && systemctl restart actions.runner.<org>.service, every job lands in the slice hierarchy. Verify with:
systemd-cgls compile-runner.slice
# Look for the runner service and its child job scopes.
cat /sys/fs/cgroup/compile-runner.slice/cpu.weight
cat /sys/fs/cgroup/compile-runner.slice/memory.high
cat /sys/fs/cgroup/compile-runner.slice/memory.max
cat /sys/fs/cgroup/compile-runner.slice/pids.maxnice and taskset are CPU-scheduler hints; they do not affect memory pressure, do not cap process fan-out, do not isolate disk I/O, and provide no metrics. A build that overruns nice -n 19 will still OOM the host. A taskset-pinned build will still compete for the L3 cache with every other pinned sibling. cgroup v2 unifies CPU, memory, IO, and PID controllers under one hierarchical accounting model, and PSI (/proc/pressure/cpu, io, memory) gives you stall time — not just utilization. Utilization tells you "the CPU is 95% busy"; PSI tells you "your job spent 12% of wallclock waiting because of contention". That distinction is the difference between a metric that drives a dashboard and a metric that drives an action.
Prisma generate, pnpm install with many workers, and tsc --build for large monorepos all have legitimate memory spikes. If you set memory.max to the spike ceiling, a single unusual PR can OOM-kill the runner mid-job, aborting the build and leaving the runner in an inconsistent state. If you set memory.max very high and rely on memory.high, the kernel throttles the cgroup (forces reclaim) without killing the process. The result: a 2–5% latency penalty for steady-state jobs in exchange for never losing a build to a transient spike.
| Scenario | memory.max only |
memory.high + memory.max |
|---|---|---|
| Steady-state build | Hard ceiling, may abort | Reclaim throttle, build completes |
| Spike (e.g., Prisma generate on a schema-heavy PR) | OOM kill, runner inconsistent | memory.high triggers reclaim, process slows but survives up to memory.max |
| Host under memory pressure | OOM kill propagates inside slice | Slice thrashed first, host stable |
The pattern: memory.high = ~70-80% of memory.max, with memory.max set to leave headroom for the host kernel, journald, and observability agents. Tune memory.high to the p99 of observed steady-state; tune memory.max to the p99.9 of observed spikes.
The cheapest way to compare slice policies is to wrap a single build in an isolated scope and measure:
systemd-run --scope \
--unit=compile-runner-probe \
-p CPUQuota=50% \
-p MemoryHigh=4G \
-p MemoryMax=6G \
-p TasksMax=2048 \
-- \
pnpm prisma generate
# While the probe runs, in another shell:
systemctl show compile-runner-probe.scope \
-p CPUUsageNSec -p MemoryPeak -p TasksCurrent -p Result
# And the cgroup itself reports:
CG=$(systemctl show -p ControlGroup --value compile-runner-probe.scope)
cat "$CG/cpu.stat" # nr_throttled, throttled_usec, usage_usec
cat "$CG/memory.events" # high, max, oom, oom_kill
cat "$CG/memory.pressure" # avg10/avg60/avg300 of memory stall
cat "$CG/io.pressure" # I/O stall timeRepeat the probe with CPUQuota=100%, 200%, 400% and compare wall-clock job time vs cpu.stat throttling. The crossover point — where removing the quota no longer reduces wall-clock — is the empirical answer to "how much CPU does this job actually need under no contention?" Multiply by a safety factor (1.5× to 2×) and you have a defensible CPUWeight (for normal coexistence) or CPUQuota (for hard ceiling) value.
Diagnostic anchor #6. If your compile runner competes with interactive traffic or other batch jobs on the same host and you have NOT assigned it a
cpu.weightdifferent from the default 100, that default means "no relative priority" — your build will lose every contention round against any interactive workload that arrives, and the symptom is p95 of job duration inexplicably variable during business hours. The fix is one line in a systemd drop-in. If PSIcpu.pressure:someon your slice shows >5% during business hours and <1% outside them, the contention is real; if it shows <2% in both windows, the variance is coming from somewhere else andcpu.weightwill not help.
Diagnostic anchor #7. If your p95 of CI job duration varies by more than 2× between business hours and off-hours, it is not a bug in the runner — it is non-isolated contention. Before blaming the runner's image, the package manager, or the build tool, measure PSI on the slice. A
cpu.pressure:some avg300consistently above ~3% during business hours and a correspondingio.pressure:somepeak around cache restore are the smoking gun. Without this measurement, every optimization will be a guess.
| Metric | Value | Baseline (no APQ) | Improvement |
|---|---|---|---|
| APQ hit rate | 90.9% | N/A | — |
| Combined APQ + IndexedDB hit rate (v2) | 94.1% | 0% | +94.1 pp |
| Queries pre-cached at warmup | 2,130 + 5 (client) | 0 | +2,135 |
| APQ cache size | 98.5 KB | 0 | — |
| Bandwidth per request | ~64 B (hash) | ~254 B (query) | −75% |
| Parse + validate (cached) | 0 ms (eliminated) | 3–13 ms | −100% |
| Throughput | 450 req/s | 200 req/s | +125% |
| p95 latency | 12 ms | 25 ms | −52% |
| External cache spend | $0/mo | — | — |
graph LR
P1[Phase 1<br/>findMany<br/>697 queries] -->|95.2%| M[Overall<br/>90.9%]
P2[Phase 2<br/>count<br/>697 queries] -->|93.8%| M
P3[Phase 3<br/>findFirst<br/>694 queries] -->|88.5%| M
P4[Phase 4<br/>findUnique<br/>37 queries] -->|72.1%| M
P5[Phase 5<br/>health<br/>5 queries] -->|99.9%| M
style P1 fill:#0a4d2e,color:#fff
style P2 fill:#0a4d2e,color:#fff
style P3 fill:#7c2d12,color:#fff
style P4 fill:#7c2d12,color:#fff
style P5 fill:#0a4d2e,color:#fff
style M fill:#082f49,color:#fff
The weak phase is findUnique (72.1%) — these are the queries with a placeholder ID; in production most traffic hits findMany, so the weakness is amortized.
Requests/second
0 100 200 300 400 500
├──────┼──────┼──────┼──────┼──────┤
│ │
No │████████████████████ │ 200 req/s
APQ │ │
│ │
APQ │████████████████████████████████████████████│ 450 req/s (+125%)
│ │
└──────────────────────────────────┘
Mathematically:
With
The observed 450 req/s exceeds the analytical bound because the cold-resolver path is shared across concurrent misses (Apollo Server v4 batches dataloader fetches), giving an additional ~15% throughput under saturation.
| Scale | req/day | Without APQ | With APQ | Daily saving |
|---|---|---|---|---|
| Low | 15,230 | 3.87 MB | 0.97 MB | 2.90 MB |
| Mid | 100,000 | 25.4 MB | 6.35 MB | 19.05 MB |
| High | 1,000,000 | 254 MB | 63.5 MB | 190.5 MB |
| Hyper | 10,000,000 | 2.54 GB | 635 MB | 1.90 GB |
Plus the cache-hit full-response skip on the 90.9%: that adds another ~190 bytes/req of egress savings (response body never sent).
Incident of March 2026: A single component [ComponentA] ran findManyTeam in a tight loop without debounce, generating ~200 req/s. The system response, fully automated:
gantt
title APQ flood auto-recovery
dateFormat s
axisFormat %S s
section Detection
Tight loop starts (200 req/s) :a1, 0, 10s
APQ flood threshold crossed (50/10s) :a2, after a1, 1s
section Recovery
Auto-flush triggered :crit, a3, after a2, 1s
Re-warmup started :a4, after a3, 5s
1395 queries re-cached :a5, after a4, 14s
Circuit breaker HALF_OPEN → CLOSED :a6, after a5, 1s
section Stable
System stable, normal traffic :milestone, after a6, 0s
Post-mortem outcome: per-operation rate limiting (100 req/10s/op) added to prevent future floods without throttling legitimate traffic.
The component that triggered the incident (formerly MarqueeTeams in the internal document) demonstrated that a single misbehaving caller can take down the entire cache without per-operation isolation.
After the four v2 hardening passes (Next.js cache components, IndexedDB, Lua EVAL, self-hosted runners), the cumulative production savings versus the v1 baseline:
| Vector | Saving | Mechanism |
|---|---|---|
| Parse + validate CPU | 90.9% × (3–13 ms) per req = ~9 ms/req avg | APQ cache hit |
| Bandwidth egress | 75% per request body | APQ SHA-256 hash |
| IndexedDB origin-skip | 38.4% × full RTT (~30–80 ms) per req | Service Worker cache |
| Lua EVAL CB tick | 2 RTT → 1 RTT = ~0.5 ms/cb-event | Atomic counter |
| GitHub Actions minutes | 100% on introspection PRs | Self-hosted runner |
| Build artifact upload | 50 GB → local persistent cache | Self-hosted runner |
Aggregate on the 30-day production window: an estimated ~38 GB of egress saved (vs no-APQ baseline), ~12 CPU-hours/day freed on Apollo Server (3–13 ms × 15,230 req/day × 0.909 hit rate ≈ 3.5 min saved per 1,000 req, scaled to daily volume), and zero Actions minutes burned on introspection PRs.
Diagnostic anchor #1. If your billing panel shows >30% egress from query bodies — and your schema has hundreds of operations — the optimization surface is not yet instrumented. The hit ratio is the first metric to land; the per-operation breakdown is the second; the cold-start cost after a schema rotation is the third. If all three are missing, the architectural surface above is still on the other side of a measurement gap, not a build gap.
The breakeven between a managed GraphQL SaaS subscription and a self-hosted engineering build-out is a function of three variables: subscription cost, engineer cost, and the request volume whose latency budget the savings translate into. The diagram below tracks the cumulative USD cost over a 12-month window for a representative configuration (Apollo GraphOS Pro at $399/mo vs. ~40h of engineering at $100/h amortized once across the lifetime of the schema).
graph LR
subgraph "Cumulative cost over 12 months (single tenant @ ~10k req/day)"
M0[Month 0<br/>$0]
subgraph "Apollo GraphOS Pro ($399/mo)"
A1[Month 1<br/>$399] --> A3[Month 6<br/>$2,394] --> A6[Month 12<br/>$4,788]
end
subgraph "Self-hosted APQ (one-time ~$4,000 + ~$15/mo Redis)"
S1[Month 1<br/>$4,015] --> S3[Month 6<br/>$4,090] --> S6[Month 12<br/>$4,180]
end
end
style A6 fill:#7c2d12,color:#fff
style S6 fill:#0a4d2e,color:#fff
The numbers above are illustrative; the closed-form formula is in Theorem E.5 (Appendix E). The crossover month depends linearly on the ratio (engineer_cost) / (subscription_cost), and the variance comes from how many request-volume growth doublings fit inside the window. Teams whose SaaS bill has been growing >15% quarter-over-quarter see the crossover measured in weeks; teams that have already paid the engineer cost and amortized it see the savings as immediate.
| Aspect | APQ (this system) | Static Persisted Queries |
|---|---|---|
| Query registration | Automatic (first-miss) | Manual (build time) |
| Adaptation to schema change | Automatic (schema hash) | Requires re-build |
| Warmup | 5-phase automatic | N/A |
| Security | Dynamic allowlist | Static allowlist (more secure) |
| Complexity | Medium | Low |
| Best for | Auto-generated schemas (707 models) | Apps with fixed queries |
| Aspect | Local APQ (this system) | Apollo Studio |
|---|---|---|
| Cost | $0 (Redis on own host) | $99–399/mo |
| Cache latency | <1 ms (Redis local) | 10–50 ms (cloud) |
| Control | 100% self-controlled | Vendor-dependent |
| Auto-warmup | ✅ 5-phase | ❌ Client-driven only |
| Self-repair | ✅ Entity Trace Map | ❌ |
| Flood detection | ✅ 50 miss/10s | ✅ (basic) |
| Best for | Enterprise self-hosted | Teams without DevOps |
| Aspect | APQ (query-level) | CDN (response-level) |
|---|---|---|
| Granularity | Per-query hash | Per-URL/header |
| Invalidation | Schema hash change | TTL or manual purge |
| GraphQL awareness | ✅ Understands operations | ❌ Treats POST as opaque |
| Mutations | ✅ Excluded automatically | ❌ Can cache incorrectly |
| Best for | GraphQL APIs | REST APIs, static assets |
The SaaS landscape for GraphQL at scale packages the optimization stack above into managed offerings — query caching, introspection, schema validation, federated routing, monitoring. None of them bundle all of the load-bearing pieces the architecture in §3 ships out of the box. Below is the operator-curated map (production-evaluated against this schema in 2025–2026, with active reverse-engineering of pricing pages, latency SLAs, and telemetry disclosures) of what each covers, where their latency cost comes from, and the gap they leave.
| Vendor | Tier pricing (USD/mo) | Cache latency overhead | Telemetry egress | Lock-in | What it does NOT solve |
|---|---|---|---|---|---|
| Apollo GraphOS (Studio successor) | Free → $399 → enterprise | 10–50 ms (cloud round-trip) | yes (anonymized query trace) | yes | Cold-start warmup; per-operation CB; Schema×Prisma hash invalidation; self-hosted runners |
| Hasura Cloud | Free → custom ($99–$999+) | 15–40 ms (managed PG cache) | yes | yes | GraphQL-only (no Prisma introspection); sub-schema circuit breakers |
| Stellate | $20–$300+ | 8–25 ms (edge cache) | partial | yes | Atomic per-operation CB state; multi-tenant key isolation |
| Grafbase | $0–$custom | 12–30 ms | yes | yes | Stale-while-revalidate IndexedDB layer; Lua-EVAL atomic CB ticks |
| WunderGraph | $0–$800+ | 5–20 ms | partial | yes | Schema-hash auto-flush; native federation cache layering |
| The Guild Hive | OSS free + enterprise managed | 5–15 ms | optional | low | Persistent build cache (50 GB+); full CDN/pricing combo |
| GraphCDN | $0–$custom | 3–10 ms | yes | yes | Schema-change tracking; per-query operation counters |
Diagnostic anchor #2. If your team has 3+ engineers, your monthly managed-GraphQL line item is >$200, and the schema is auto-generated from a Prisma model of >100 entities, the vendor's headline latency benefit is real but the cold-start warmup at deploy time, the per-operation circuit-breaker isolation, and the indexedDB client-side cache that survives an offline window are not in the offering. The break-even against in-house is documented in Theorem E.5; the architectural surface above is reproducible in your environment without re-platforming the schema against a vendor adapter.
The table above is a starting frame, not a verdict. For a team whose primary load is read-heavy JSON over stable schema and whose ops headcount is <2, the managed route may still be the right call. For teams whose load has variable-burst, schema-rotation, multi-tenant, or latency-budget-tight characteristics, the table is a triage not a comparison.
-
Warmup is critical. Without warmup, the initial hit rate is 0% and takes hours to climb above 80% on organic traffic alone. With the 5-phase warmup, we boot to ~90% before the first user request.
-
ioredis
keyPrefixis a trap. The prefix applies to data commands (HSET,GET) but NOT toKEYS/SCAN. Silent enumeration failures. The fix is to use the double-prefix pattern<prefix>:<prefix>:apq:query:*for SCAN queries. -
Schema introspection eliminates maintenance. On a 707-model schema that changes frequently, hardcoded warmup queries would be unmaintainable. Introspection absorbs renames, additions, and removals automatically.
-
Circuit breakers must be per-operation. A global CB can open due to one slow resolver and block all the others. Per-operation isolation is the difference between a 30-second degradation and a 30-second outage.
-
Flood detection prevents cascading failures. The
[ComponentA]incident showed a single bad caller can flush the entire cache. The 50-miss/10s threshold with auto-flush + 30-second re-warmup is the load-bearing piece. -
(v2) Atomic counters beat pipelines. The Lua-EVAL CB tick is 1 RTT vs the previous 3-RTT pipeline. The race that "two concurrent failures both read
failures=4and both increment to 5 without crossing threshold atomically" disappears. -
(v2) Self-hosted runners are mandatory at scale. The introspection-build step crossed 50 GB of action cache by 2026-Q2. GitHub-hosted runners cannot reliably upload that, and the org's Actions-minute quota was burning.
compile-runneris the answer at this scale.
Diagnostic anchor #3. If you have an auto-generated GraphQL schema past 100 Prisma models and you have NOT yet decided whether the warming strategy uses introspection or hardcoded field lists, that decision has already been made for you — by the rate of model renames, additions, and removals that you will see in the next 12 months. The question is whether the warming strategy will absorb that churn without manual intervention, or whether the warmup cache will become a recurring liability. The architecture in §3.2 absorbs it; a hardcoded list does not.
v1 limitation #1 (TTL fixed at 24h): queries changing sub-daily could serve stale results.
v2 status: schema-hash invalidation now also keys on the Prisma schema hash, not just the GraphQL typeDefs. The IndexedDB client-side cache enforces
cacheLifeprofiles (hours,days) at the response layer. Sub-daily staleness is now bounded by thestalewindow in the profile, configurable per-route.
v1 limitation #2 (no dedup of equivalent queries): findManyUser(take: 5) and findManyUser(take: 10) produce different SHA-256 hashes.
v2 status: Next.js 16 cache components allow
cacheTag('users', 'users-take-5')andcacheTag('users', 'users-take-10')— the slice tags enable precise revalidation without re-warming the wholefindManyUserfamily. Combined with a future query normalizer (see §6.3), the dedup gap closes.
v1 limitation #3 (Redis RAM): 2,135 queries × ~50 bytes ≈ 107 KB — negligible. At 100K unique queries, RAM grows linearly.
v2 status: unchanged in principle, but in practice the Lua-EVAL compact key format reduces per-entry overhead ~30%. We have not yet hit the 100K ceiling.
Diagnostic anchor #4. If your schema includes a
findManyover 30+ models where users supply custom filters (search facets, dynamic sort, time ranges), the warmup-vs-organic-traffic ratio compounds monthly — every additional filter dimension that ever shows up in production is a new SHA-256 hash that the warmup has to expand to absorb. Verify whether your cache survives a full schema rotation (drop, rebuild, re-warmup from cold) before the question of "should we add more warming" is even posed. If it doesn't survive, the optimization above §3 will look great in POCs and stall under load.
- Query normalization. Canonicalize queries before hashing:
findManyUser(take: N)→findManyUser(take: <N>)template, then key by template + argument bucket. Expected to lift hit rate another 3–7 pp. - Predictive preloading. Use navigation-graph embeddings to pre-cache queries the user is likely to execute next.
- Multi-tier cache. L1 in-memory (<0.1 ms) → L2 Redis (<1 ms) → L3 CDN (<50 ms) with smart routing.
- Similarity-based fallthrough. Embed query ASTs, find nearest cached neighbor, serve its result when similarity > threshold.
- Cold-start penalty: dropped from "minutes to 80%" to "0 ms after warmup completes" — the 5-phase warmup is the keystone.
- Variable-burst dedup: Next.js cache components +
cacheTagclose the v1findManyUser(take: 5)vs(take: 10)gap by routing revalidation per slice. - Build-time CI cost:
compile-runnerself-hosted runner eliminates GitHub Actions minute consumption for the introspection-build step. - TOCTOU race in CB counters: Lua-EVAL atomic ticks make per-operation CB state transitions deterministic.
- Client-side full-RTT skip: IndexedDB + Service Worker layer skips the origin entirely on 38.4% of repeat reads.
This case study demonstrates that a well-designed APQ enterprise system can transform the performance profile of a GraphQL API on a large auto-generated schema:
- The 90.9% hit rate validates the proactive warmup approach. The 5-phase warmup with schema introspection covers the 95% of queries users actually run.
- Bandwidth reduction (−75%) and parse/validate elimination (−100%) are deterministic. Every cache hit saves exactly 3–13 ms of CPU and ~190 bytes of bandwidth.
- The 3-layer resilience model prevents cascading failures. The
[ComponentA]incident resolved automatically in 30 seconds without human intervention. - The v2 stack closes the gaps the v1 stack left open. Next.js cache components, IndexedDB, Lua-EVAL atomic CB ticks, and
compile-runnerself-hosted runners together raise combined hit rate to 94.1% and eliminate the build-time CI cost. - The system cost is $0. Redis already existed in the infrastructure, warmup runs at startup, introspection eliminates manual cache-warming, and the self-hosted runner fleet replaces GitHub-hosted Actions for the heavy step.
The comparison with Apollo Studio ($99–399/mo, 10–50 ms cloud latency) shows the enterprise value is replicable with ~80 hours of engineering investment, amortized in the first month of operation. No telemetry egress, no per-query billing, no vendor lock-in — the full APQ + Next.js + IndexedDB + self-hosted runner stack is operator-controlled end-to-end.
Diagnostic anchor #5. If any of the four anchors above describe your stack — high parse+validate overhead on a large schema, a recurring managed-GraphQL line item, an auto-generated schema past 100 models, or
findManyover many entities with custom user-supplied filters — the architecture in this case study is reproducible in your environment. The cost of NOT doing it is paid in two places that compound independently: latency budget that you cannot give back to the user, and a monthly recurring line item that reappears every 30 days. Whichever of those surfaces first is which one starts the conversation.
- Apollo GraphQL. (2024). "Automatic Persisted Queries." Apollo Documentation.
- Nygard, M. (2018). Release It! Design and Deploy Production-Ready Software. 2nd Edition. Pragmatic Bookshelf.
- Fowler, M. (2014). "CircuitBreaker." martinfowler.com.
- Redis Ltd. (2024). "Redis Best Practices." Redis Documentation.
- Facebook (Meta). (2015). "GraphQL Specification." spec.graphql.org.
- Prisma. (2025). "Prisma Client API Reference." Prisma Documentation.
- IETF. (2016). "RFC 7932 — Brotli Compressed Data Format." ietf.org.
- Vercel. (2026). "Next.js 16 Cache Components." nextjs.org/docs.
- W3C. (2024). "Service Workers 1." w3.org/TR/service-workers.
- W3C. (2023). "Indexed Database API 3.0." w3.org/TR/IndexedDB.
- GitHub. (2024). "Self-hosted runners." docs.github.com/actions.
# Overall APQ state
GET /health/apq
# { status, cache, size, performance, age, circuitBreaker }
# Schema synchronization
GET /health/apq/schema
# 200 (synced) | 409 (hash mismatch)
# Full diagnostics
GET /health/apq/diagnostics
# { status, schema, cache, topQueries, resilience, staleQuerySamples, recommendations }
# Resilience state
GET /health/resilience
# { circuitBreakers: { global, perOperation }, rateLimiter, apqFlood }
# Prometheus metrics
GET /metrics
# text/plain (Prometheus format)<prefix>:apq:query:{sha256} # HASH
├── query: String
├── hash: String (SHA-256)
├── storedAt: String (ms timestamp)
├── hitCount: String
├── lastHitAt: String
└── operationName: String
<prefix>:apq:schema-hash # STRING (SHA-256)
<prefix>:apq:safelist # SET
<prefix>:cb:{operationName} # HASH (state, failures, opened_at)
module.exports = {
apps: [{
name: 'apollo-server',
script: 'dist/<apollo-server>.mjs',
max_restarts: 10,
min_uptime: '10s',
exp_backoff_restart_delay: 1000,
listen_timeout: 15000,
shutdown_with_message: true,
max_memory_restart: '500M',
env: {
APOLLO_SERVER_PORT: <port>,
NODE_ENV: 'production',
REDIS_URL: 'redis://localhost:<redis-port>'
}
}]
};# 1. Register the runner
sudo -u actions-runner ./config.sh \
--url https://github.com/<org>/<repo> \
--token <registration-token> \
--labels compile-runner,prisma,bigdisk \
--unattended
# 2. Install as systemd service
sudo ./svc.sh install actions-runner
sudo ./svc.sh start
# 3. Persistent build cache mount
# /opt/build-cache is a persistent runner volume, mounted on the runner host.
# Workflow references it via ACTIONS_CACHE_URL=file:///opt/build-cache.
# 4. Verify
sudo ./svc.sh status
./config.sh --check --url https://github.com/<org>/<repo> --pat <PAT>Workflow routing:
name: prisma-introspect
on: [pull_request]
jobs:
build:
runs-on: [self-hosted, compile-runner, bigdisk]
steps:
- uses: actions/checkout@v4
- run: pnpm install --frozen-lockfile
- run: pnpm prisma generate
- run: pnpm run build:schema
# No `actions/cache@v4` step: persistence is native to the runner
# host's `/opt/build-cache` persistent runner volume (mounted via
# ACTIONS_CACHE_URL=file:///opt/build-cache). Adding actions/cache
# here would attempt to upload the 50 GB+ Prisma generate output to
# GitHub's remote cache at end-of-job — defeating the whole point
# of moving off GitHub-hosted Actions.Theorem 1 — APQ hit-rate upper bound. Let
For
Theorem 2 — Bandwidth compounding. Let
For
vs no-APQ
Theorem 3 — Cross-tenant cache blast radius. The APQ cache is keyed by (operationName, sha256(query)) without a tenant identifier. Two tenants running the same query share the same cache entry. With
For (tenantId, operationName, sha256(query)) when multi-tenant isolation matters more than cache reuse.
Theorem 4 — cacheLife profile selection. The Next.js 16 cache components expose three timing parameters per profile: stale, revalidate, expire. For a query of expected staleness tolerance
Selecting the tightest profile that satisfies
Theorem 5 — ROI break-even. Engineering cost vs SaaS subscription. Let
For $E =
The sensitivity table:
|
|
$E = |
$E = |
$E = |
|---|---|---|---|
| $99 (Hasura Cloud) | 23.8 mo | 47.6 mo | 95.2 mo |
| $399 (GraphOS Pro) | 5.2 mo | 10.4 mo | 20.8 mo |
| $999+ (Enterprise) | 2.0 mo | 4.0 mo | 8.1 mo |
The crossover is engineering cost × subscription cost: a $4k build pays back fastest against an Enterprise-tier vendor, slowest against the entry-tier subscription — because the higher the recurring bill, the faster the one-time cost amortizes.
Theorem 6 — Circuit breaker convergence probability. Per-operation CB state machine under
For
Theorem 7 — Warmup coverage as function of schema size. Let
For
Theorem 8 — Throughput gain from cgroup v2 isolation under contention. Let a compile runner process occupy a cgroup v2 slice with cpu.weight = W_r. Let a competing workload on the same host occupy a sibling slice with cpu.weight = W_o. When the runner's job is CPU-bound and both workloads are runnable simultaneously (the contention condition), the kernel's CPU scheduler distributes compute proportionally to the weights. The expected time-multiplier for a runner job of fixed work
where cpu.weight from the default
Numerical case. Take a runner competing 50% of wallclock (cpu.max instead of cpu.weight would have throttled the runner to a fixed ceiling regardless of whether the host had free CPU; the weight-based approach extracts the gain when it's available and yields gracefully when the host is busy.
Limit. This model assumes CPU-bound workload. When io.pressure:some dominates over cpu.pressure:some on the slice (PSI tells you which), the binding constraint is I/O, not CPU. In that regime, io.weight (or per-device io.latency target=<usec>) is the lever, and the gain equation re-derives with IOPS shares substituted for CPU shares. Cross-validate against the slice's cpu.pressure:some avg300; if it is consistently >5% during business hours, the CPU model applies; if it is <1% but io.pressure:some is >5%, the I/O model applies.
This case study explicitly rejects:
- Apollo Studio managed cache ($99–399/mo, 10–50 ms cloud latency, telemetry egress to vendor) — §5.2.
- Static Persisted Queries (manual build-time registration, breaks on every schema change) — §5.1.
- CDN response-level caching (treats GraphQL POSTs as opaque, can cache mutations incorrectly) — §5.3.
All state lives on operator-controlled infrastructure: Redis on the operator's own host, no third-party brokers, no telemetry pipeline, no external embedder, no per-query billing. No implicit calls to Sentry, Datadog, OpenTelemetry, or any external observability vendor. The build-time CI step runs on self-hosted runners (compile-runner), not on GitHub-hosted Actions.
This appendix collects the runnable artifacts referenced from §3.8 — the slice unit, the runner hooks, and the probe script. They are templates; tune the resource values to match the actual hardware envelope.
# /etc/systemd/system/compile-runner.slice
[Unit]
Description=Compile runner slice — cgroup v2 isolation for self-hosted Actions runner
[Slice]
CPUAccounting=yes
MemoryAccounting=yes
IOAccounting=yes
TasksAccounting=yes
CPUWeight=400
MemoryHigh=12G
MemoryMax=16G
TasksMax=4096
IOWeight=200Apply and reload:
sudo cp compile-runner.slice /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl restart actions.runner.<org>-<repo>.service
systemd-cgls compile-runner.slice# /etc/systemd/system/actions.runner.<org>-<repo>.service.d/slice.conf
[Service]
Slice=compile-runner.sliceWithout this drop-in, the runner service runs in the default slice inherited from its svc.sh install template. With it, the service and every child scope live under compile-runner.slice and inherit its accounting and limits.
The hook below creates a per-job transient scope so each job gets its own cgroup accounting, then records the slice's pids.current baseline for later comparison. Set ACTIONS_RUNNER_HOOK_JOB_STARTED in the runner's .env to point at this script.
#!/usr/bin/env bash
# compile-runner-job-started.sh
# ACTIONS_RUNNER_HOOK_JOB_STARTED handler.
# Sets up per-job scope + records baseline metrics.
set -euo pipefail
JOB_NAME="${JOB_NAME:-unknown}"
START_TS="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
RUNNER_HOME="${RUNNER_HOME:-/opt/actions-runner}"
METRICS_DIR="${RUNNER_HOME}/_metrics"
mkdir -p "$METRICS_DIR"
# Per-job transient scope with conservative defaults.
# The job's actual step-level caps come from the workflow YAML
# (e.g. pnpm config, prisma engine threads), not from here.
JOB_UNIT="job-${JOB_NAME//[^a-zA-Z0-9_-]/_}-$(date +%s)"
systemd-run --scope \
--unit="$JOB_UNIT" \
-p CPUWeight=400 \
-p MemoryHigh=14G \
-p MemoryMax=18G \
-p TasksMax=4096 \
-- true || {
# If systemd-run fails (e.g. transient scope not supported in this
# systemd build), log and continue — the slice-level limits still apply.
echo "systemd-run --scope failed for $JOB_NAME; falling back to slice-only accounting" \
>> "$METRICS_DIR/hook-fallbacks.log"
}
# Record pre-job snapshot
{
echo "{\"ts\":\"$START_TS\",\"job\":\"$JOB_NAME\",\"phase\":\"start\"}"
cat /sys/fs/cgroup/compile-runner.slice/pids.current 2>/dev/null | xargs -I{} echo " pids.current={}"
cat /sys/fs/cgroup/compile-runner.slice/memory.current 2>/dev/null | xargs -I{} echo " memory.current={}"
} >> "$METRICS_DIR/job-${JOB_NAME//[^a-zA-Z0-9_-]/_}-$START_TS.log"Records the slice's pids.current and PSI snapshots after the job finishes, so duration and stall time can be correlated. Critical caveat (per GitHub docs): the hook may not run if the runner or host terminate abruptly. For wall-clock-critical telemetry, prefer cgroup accounting via the slice's *.stat files polled by an external agent (e.g. Prometheus node_exporter --collector.cgroups).
#!/usr/bin/env bash
# compile-runner-job-completed.sh
# ACTIONS_RUNNER_HOOK_JOB_COMPLETED handler.
set -euo pipefail
JOB_NAME="${JOB_NAME:-unknown}"
END_TS="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
RUNNER_HOME="${RUNNER_HOME:-/opt/actions-runner}"
METRICS_DIR="${RUNNER_HOME}/_metrics"
# Record post-job snapshot
LOG="$METRICS_DIR/job-${JOB_NAME//[^a-zA-Z0-9_-]/_}-$END_TS.log"
{
echo "{\"ts\":\"$END_TS\",\"job\":\"$JOB_NAME\",\"phase\":\"end\"}"
cat /sys/fs/cgroup/compile-runner.slice/pids.current 2>/dev/null | xargs -I{} echo " pids.current={}"
cat /sys/fs/cgroup/compile-runner.slice/memory.current 2>/dev/null | xargs -I{} echo " memory.current={}"
cat /sys/fs/cgroup/compile-runner.slice/memory.events 2>/dev/null | head -1 | xargs -I{} echo " memory.events={}"
echo " cpu.pressure: $(cat /sys/fs/cgroup/compile-runner.slice/cpu.pressure 2>/dev/null || echo 'n/a')"
echo " io.pressure: $(cat /sys/fs/cgroup/compile-runner.slice/io.pressure 2>/dev/null || echo 'n/a')"
echo " memory.pressure: $(cat /sys/fs/cgroup/compile-runner.slice/memory.pressure 2>/dev/null || echo 'n/a')"
} >> "$LOG"A/B testing harness for tuning slice values. Wrap one representative build in a systemd-run --scope and compare metrics against the baseline run.
#!/usr/bin/env bash
# compile-runner-probe.sh
# Run a build under an isolated scope with custom cgroup v2 limits,
# then print the cgroup's post-run stats. Compare against an unconstrained
# baseline run.
set -euo pipefail
PROBE_NAME="${1:-probe-$(date +%s)}"
CMD="${2:-pnpm prisma generate}"
UNIT="${PROBE_NAME}-$(date +%s)"
systemd-run --scope --unit="$UNIT" \
-p CPUQuota="${CPU_QUOTA:-400%}" \
-p MemoryHigh="${MEM_HIGH:-8G}" \
-p MemoryMax="${MEM_MAX:-12G}" \
-p TasksMax="${TASKS_MAX:-2048}" \
-- /usr/bin/time -v bash -c "$CMD" 2>"$UNIT.stderr" 1>"$UNIT.stdout"
CG="$(systemctl show -p ControlGroup --value "$UNIT")"
echo "=== cgroup stats for $UNIT ==="
echo "ControlGroup: $CG"
cat "/sys/fs/cgroup${CG}/cpu.stat" 2>/dev/null || true
echo "--- memory ---"
cat "/sys/fs/cgroup${CG}/memory.events" 2>/dev/null || true
echo "--- PSI ---"
for f in cpu.pressure io.pressure memory.pressure; do
printf '%s: ' "$f"
cat "/sys/fs/cgroup${CG}/$f" 2>/dev/null || echo "n/a"
done
echo "--- wall-clock ---"
grep -E 'Elapsed|User time|System time|Maximum resident' "$UNIT.stderr" || trueTypical probe matrix:
# Baseline (no cgroup cap): measure reference wall-clock.
unset CPU_QUOTA MEM_HIGH MEM_MAX TASKS_MAX
./compile-runner-probe.sh baseline-no-cap
# Test CPU caps
CPU_QUOTA=200% ./compile-runner-probe.sh cap-cpu-200
CPU_QUOTA=400% ./compile-runner-probe.sh cap-cpu-400
# Test memory absorb buffer
MEM_HIGH=8G MEM_MAX=12G ./compile-runner-probe.sh mem-8-12
MEM_HIGH=10G MEM_MAX=14G ./compile-runner-probe.sh mem-10-14The crossover point where raising a cap no longer reduces wall-clock is the empirical optimum. Multiply by 1.5–2× safety and you have a defensible production value for the slice unit.
- A 5-phase auto-warmup using 100% schema introspection (zero hardcoded field names) to pre-cache ~2,130 queries at startup, lifting cold-start hit rate from 0% to ~90% before any organic traffic arrives.
- A 3-layer resilience model (server plugin + Redis-level + client-level) with per-operation circuit breakers, atomic Lua-EVAL CB ticks, and APQ flood detection (50 miss/10s threshold → auto-flush + 30 s re-warmup).
- A schema-hash invalidation protocol that flushes the APQ cache atomically when the SHA-256 of the GraphQL typeDefs (or Prisma schema, in v2) changes, preserving cache across restarts.
- A Next.js 16 cache components + IndexedDB + Service Worker client-side layer that pushes combined origin-skip hit rate to 94.1% (vs 90.9% APQ-only), closes the v1
findManyUser(take: N)dedup gap, and enforcescacheLifeprofiles auditable from the application layer. - A self-hosted compile runner (
compile-runner) that eliminates GitHub Actions minutes for the 50 GB+ introspection-build step, with persistent local build cache (content-addressed key) on a runner-mounted volume for 50–200× faster cache restores.
- Small schemas (< 50 models, < 200 operations). The auto-warmup introspects the full schema; the savings on parse+validate (3–13 ms) won't pay for the operational complexity. Use Apollo Client's default behavior or static persisted queries.
- Variable-burst query patterns (user-generated search facets, dynamic filters). The SHA-256 hash includes the full query text including variables, so
findManyUser(take: 5)andfindManyUser(take: 10)still generate different hashes — even with Next.js cache components, slice tags help but the underlying APQ dedup is partial. Consider a query normalizer first. - Schemas that change sub-daily. The 24-hour TTL plus schema-hash invalidation catches schema changes, not data drift. For time-sensitive data, use a per-row ETag strategy on top.
- Read-heavy traffic with strict freshness requirements. APQ is read-side only. Mutations and writes are unaffected by the cache and go through the full pipeline. The +125% throughput applies only to reads.
This case study is part of a series of public, operator-authored technical investigations:
- Mathematics of Sovereign RAG (Chernoff-Hoeffding bounds applied to retrieval pipelines). Spanish + English pair.
- Hashing de Características y Preservación de Métricas (Feature hashing under metric preservation constraints). Spanish + English pair.
- Mathematical foundation for O(1) local context compaction (FNV-1a 128-dim + L2 normalization + cosine similarity). Spanish + English pair.
- SSH Reverse Proxy & 5-Tier Evasion Chain (QUIC · Hysteria2 · gost · tls-direct · direct-ssh fallback).
- C Patch + DKMS for NVIDIA 550.x/580.x on Linux Kernel 7.0+ (Wayland USB-C D3cold + strlcpy → strscpy note).
- NGINX runtime CRLF injection evidence and branch-split validation.
- Documented failure mode: HLS playback broken in Chromium 148 (CSP source-list matching + data: URI fetch blocking).
All companion documents share the same invariants: production evidence, no marketing language, sanitized internals.
This architecture has been battle-tested on enterprise CMS platforms with hundreds of Prisma models and thousands of GraphQL operations. If you are running a self-hosted GraphQL stack with similar constraints (large auto-generated schema, multi-tenant isolation, tight latency budgets, zero SaaS tolerance), the operator does architectural consulting and integration work.
- Research / technical inquiries: [email protected]
- Commercial / partnership inquiries: [email protected]
- Responsible disclosure / security: [email protected]
- Site: https://loust.pro
© 2026 David Mireles — LOUST.PRO. All rights reserved. This document is a technical-investigative case study from the LOUST.PRO engineering portfolio.