[Scratchpad] Inference signals for turn-based routing with Dynamo #1307

What we want to test

StageRouter uses the conversation and tool results to choose an efficient or capable model. This proposal adds Dynamo's cache and load observations before the answer-generating request is sent.

The question is whether those observations can reduce coding-task completion time while maintaining task success. The proposed first experiment uses two fixed models, both served by Dynamo. The exact models are still to be chosen.

Only Stage's efficient selections would be reconsidered. Its capable selections stay unchanged. The five rules below are separate benchmark variants, with one rule selected for each run.

How a request would run

Stage proposes a model before the request is sent. The experimental rule uses Dynamo's worker/cache/load observations to keep that choice or change efficient to capable. Switchyard then reserves the chosen worker, sends the request, and releases the reservation on completion or cancellation.

flowchart TD
    T["Agent request"] --> S["Stage proposes a model"]
    S --> D["Dynamo observations<br/>candidate worker, cache and load"]
    S --> H["Apply one rule"]
    D --> H
    H --> O{"Change efficient to capable?"}
    O -->|Yes| C["Choose capable"]
    O -->|No| K["Keep Stage's choice"]
    C --> B["Reserve chosen worker<br/>and send request"]
    K --> B
    B --> W["GPU worker generates response"]
    W --> F["Complete or cancel<br/>release reservation"]
    H -.-> L["Benchmark logs<br/>offline analysis"]
    W -.-> L

Switchyard chooses the model; Dynamo chooses the worker within that model's pool. The selector is a Rust library inside Switchyard; GPU workers hold the models and KV cache. Inspecting a candidate does not reserve it or run inference.

For example, Stage proposes efficient, but its candidate worker is busy. Predicted TTFT (time to first token) is 1.4 seconds for efficient and 0.9 seconds for capable. With a required advantage of 0.2 seconds:

advantage = 1.4 - 0.9 = 0.5 seconds
0.5 > 0.2 -> choose capable

If the predicted advantage is only 0.05 seconds, the rule keeps efficient. The margin is a comparison threshold, not a delay. These numbers are illustrative.

On the next turn, the rule uses current observations and the same fixed parameters. It keeps no decision history of its own. Stage retains its normal session behavior, and Dynamo maintains its cache index and request accounting. Benchmark logs are used afterward for analysis and offline fitting, not as inputs to the next decision.

The observations

The rule starts with Stage's proposed model. Its serving inputs describe the prompt and Dynamo's proposed worker/rank for each candidate model, rather than a pool-wide average. Each model's cache is evaluated independently.

For example, one candidate could have:

N = 100,000 prompt tokens
K =  90,000 reusable GPU-prefix tokens
R =       4 booked requests

Reusable fraction = K / N = 90%
Uncached suffix   = N - K = 10,000 tokens
Prefill pressure  = 16,000 tracked prefill tokens / 8,000-token batch budget = 2

If reuse succeeds, about 10,000 tokens need fresh prefill. That does not imply 90% lower latency: processing the suffix and generating output still involve the existing context. P = 2 means two budget-equivalents of booked work; it does not establish an exact two-batch wait. Missing capacity remains unknown.

The adapter can record additional values without using them in the initial rule:

The observation combines separate selection, load and metadata reads, so it is not an atomic snapshot. A recent query also does not guarantee that underlying worker events are current. Competing traffic must appear in the same accounting path for its load to be visible.

The five comparisons

Normal Stage is the baseline. Each new variant uses this decision:

choice = Stage's choice
if choice == efficient AND G AND the rule's trigger is true:
    choice = capable

G is the same destination check in every experimental variant:

G =
    capable worker is schedulable
    AND required observations are valid
    AND time since collection <= configured maximum age
    AND capable booked requests < configured destination limit
    AND local capable in-flight calls < configured local limit

These checks prevent an override when the destination is unavailable, observations are unusable or too old, or the destination has reached its request limit. The local in-flight count is temporary accounting for this Switchyard process, not decision history. Limits stay fixed across variants. Collection age does not establish worker-event freshness. Missing or unusable required inputs leave Stage's choice unchanged.

The count rules compare current booked work with fixed thresholds. In count + prefill, either condition can trigger an override.

The two estimators predict the next request's TTFT as measured latency without competing traffic, plus a fitted correction for current load. The cache-blind estimator assumes no reusable prefix; the cache-aware estimator uses the router's prefix estimate. The actual caches and Dynamo worker-selection policy are identical in both variants.

zN_m = standardize(log(1 + N_m))
zR_m = standardize(log(1 + R_m))
zP_m = standardize(log(1 + P_m))

T_blind_m = Q_m(N_m, 0)
            + b0_m + bN_m * zN_m + bR_m * zR_m + bP_m * zP_m

T_cache_m = Q_m(N_m, K_m / N_m)
            + c0_m + cN_m * zN_m + cR_m * zR_m + cP_m * zP_m

How we would evaluate it

Prefix reuse and booked prefill affect when generation starts, which is why TTFT is the proposed first prediction target. Generation speed, tool calls and model quality also affect whole-task performance.

A key limitation is remaining generation work. Four requests about to finish and four beginning long generations can have similar request counts and prefill pressure. Prefill pressure can fall toward zero while the worker remains busy decoding. The controlled experiments below must test this gap.

1. Check a live baseline. Run a few Harbor coding tasks through Stage to verify scoring, model behavior, tokenization, signal capture and worker placement.
2. Collect GPU calibration data. Vary prompt length, reusable prefix length, prefill contention and generation contention. Include equal request counts with short versus long background generations, and a busy capable destination. Background requests use the same accounting path so they appear in Dynamo's load. Measure both candidates here to assess predictions.
3. Fit and select candidates. Use separate measurements for fitting, estimating uncertainty and selecting candidates. Keep all repeats of a workload condition in the same split. Select one threshold rule and one estimator for coding-task comparisons.
4. Run coding evaluations. Use the Harbor workflow and 20-task TB-Lite subset for development. Freeze settings before held-out tasks. Use repeated attempts, fresh sessions, matched starting conditions and randomized policy order. Each turn executes only the selected model.

All comparisons use the same model settings, hardware, worker-selection and dispatch path, and KVCR settings. Experimental variants collect observations in the same way. Ordinary Stage runs without the extra candidate probes; each variant includes its actual overhead.

Measured outcomes are separate from decision inputs: task success, failures/timeouts, complete task durations, actual TTFT, full-response latency, uncached input by model, routing overhead and effects on competing traffic. These results are recorded for offline analysis, not fed into the next turn. Per-request records include Stage's proposal, final choice, override/fallback reason, observations, predictions, actual worker, input/output usage and reported cache reuse. Report how often each rule could act, changed the choice or fell back to Stage.

Complete trajectories matter: paying cold prefill now may help later turns. Calibration checks predictions; coding runs test whether changing the route improves tasks. Only executed requests provide measured outcomes.

Implementation and status

The local Rust observation adapter and CPU replay work. The host would supply events and request accounting; libsy would own the rule. The proposed overrides and live discovery/tokenization/dispatch integration remain unimplemented. No live GPU or coding-task evaluation has run for this study. Implementation and deployment are paused during review.

Open decisions: model pair, benchmark cohort, generation settings, shared destination limits, rule set, prediction target, required improvement and task-success tolerance.

KV pressure and pool backlog can be recorded for later analysis. Extra KV-pressure inputs and cache-lifetime prediction are deferred. Hosted-provider history, monetary cost modeling, cross-model KV transfer, shared-cache training, a frontier-judge workflow and a second worker scheduler are outside this first study.

References

• feat(routing): add queue-aware task routing to Dynamo PreProc #1073: existing queue-aware routing in Dynamo PreProc.
• Dynamo selection service.
• Published Rust crate.