Skip to content

Trace import loses per-request fidelity; explore a replay/recording mode #209

Description

@andrewh

Summary

When importing a topology from real traces (pkg/synth/traceimport), the importer collapses everything into an aggregate probabilistic model. Running that topology then produces a statistically similar synthetic population, never the imported requests themselves. This issue captures the fidelity gap and lays out options — from a cheap correctness fix to a full replay mode — for maintainers to choose between.

How the engine works today

The engine is a top-down generator from aggregate parameters. Per trace (engine.go:176-194, mirrored for plan mode at :321-353):

  1. Traffic rate decides how many root traces and when (trafficPattern.Rate(elapsed)).
  2. A root is chosen at random: root := e.Topology.Roots[e.Rng.IntN(len(e.Topology.Roots))].
  3. walkTrace recurses, sampling at every span:
    • ownError = e.Rng.Float64() < errorRate
    • ownDuration = duration.Sample(e.Rng)
    • call firing: e.Rng.Float64() < call.Probability
    • attribute values: a.Gen.Generate(e.Rng)
    • multiplicity: a static Count loop (engine.go:563-564)

A "trace" is never stored — it is synthesized on the fly from per-operation distributions.

What the importer discards

The importer builds per-trace trees (traceimport/tree.go) and then aggregates them away (stats.go, marshal.go):

  • Exact per-request call sequences and multiplicity. cs.Count / TotalCount becomes a single probability; per-request call count is lost, and prob >= 1.0 collapses to a bare unconditional target — so "called 3× every request" and "called once every request" emit identically.
  • Per-span durationsmean +/- stddev.
  • Per-span attribute values (only attributes constant across all spans survive as resource attributes).
  • Which conditional branch actually fired (on-error/on-success history → aggregate probability).
  • Actual parallel/sequential ordering → a single voted call_style per operation.

Why the existing determinism primitives don't solve this

Two things look like replay infrastructure but are the wrong shape:

Seeded RNG gives byte-identical runs, but reproduces a synthetic population, not the imported one.

forcedChoice / choiceDecisions (swarm.go:60-91) override probabilistic branches, but:

  • cover only three boolean choice kinds (operation-error, call-probability, retry-activation) — nothing for durations, attributes, multiplicity, fan-out, root choice, or arrival timing;
  • are keyed {kind, operationRef, targetRef, callIndex} globally per run, so one decision map applies to every trace — it cannot express "fire in trace 1 but not trace 2";
  • only engage for probabilities strictly in (0,1) (isChoiceRate), so the prob >= 1.0 collapse sits outside them entirely.

They were built to explore the probability space deterministically (swarm), not to reproduce a recorded trace.

Fidelity gap, dimension by dimension

Dimension Current source Override path today?
Which root + arrival time Rng.IntN(roots) + traffic No
Whether each call fires Rng < probability Partial (global-keyed forcedChoice)
Call multiplicity static Count No — fixed per trace
Per-span error Rng < errorRate Partial (global-keyed)
Per-span duration duration.Sample(Rng) None
Attribute values Gen.Generate(Rng) None
Parallel vs sequential static CallStyle No
Fan-out / parent-child shape topology recursion No

Duration, attributes, and multiplicity have no override mechanism at all — so exact replay is a genuinely new execution mode, not a flag on the current one.

The tension worth deciding up front

Exact replay partly defeats the purpose of importing into a topology. Motel's value is a compact, editable, generative model; a verbatim per-trace recording is large, opaque, and non-generative, and it does not compose with traffic scaling, scenarios, or fault injection (all of which mutate exactly the durations/errors/calls you'd be pinning). If the goal is "the same requests back," the raw traces may be the better artifact than a topology.

Useful asset for whichever direction wins: SpanPlan (plan.go:13-29) is already almost exactly a recorded-span record (Index, ParentIndex, Service, Operation, Kind, StartTime, EndTime, Attrs, IsError), and the importer already has the per-trace trees before aggregation.

Options

A. Fixture / passthrough replay. Preserve per-trace trees and re-emit them verbatim (new IDs/timestamps), gated by something like a top-level imported: true / mode: replay. Maximum fidelity; effectively a player rather than the simulation engine. Must explicitly define how it interacts with State (queues/circuit breakers/backpressure), scenarios/overrides, and traffic shaping — most likely by disabling them. Recording could live inline in the topology file or in a sidecar; left open here.

B. Statistical reproducibility. Make the aggregate model lossless enough that a seeded run matches the source population distribution. Cheaper, composes with everything, but never reproduces an individual request.

C. Pragmatic fidelity wins on the existing schema. The schema already has count on CallConfig (config.go:113); the importer throws multiplicity into probability instead of emitting it. Emitting modal/mean count for repeated call sites, plus per-call-site duration distributions, recovers a lot of fidelity with no new execution mode. This is the cheapest correctness win and could stand alone.

Suggested direction

Treat C as a near-term, low-risk improvement regardless of the rest. Treat A as the real "replay" feature, where the bulk of the work is recording and serialising the per-trace data the importer currently discards and adding a data-driven emitter — the imported flag is just the marker that selects it. Recording shape (inline vs sidecar) and the precedence rules against State/scenarios/traffic are open design questions for maintainers.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions