Validate motel check against published empirical trace metrics#144
Merged
Conversation
Add topologies modelling the structural characteristics reported by the Alibaba (Luo et al. TPDS 2022) and Meta (Huye et al. ATC 2023, Du et al. ICS 2025) trace studies, and a test suite verifying that motel check's static bounds and Monte Carlo distributions reproduce the published values. Document the comparison and its limitations in docs/research/empirical-validation.md. Closes #94
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #94.
Validates the
motel checkanalysis engine against independent ground truth from the Alibaba and Meta trace studies by constructing topologies that exhibit the published structural characteristics and verifyingcheckreproduces them.What's included
Example topologies (
docs/examples/):alibaba-call-graph.yaml— models Luo et al. (IEEE TPDS 2022): call graph depth 6 (published range 2–6 for top services), max children set size 10 (published 1–10), repeated call rate 16.7% (published 16.2%), memcached-dominated leaf tier. Static results: depth 6, fan-out 10, spans 24.meta-wide-fanout.yaml— models Huye et al. (USENIX ATC 2023) and Du et al. (ICS 2025): wide, shallow feed-style workflow with 50 children at the aggregation tier (published: up to 50 at Meta). Static results: depth 3, fan-out 50, spans 54. Fully deterministic, so it doubles as an exactness check — sampled distributions must be constant and equal to the static bounds.Test suite (
pkg/synth/empirical_test.go):MaxDepth/MaxFanOut/MaxSpansmatch hand-computed expectations, including the worst-case path and refs.checkdoes not report repeated calls as a metric.Documentation (
docs/research/empirical-validation.md):motel checkoutput, and reproduction steps.checkcannot report; repeated call rate is validated against the topology rather thancheckoutput; topology dynamics and population-level depth distribution shape are out of scope for a single-topology analysis (the latter is covered by the existinggenRealisticConfigproperty tests).Results
No disagreements between
checkand the modelled values were found — all static bounds match exactly, and sampled distributions behave as the model predicts.make testandmake lintpass.https://claude.ai/code/session_01XUMUcHZyxiTbC8VYGcGHyu
Generated by Claude Code