Repository navigation
Parity scoreboard: trailing-stop bucket is graded on a 100% PnL tolerance #664
Description
Your scoreboard says 7,989 probes, 0 below strong. I read the grading spec: trailing-stop scripts are graded on the production profile — exits within 0.05%, PnL within 100%.
That is a 100x wider PnL tolerance than the 1% the excellent tier uses everywhere else, and it still goes into the same 7,989 denominator. So the headline is true and also not the number a user needs. If someone asks "how close is this to TradingView on a trailing stop", the honest answer is that bucket on its own.
Do you have the split? I'd guess the trailing-stop group is a few hundred probes, and that its PnL spread is where the interesting failures live.
If it helps, I do paid passes: $29, 48 hours, no charge if I find nothing reproducible. But the bucket number is what I'd want first.
Activity
luisleo526 commentedon Oct 31, 2026
Thanks, you're right. Strategies whose
strategy.exituses atrail_*parameter were graded on a looser profile (exit price within 0.05%, P&L within 100%), and they were counted in the same 7,989 headline.We split that bucket out and re-graded it with the same grader:
trail_*(58 distinct strategies; one strategy on several symbols or timeframes counts as several probes). The other 7,648 were already graded on the strict profile (exit 0.01%, P&L 1%).So the headline was true, but the bucket you asked about is the number a user needs, and here it is.
What we will change:
We don't take paid audits, but thank you for the report.
rytjy commentedon Nov 1, 2026
Thanks — the split is the answer I asked for, and it's better than my guess was: 0 of 341 failing under the 90th-percentile rule means the trailing bucket is not where the failures live, and I assumed it would be.
Two things for the re-grade:
Name the rule on the scoreboard. "0 below strong" holds under either rule for the headline, but not for this bucket: 0/341 at the 90th percentile, 8/341 by worst trade. Whichever you adopt, print it next to the number — the two readings don't disagree by a little, they disagree about whether the tail exists.
The 2,200% max P&L is a tolerance artifact: a percentage on near-zero notional. If the re-grade flags that trade, report ticks or % of notional beside it, or the worst row in your table will be its smallest trade.
When the 7,989 rerun lands I'd read the 341-row per-probe table before the changed counts — that's the part I can check in an afternoon.
Noted on paid audits; no pitch intended.
luisleo526 commentedon Nov 1, 2026
Agreed on both points. In the re-grade:
Same date as committed: 2026-11-04.
Grade trailing-stop strategies on the strict profile (#678)
rytjy commentedon Nov 1, 2026
One thing I can hand you on 11-04.
When the rerun lands, post or link the 341-row per-probe table and I'll recompute it under whichever rule you settle on — 90th percentile and worst-trade side by side, ticks and % of notional on the flagged rows — and give it back in your columns, not mine. Free, no pitch, nothing asked in return.
Why it's worth your time: those 341 rows are the only part of the grade where the arithmetic can be checked end to end, and they're the part most likely to move when the rule changes. If my numbers disagree with yours, we'll see which row and why.
If it helps, the same three checks run offline on a plain CSV:
python3 -m auditkit.cli trades.csv --trials N(look-ahead, multiple-testing, out-of-sample decay; stdlib only). Not a pitch — it just means you don't have to trust my arithmetic, you can run it.If you'd rather keep it in-house, ignore this.
luisleo526 commentedon Nov 1, 2026
Correction to our 10-31 numbers
On 10-31 we said that 8 of the 341 probes had a trade over the limit and that the largest P&L delta was 2,200%. Those two figures forgave a trade's P&L difference only when the trade's exit price was within 0.01% of TradingView's, although our comment stated 0.02%; 0.02% is the limit the grader applies to strategies with trail parameters, and it is the one used in the re-run below. With it, 7 of the 341 fail on their worst trade and 332 pass; 2 probes have no P&L observation, so they are counted as neither. The largest worst-trade P&L delta is 183.33%. The count moves from 8 to 7 because of one trade in one probe (its exit differs by 0.0128%); the 2,200% was one trade whose exit differs by 0.0187%, and its P&L delta is 0 at the 0.02% limit. Everything below uses the grader's deltas at 0.02%.
What was re-run
The 341 probes (58 distinct strategies) in the validation population whose
strategy.exituses atrail_points,trail_offsetortrail_priceparameter were graded again, in our internal re-run (the sweep), by the grader as merged in #678 (tree of the PR heade795bcd9864a3bb25d860ab577e730f6f81b362c,scripts/verify_corpus.pysha25643e088e8d7530be143def61cde1c31222761f0a05cd1acc4af4eda3f27ff892d), next to the grader of v1.9.1 (fc0a2df5deca07f8a7e59995a17e71203fb19f34, same file sha2561d2d8dbe584ae7902a32e096f2a8cd744d03d366fe39f9d16c9178e93a98e94e) on the same recorded engine trades and TradingView trades. The new grader's trail_* profile: exit limit 0.02%, P&L limit 1%, count and entry limits unchanged. The merged change is maina3e7822e.Results (341 of 341 probes read, 0 exceptions)
excellent). Across the full 7,989-probe re-grade of the same change: 0 tier changes, 0 control differences.In every row the percentage of notional is at most 0.26%: the large P&L percentages come from trades whose own P&L is small, not from large price gaps. Ticks are not comparable between rows (each row's instrument has its own minimum price step).
What this does not show
The sweep's verifier documents do not record the profile or the limits they used, so "the new limits were applied" is shown from the recorded code and route, not from the documents: the sweep's engine code bundle contains
scripts/verify_corpus.pywith the same git blob (796753fc9d783931e5ca3b24a096e3629ac48070) and sha256 as the PR head, which sets the trail_* limits to 0.02% (exit) and 1% (P&L) (PRODUCTION_EXIT_DELTA,PRODUCTION_PNL_DELTA), and the grading harness loads the verifier from that checkout. One worst-trade delta (97.36%, within a 100% limit and outside a 1% limit) is a per-trade witness, and the same probe still passes Rule A because its 90th-percentile delta is 0%.rytjy commentedon Nov 1, 2026
Thanks — the 2,200% → 183.33% line is the one I care about, and your own % of notional column shows why: 0.0204% on a trade whose P&L is tiny. That is the tolerance artifact gone, not argued away.
One thing worth stating on the scoreboard: 8 → 7 and 2,200% → 183.33% both come from the limit change (0.01% → 0.02%/1%), not from a re-grade of the same data. Readers will otherwise assume the numbers moved because the measurement improved.
Rule A 0/341 next to Rule B 7/341, plus the 2 with no observation and the 39 without tick steps, is the honest shape. Naming what cannot be computed is better than dropping it.
On "what this does not show": I read that as "the limits are established from the code blob, not from a recorded run". Fine as long as it stays in the text. If the per-probe table ever becomes shareable, the offer stands — I will recompute the 341 rows under both rules and return the same columns. If not, no loss.
verify_corpus: read the TradingView percent column as "Return %"; a m…
luisleo526 commentedon Nov 4, 2026
As promised, here is the per-probe table behind the figures above: one row per probe, 341 rows numbered 1 to 341, with no strategy, symbol, price or trade-value column. Each number sits next to the rule it was graded by.
A correction to the count in our earlier comment: ticks are available for 300 of the 341 probes, not 302. 39 probes have no recorded minimum price step (the two ETH datasets), and 2 more (rows 120 and 174) have no eligible worst-trade P&L observation, so their tick values are reported as missing, not zero. % of notional is available for 339 probes (the same 2 rows have no observation). None of the seven Rule B failures is among the missing rows.
#664: per-probe results of the trailing-stop grading re-run (341 rows)
One row per probe, in a random order;
rowis only a running number. The file carries no strategy or instrument names, symbols, dataset names, prices, trade values or hashes, so a row cannot be traced back to a probe from this file.Grader: the grader as merged in #678, trail_* profile: exit limit 0.02%, P&L limit 1% (limits are the same for every row).
Columns:
tier_before,tier_after: the grade tier with the grader of v1.9.1 and with the grader of Grade trailing-stop strategies on the strict profile #678 (all 341 unchanged).rule_a_result: the grader's own rule: PASS when the 90th percentile of the probe's exit deltas and of its P&L deltas are each below the limits.rule_b_result: the stricter reading: PASS when the probe's worst single trade is below the same limits on both;no observationwhen the probe has no eligible worst-trade P&L observation (not counted as pass or fail, never 0).worst_trade_exit_delta_pct,worst_trade_pnl_delta_pct: the worst single trade's exit and P&L deltas, in percent, as the grader measures them.worst_pnl_trade_gap_pct_of_notional: for the trade with the largest P&L delta, |engine P&L - TradingView P&L| as a percentage of that trade's TradingView entry notional.no observationwhere the probe has no eligible worst-trade P&L observation (2 rows).worst_pnl_trade_exit_gap_ticks: for the same trade, |TradingView exit - engine exit| in ticks of the instrument's minimum price step, fractional ticks kept. Ticks are not comparable between rows (each row's instrument has its own step).no observationwhere the minimum price step is not recorded or where the probe has no eligible worst-trade P&L observation (2 rows).Totals: Rule A 0 FAIL / 341 PASS; Rule B 7 FAIL / 332 PASS / 2 no observation; tier changes 0 of 341.
The table (CSV, 341 rows)
rytjy commentedon Nov 4, 2026
Recomputed your 341 rows before replying. I parsed the table as you posted it and re-derived the aggregates with my own script — none of your counts reused.
What reproduces exactly:
tier_before=tier_afteron all 341.rule_a_result: 341 PASS, 0 FAIL.rule_b_result: 332 PASS, 7 FAIL, 2no observation— rows 120 and 174, as you said.One check you did not run, which the file allows: no PASS row breaches either limit, and no FAIL row sits inside them. So the labels agree with the numbers standing next to them, in the same table, checkable by anyone holding it. That check passes.
One number in the table that is not in your text: the seven failures'
worst_pnl_trade_gap_pct_of_notionalare 0.0204, 0.0210, 0.0351, 0.0826, 0.1105, 0.1721, 0.2612. So the entire distance between the two readings is bounded at 0.26% of notional — on one trade, in the worst of 341 probes. The largestworst_trade_exit_delta_pctis 0.2604%, the same row.Row 232 deserves naming, because it is where the two readings meet: exit delta 0.02038% against a 0.02% limit, over by four ten-thousandths of a point — and the same trade behind the 183.33% P&L delta. Both sentences are true at once: as a ratio of that trade's own P&L it is 183%, as a share of notional it is 0.0204%. Your table now carries both columns, which is what I asked for.
What I still cannot check, and will not dress up: that these numbers came out of the merged grader on your data. The table is consistent with your description; it is not evidence of the run, and the sweep documentation still records no profile or limits. That is the gap you flagged yourself, and it stays flagged.
What I owe from 11-01 was a recomputation handed back to you. This is it, on the artifact you published: zero discrepancies, nothing to correct.
thanks — Carl
luisleo526 commentedon Nov 4, 2026
Revised grading:
trail_*probes use exit 0.02% and P&L 1% limits, graded per strategy at p90.Profile limits (count / entry / exit / P&L): strict
1.0% / 0.01% / 0.01% / 1.0%; trail / production1.0% / 0.01% / 0.02% / 1.0%.Both profiles require exact gated trade-count parity for an
excellentgrade, despite the declared 1.0% count limit.The entry, exit and P&L gates are graded per strategy at the 90th percentile: each strategy's p90 delta must be strictly below its limit; exit and P&L are not gated trade by trade.
p90 is linearly interpolated over the sorted deltas computed by the verifier.
The sweep selects each probe's profile automatically from its strategy source with
detect_parity_profile: a probe whose source assignstrail_points,trail_offsetortrail_price(outside comments) is graded on the production profile; every other probe on the strict profile.The sweep uses production for 341 probes (58 distinct strategies) and strict for 7,648.
The sweep's re-run of all 7,989 probes under the revised trail profile, with the grader change merged in #678, produced 0 of 7,989 grade changes; the 341 trail probes listed in the issue's table are the only probes this profile touches.
The profiles and sweep limits are recorded in docs/coverage.md, section “Grading profiles and sweep limits”, in pull request #732 (the link to the main branch resolves once it is merged).
Thanks, Carl, for recomputing the 341-row table independently; it made this record better.
luisleo526 commentedon Nov 5, 2026
The grading profiles and the sweep limits behind this scoreboard bucket are now documented in docs/coverage.md (#732), as described in the comment above. Closing; please reopen if the documented profile does not answer the question.