Repository navigation
traj: recursive tail and cat return the whole tree, in order - #110
Merged
nickjalbert merged 1 commit intoSep 5, 2026
Merged
Conversation
`traj tail -r` and `traj cat -r` are the only readers in traj that sort steps, and the key comes from to_epoch in _sorted_tree_steps. Three things were wrong with it, all of them silent. The catch-all branch — taken for any stamp that is not "<datetime><numeric offset>" — kept the whole string as .dt, and the next line appends "Z" to that. Every stamp traj writes is UTC and already ends in "Z" (_iso_ts), so fromdateiso8601 was handed "2026-09-04T14:13:18ZZ" and errored on every step. jq's stderr is discarded and its status is `|| true`, so both commands printed nothing at all and still exited 0. Only steps still carrying the legacy "+hh:mm" spelling survived, so a tree containing one came back as just that one step. The strip also takes the RFC-3339 lowercase "z", and trailing whitespace after it, which previously reached the parser malformed. The key also floored to the second, because the fraction is stripped before parsing. A trajectory writes many steps a second, so most keys tie and sort(1) falls back to comparing whole lines — the JSON text. Fixing only the "Z" would have traded an empty result for a plausible but wrong one: on a real tree `tail -r -n 2` returned the third- and fifth-newest steps, and GNU and BSD sort disagreed about which. The key is now integer milliseconds, folded into the integer rather than left as a decimal so the numeric sort cannot depend on the locale decimal point. Finally, an unparseable stamp took its step out of the tree rather than sorting it somewhere. Every other date read in the tree already guards that (bin/chat, thinkers/_lib/common.sh, bin/thinkers, thinkers/responder/step); this was the one call site that did not, which also left the function's own `(.ts // .timestamp // "0")` fallback dead. search -r and check -r walk the same files without to_epoch and were never affected, so the halves of the same flag disagreed. to_epoch has no other caller, and _sorted_tree_steps is reached only from the -r branches of cmd_tail and cmd_cat, so nothing outside those two paths changes. Every stamp shape that produced a value before produces the same relative order now. tests/test_traj_recursive_tree.sh has two parts. The first runs the real commands and leaves their stamps alone — six appends at machine speed, the case a whole-second key gets wrong — and checks every step on disk comes back, and comes back non-decreasing. The second builds a tree two levels deep with two branches out of the root, and rewrites the stamps to defeat three wrong sort keys at once: every step but one lands inside a single second, their alphabetical order differs from their real one, and one sits in 1970 so its key is shorter than the rest. It pins the interleaving, both levels and both branches of the recursion, both signs and the minutes of the legacy offset, the third millisecond digit, --filter, the per-file _traj_hex tag, `tail -r -n` bounding the merged stream rather than one file, an unparseable stamp keeping its step, the formatted view a terminal gets, and the non-recursive view staying as it was.
Contributor
|
LGTM thanks for the PR! Ubuntu smoke test CI job is being flakey (seemingly unrelated to this code) so merging. 🚢 |
This was referenced Sep 5, 2026
Contributor
Author
|
I did a little digging, and I think this is the reason, cheers. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
traj tail -randtraj cat -rreturn nothing — on any trajectory whose stepscarry the stamps
trajitself writes — and still exit 0.-r, --recursive Include steps from all sub-trajectories, interleavedisoffered on
tail,cat,searchandcheck.search -randcheck -rwalkthe tree fine; the two views that interleave are the ones that come back empty,
so the halves of the same flag disagree.
Why
_sorted_tree_stepsis the only place intrajthat sorts steps, and its keycomes from
to_epoch. Three things were wrong with it, all silent.1. The
Zthat gets added twice.capture()matches<datetime><numeric offset>(colon optional); anything else falls through tothe catch-all, which keeps the whole string as
.dt— and the next lineappends a
"Z"to it:Every stamp
trajwrites is UTC and already ends inZ(_iso_ts):jq's stderr goes to
/dev/nulland its status is|| true, so nothingsurfaces:
sortgets an empty stream and the command exits 0. The only stepsthat survive are ones still carrying the legacy numeric-offset spelling, which
capture()does match — so a tree containing one such step returns exactlythat one step and nothing else, which is what makes this confusing to hit. The
strip also takes the RFC-3339 lowercase
z, which reached the parser malformedbefore.
2. The key floored to the second. The fraction is stripped before parsing,
and a trajectory writes many steps a second, so most keys tie and
sort(1)falls back to its last-resort comparison of whole lines — the JSON text.
Fixing only the
Ztrades an empty result for a plausible but wrong one. On atree built with real
traj appendcalls, with just theZfixed:GNU and BSD
sortalso disagree about which wrong answer you get. So the keyis now integer milliseconds — folded into the integer rather than left as a
decimal, so the numeric sort cannot depend on the locale decimal point.
3. An unparseable stamp deleted its step instead of sorting it somewhere.
Every other date read in the tree already guards this —
bin/chat:487,554,thinkers/_lib/common.sh:365,bin/thinkers:1033,thinkers/responder/step:136,228all usetry fromdateiso8601 catch …orfromdateiso8601?.bin/traj:782was the one call site that did not, whichalso left the function's own
(.ts // .timestamp // "0")fallback dead —"0"became
"0Z"and errored.Long-standing rather than a regression: (1) reproduces at
4ee873e~1as wellas at
main.Why this is one change and not three
They are one defect —
-rdoes not deliver what its help line promises — andthe parts are not separable in a useful way. Landing only (1) would replace a
visibly empty result with a plausible, silently mis-ordered one, which is
harder to notice and harder to debug; the red/green table below shows that
state failing 8 of 15 assertions. Landing (1) and (2) without (3) leaves the
same silent-drop class the PR exists to close, one line away in the same
function. Happy to split it if you would rather review them separately.
Not in this change
to_epochstill does not accept a bare epoch-secondsts, whichbin/traj'sdisplay code does handle at lines 910 and 994. I left it alone because nothing
writes that shape — no code in
bin/,thinkers/,tools/,web/,slack/or
telegram/emits a numericts, and every stamp intests/fixtures/andtests/golden/is a quoted ISO string. Such a step is no longer droppedeither way: with the guard it sorts to the front instead of vanishing.
After
to_epochhas no other caller, and_sorted_tree_stepsis reached only fromthe
-rbranches ofcmd_tailandcmd_cat, so nothing outside those twopaths changes. I put every timestamp shape in the tree through the old and the
new key: every input that previously produced a value keeps its exact relative
order (new = old × 1000 + ms). The only behaviour that changes is for the
Zand lowercase-
zforms and the unparseable ones, which previously producednothing at all. The
+0800stamps intests/fixtures/take thecapture()branch and are untouched.
Worth saying plainly: nothing in the repo passes
-rtotraj— everyinternal caller uses the single-file forms — so this fixes a documented,
advertised flag rather than an automated path. It changes what a person at a
terminal, or a model reading a sub-run, sees.
Tests
tests/test_traj_recursive_tree.shhas two parts.Part A runs the real commands and leaves their stamps alone: six appends at
machine speed, which is the case a whole-second key gets wrong. It checks every
step on disk comes back, and comes back non-decreasing.
Part B builds a tree two levels deep with two branches out of the root, and
rewrites the stamps to defeat three wrong sort keys at once — every step but
one lands inside a single second, their alphabetical order differs from their
real one, and one sits in 1970 so its key is shorter than the rest. It pins the
interleaving, both levels and both branches of the recursion, both signs and
the minutes of the legacy offset, the third millisecond digit,
--filter, theper-file
_traj_hextag,tail -r -nbounding the merged stream rather thanone file, an unparseable stamp keeping its step, the formatted view a terminal
actually gets, and the non-recursive view staying as it was.
Red/green, by state of
bin/traj:bin/trajmainZfix onlyZ+catch, no millisecondsZ+ milliseconds, nocatchI also mutation-tested it: 20 deliberate breakages of
bin/traj— wrong sign,dropped offset minutes, truncated milliseconds, lexical instead of numeric
sort, constant
_traj_hex, recursion stopped at depth 1, only the first forkper file followed, a per-file step cap,
--filterignored, and others — andall 20 are caught.
Checked locally:
tests/run-all.shgreen apart from one pre-existing failureunrelated to this change (
test_persona_chat_exit.shpinsPATHto/usr/bin:/bin, which has nojqon a Homebrew Mac, soidentity newexits127 — it fails the same way on
mainhere, and is green in CI).shellcheck -S warningclean over the CI file set, and the new test is cleanat
-S infotoo. Parses and runs under Apple's bash 3.2; runs green under apure BSD userland and under GNU coreutils, which now agree on the order; stable
across timezones, across
LC_ALL=C/en_US.UTF-8/de_DE.UTF-8, and over tenconsecutive runs. The bash-4-only construct grep is clean.
cloc bin/ thinkers/goes 9,912 -> 9,914 code lines against a limit of 11,000.pytestgreen for web, slack and telegram.