Repository navigation
fix(tracing): record zero top_k / chunk_size in update_retriever_span - #3406
Open
Lesereingrape wants to merge 1 commit into
Open
Lesereingrape wants to merge 1 commit into
Lesereingrape wants to merge 1 commit into
Conversation
`None` is the "not provided" marker in this signature, so truthiness guards swallow a legitimately reported 0 and leave the span field untouched. `update_llm_span` already documents this exact rule for its numeric fields, and `next_retriever_span` keeps 0 on its way through `_drop_none`, so the declarative and imperative paths disagreed.
|
@Lesereingrape is attempting to deploy a commit to the Confident AI Team on Vercel. A member of the Team first needs to authorize it. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
update_retriever_span(...)guardedtop_kandchunk_sizeon truthiness, so a caller that legitimately reportstop_k=0(orchunk_size=0) silently gets the value dropped instead of recorded on the span.This changes both guards to
is not None, which is what the signature already implies:Noneis the "not provided" marker, so0should be a recordable value.Why
next_retriever_span(...)already persists zeros for these fields through_drop_none, so the two entry points disagree — the same trace built through the update path loses data that the constructor path keeps.How I checked
Added
tests/test_core/test_tracing/test_update_functions/test_update_retriever_span_zero_values.py:test_zero_top_k_and_chunk_size_are_recorded— fails on currentmainwithAssertionError: assert None == 0, passes with the guard change.test_omitted_fields_are_left_untouched— pins the existing behavior that omitted (None) fields must not overwrite values already on the span.Measured on
origin/main(a200ecede) with the fix reverted vs applied:embedderkeeps its truthiness guard on purpose — an empty embedder name carries no information, unlike a zero count.Notes
deepeval/tracing/context.py.CI note
This branch has 11/13 checks green; the two red
testshards fail on tests that are already red onmain(a200ecede) with the same node id and the same message:tests/test_core/test_sqlite_store.py::TestTestRunManagerIntegration::test_storage_failure_is_a_warning_not_an_error—AssertionError: assert 'disk I/O error' in 'Warning: fa…'(therich80-column wrap of the saved-run path indeepeval/test_run/test_run.py).tests/test_metrics/test_eval_mode_metrics.py::test_hybrid_jev_takes_the_decisions[sync-MultiTurnMCPUse]—AttributeError: 'CallToolResult' object has no attribute 'structuredContent'.Both look unrelated to this diff, and #3316 / #3373 appear to already target them. Because the core shard runs with
--maxfail=1, it stops at the sqlite failure before reachingtests/test_core/test_tracing/, so the tests added here don't show up in that log; locally this whole directory is green (18 passed).