fix(internal/civisibility): fix known-tests API pagination by anmarchenko · Pull Request #4824 · DataDog/dd-trace-go

anmarchenko · 2026-05-29T11:23:00Z

What does this PR do?

Moves known-tests pagination metadata into the JSON API attributes payload for both requests and responses.

The known-tests client now sends data.attributes.page_info on the initial request and uses data.attributes.page_info.page_state for follow-up requests. It also reads cursor and has_next from data.attributes.page_info in backend responses, so it continues paginating after page 1.

The regression test now asserts that page_info is not emitted at the top level, verifies the attributes-level request shape, and exercises cursor-based follow-up requests without sending a page size.

Motivation

https://github.com/DataDog/shepherd/issues/46

Go was modeling page_info as a sibling of data, while the known-tests API contract used by the backend and other tracers places pagination under data.attributes. With a contract-correct backend response, Go could parse the first page of tests but missed has_next and stopped without requesting subsequent pages.

Testing

go test ./internal/civisibility/utils/net
go test ./internal/civisibility/...
git diff --check -- internal/civisibility/utils/net/known_tests_api.go internal/civisibility/utils/net/known_tests_api_test.go

codecov · 2026-05-29T11:27:03Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 62.56%. Comparing base (8b52c4a) to head (b7356c7).

Additional details and impacted files

Files with missing lines	Coverage Δ
internal/civisibility/utils/net/known_tests_api.go	`83.14% <100.00%> (-6.75%)`	⬇️

... and 282 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

datadog-datadog-prod-us1-2 · 2026-05-29T11:27:37Z

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 61.90% (-0.08%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: b7356c7 | Docs | Datadog PR Page | Give us feedback!}

pr-commenter · 2026-05-29T11:47:12Z

Benchmarks

Benchmark execution time: 2026-06-01 08:44:05

Comparing candidate commit b7356c7 in PR branch anmarchenko-fix-known-tests-pagination with baseline commit 8b52c4a in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 297 metrics, 2 unstable metrics, 1 flaky benchmarks without significant changes.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

🟩 = significantly better candidate vs. baseline
🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Known flaky benchmarks

These benchmarks are marked as flaky and will not trigger a failure. Modify FLAKY_BENCHMARKS_REGEX to control which benchmarks are marked as flaky.

Known flaky benchmarks without significant changes:

scenario:BenchmarkOTLPTraceWriterFlush

anmarchenko · 2026-05-29T12:09:27Z

E2E Test Report: SUCCESS

Tested by: Shepherd Agent (autonomous QA for Datadog Test Optimization)

Test Environment

Method: Local Shepherd E2E with gin, Go testing, orchestrion instrumentation, and mockdog-backed CI Visibility responses
PR: fix(internal/civisibility): fix known-tests API pagination #4824
Revision tested: 75ca8cd75530a6db54947b2e8105dd4195acf581
dd-trace-go dependency: anmarchenko-fix-known-tests-pagination

Results

Check	Status	Evidence
Single-page known-tests response	Passed	Run `20260529-140335`: known-tests requests used `data.attributes.page_info`, omitted top-level `page_info`, omitted `page_size`, and the selected known test passed with `0` new-test spans and `0` retries.
Multi-page known-tests response	Passed	Run `20260529-140529`: requests were split into first page plus `page-2`; follow-up cursor was sent as `data.attributes.page_info.page_state = "page-2"`; no top-level `page_info`, no `page_size`; the page-2 known test passed with `0` new-test spans and `0` retries.
First page succeeds, second page errors	Passed	Run `20260529-140605`: first-page requests succeeded, `page-2` returned 500 and was retried; the executed test was present only on page 2, and still produced `0` new-test spans and `0` retries, confirming partial page-1 known-tests data was not used after the page-2 failure.

Request Shape Verification

Mockdog request recordings confirmed:

page_info is present under data.attributes.
No top-level page_info is sent.
No client page_size is sent.
Follow-up requests use data.attributes.page_info.page_state = "page-2".

Span Verification

The executed gin test span for the multi-page and second-page-error runs had no test.is_new, no test.is_retry, and no test.retry_reason tags. This is the expected behavior for a known test, and in the error case confirms the client did not treat partial first-page data as usable known-tests data.

Issues Found

No dd-trace-go regressions found in this pass. One local command invocation initially failed because a shell regex alternation was not quoted; it was rerun with a shell-safe single-test regex and passed.

Datadog UI Verification

Not applicable for this request: the behavior under test is dd-trace-go's known-tests client pagination contract and error handling using mockdog as the CI Visibility backend.

Test Methodology

Added local Go mockdog scenarios matching real gin Go test identities.
Ran Shepherd/crook against gin with --dep dd-trace-go=anmarchenko-fix-known-tests-pagination and --debug.
Exercised three cases: single page, multiple pages, and first-page-success/second-page-error.
Inspected mockdog known_tests.jsonl request recordings for pagination shape.
Queried mockdog test spans to verify known-test behavior and absence of EFD retry/new-test tags.

This E2E test was performed by Shepherd - autonomous QA agent for Datadog Test Optimization.

tonyredondo · 2026-06-01T08:52:47Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-06-01T08:52:51Z

View all feedbacks in Devflow UI.

2026-06-01 08:52:51 UTC ℹ️ Start processing command /merge

2026-06-01 08:53:00 UTC ℹ️ MergeQueue: waiting for PR to be ready

This pull request is not mergeable according to GitHub. Common reasons include pending required checks, missing approvals, or merge conflicts — but it could also be blocked by other repository rules or settings.
It will be added to the queue as soon as checks pass and/or get approvals. View in MergeQueue UI.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.

2026-06-01 08:57:19 UTC ℹ️ MergeQueue: merge request added to the queue

The expected merge time in main is approximately 16m (p90).

2026-06-01 09:10:54 UTC ℹ️ MergeQueue: This merge request was merged

### What does this PR do? Moves known-tests pagination metadata into the JSON API attributes payload for both requests and responses. The known-tests client now sends `data.attributes.page_info` on the initial request and uses `data.attributes.page_info.page_state` for follow-up requests. It also reads `cursor` and `has_next` from `data.attributes.page_info` in backend responses, so it continues paginating after page 1. The regression test now asserts that `page_info` is not emitted at the top level, verifies the attributes-level request shape, and exercises cursor-based follow-up requests without sending a page size. ### Motivation DataDog/shepherd#46 Go was modeling `page_info` as a sibling of `data`, while the known-tests API contract used by the backend and other tracers places pagination under `data.attributes`. With a contract-correct backend response, Go could parse the first page of tests but missed `has_next` and stopped without requesting subsequent pages. ### Testing - [x] `go test ./internal/civisibility/utils/net` - [x] `go test ./internal/civisibility/...` - [x] `git diff --check -- internal/civisibility/utils/net/known_tests_api.go internal/civisibility/utils/net/known_tests_api_test.go` Co-authored-by: tonyredondo <[email protected]>

fix(internal/civisibility): move known-tests pagination into attributes

75ca8cd

anmarchenko requested a review from a team as a code owner May 29, 2026 11:23

Merge branch 'main' into anmarchenko-fix-known-tests-pagination

72e3a9e

anmarchenko changed the title ~~fix(internal/civisibility): fix known-tests pagination shape~~ fix(internal/civisibility): fix known-tests API pagination May 29, 2026

tonyredondo approved these changes Jun 1, 2026

View reviewed changes

Merge branch 'main' into anmarchenko-fix-known-tests-pagination

b7356c7

gh-worker-dd-devflow-36fce6 Bot added mergequeue-status: waiting mergequeue-status: queued mergequeue-status: in_progress and removed mergequeue-status: waiting mergequeue-status: queued labels Jun 1, 2026

gh-worker-dd-mergequeue-cf854d Bot merged commit 2e14100 into main Jun 1, 2026
254 of 258 checks passed

gh-worker-dd-devflow-36fce6 Bot removed the mergequeue-status: in_progress label Jun 1, 2026

gh-worker-dd-mergequeue-cf854d Bot deleted the anmarchenko-fix-known-tests-pagination branch June 1, 2026 09:10

gh-worker-dd-devflow-36fce6 Bot added the mergequeue-status: done label Jun 1, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(internal/civisibility): fix known-tests API pagination#4824

fix(internal/civisibility): fix known-tests API pagination#4824
gh-worker-dd-mergequeue-cf854d[bot] merged 3 commits into
mainfrom
anmarchenko-fix-known-tests-pagination

anmarchenko commented May 29, 2026

Uh oh!

codecov Bot commented May 29, 2026 •

edited

Loading

Uh oh!

datadog-datadog-prod-us1-2 Bot commented May 29, 2026 •

edited by datadog-official Bot

Loading

Uh oh!

pr-commenter Bot commented May 29, 2026 •

edited

Loading

Explanation

More details about the CI and significant changes

Known flaky benchmarks

Uh oh!

anmarchenko commented May 29, 2026

Uh oh!

tonyredondo commented Jun 1, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Jun 1, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

anmarchenko commented May 29, 2026

What does this PR do?

Motivation

Testing

Uh oh!

codecov Bot commented May 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

datadog-datadog-prod-us1-2 Bot commented May 29, 2026 • edited by datadog-official Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pr-commenter Bot commented May 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Explanation

More details about the CI and significant changes

Known flaky benchmarks

Uh oh!

anmarchenko commented May 29, 2026

E2E Test Report: SUCCESS

Test Environment

Results

Request Shape Verification

Span Verification

Issues Found

Datadog UI Verification

Test Methodology

Uh oh!

tonyredondo commented Jun 1, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov Bot commented May 29, 2026 •

edited

Loading

datadog-datadog-prod-us1-2 Bot commented May 29, 2026 •

edited by datadog-official Bot

Loading

pr-commenter Bot commented May 29, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented Jun 1, 2026 •

edited

Loading