chore: extract shared e2e telemetry checker#2369
Conversation
ava-silver
added
the
serverless
ava-silver
added
the
chores
Graphite App
8cbadb2 to
83da7dd
Compare
| console.log(`[traffic] attempt ${attempt}/${attempts} returned ${response.status}`) | ||
| if (response.ok) { | ||
| successfulRequests++ | ||
| if (successfulRequests >= requiredSuccesses) { |
There was a problem hiding this comment.
Do these kinds of services not have a clean transition from not ready -> ready where we need to get multiple successful responses?
There was a problem hiding this comment.
Right -- there's no clean client-observable ready transition. During cold start these apps interleave timeouts/403s/503s with the occasional 200, so a single success can be a fluke.
But the bigger reason for requiring multiple successes is telemetry generation, not just readiness. The trace/log pipeline (the serverless-init sidecar on Linux, the site-extension on Windows) warms up a beat after the app starts serving HTTP, so the earliest requests' spans get dropped. Empirically on the AAS suite: 3 successes produced telemetry the checker never found in 5 min, while ~10 successes over ~100s landed spans+logs within seconds -- which is why the AAS tests pass {requiredSuccesses: 10}.
So the loop does double duty: confirm the app is reliably up and drive enough sustained load that telemetry actually flows before we assert. I've tightened the comment to call that out.
…elemetry, not just readiness
### What and why? Adds real AWS Lambda e2e coverage for `datadog-ci lambda instrument` and `datadog-ci lambda uninstrument`. The tests are gated in CI, target `eu-central-1`, and use the `nodejs24.x` runtime. Stacked on #2369 (shared telemetry checker). ### How? - Adds a Lambda e2e test that creates a temporary `one-e2e-ci-lambda-*` function with the AWS CLI, instruments it, verifies the AWS configuration, invokes it and asserts telemetry flows, re-instruments and asserts idempotency (exactly one of each Datadog layer), uninstruments, verifies cleanup, and deletes it. - Adds a Lambda verifier for the Datadog Node/Extension layers, handler restoration, Datadog env vars, and the `dd_sls_ci` tag. - Asserts telemetry **identity** (service + env + version) via the shared telemetry checker rather than mere existence; the unique per-run function name doubles as the run-id marker. - Updates GitHub Actions with a Lambda path filter, conditional AWS OIDC auth, Lambda e2e env vars, and Windows/standalone skips. - Allows quoted values in `e2e/.env.local`, matching the local e2e docs. - Adds CODEOWNERS coverage for the new Lambda e2e files. ### Test plan - See test run in GitHub Actions: https://github.com/DataDog/datadog-ci/actions/runs/27034018380/job/79794038275 ### Review checklist - [x] Feature or bugfix MUST have appropriate tests (unit, integration)
3 participants
What and why?
The Lambda, Cloud Run, and Container App e2e suites all need to poll Datadog for spans and logs filtered by a run-specific identity. That polling logic was being copied per suite. This extracts it into one runner-agnostic helper so every suite shares a single implementation.
Base of the serverless e2e stack (Lambda → sidecar idempotency → Container App → AAS build on top of this).
How?
e2e/helpers/telemetry-checker.ts: polls spans and logs (15s × 20 attempts) for aTelemetryIdentity(serviceplus optionalenv/version/tags), building the span (@-prefixed attributes) and log query forms from the same identity.e2e/helpers/cloud-run-telemetry-checker.tsto delegate to the shared checker, keeping its existingserviceNamesignature.e2e/helpers/http.tstriggerTraffic: polls an HTTP endpoint until it serves successful responses, so a freshly deployed app has time to cold-start before we assert telemetry. Adopt it in the Cloud Run suite (replacing a single un-retriedfetch); Container App and AAS use it too.Test plan
Review checklist