Skip to content

[SDTEST-3830] CI Visibility instrumentation telemetry: metrics + logs#279

Merged
ypopovych merged 13 commits into
mainfrom
SDTEST-3830-instrumentation-telemetry
Jun 15, 2026
Merged

[SDTEST-3830] CI Visibility instrumentation telemetry: metrics + logs#279
ypopovych merged 13 commits into
mainfrom
SDTEST-3830-instrumentation-telemetry

Conversation

@ypopovych

Copy link
Copy Markdown
Contributor

What

Brings the CI Visibility instrumentation-telemetry subsystem to a releasable state: SDK self-metrics and self-logs are collected through a custom, OpenTelemetry-free pipeline, plus the supporting tracer-lifecycle, runtime/device-info, and debug-logging fixes found while testing it.

Why

The DD telemetry intake wants distributions as raw sample arrays and logs as minimal value types — neither maps cleanly onto OTel's histogram/log SDK. Bridging through OTel was lossy (percentile distortion from bucket reconstruction) and complex. Hand-written stores export a verbatim copy and let the whole OTel metrics/logs SDK be dropped.

See Sources/DatadogSDKTesting/Telemetry/ARCHITECTURE.md for the full design.

Highlights

Telemetry metrics

  • Telemetry.metrics — a typed, discoverable tree of CI Visibility metrics over a custom MetricStore (per-interval count deltas + raw distribution samples), drained on a short timer and force-flushed on a full buffer.
  • Instruments simplified to Counter<Tags> / Distribution<Tags> with per-metric MetricTags types (dropped the parallel untyped variants and the OTel meter path).
  • Metric point timestamps sent as integer unix seconds, matching the intake.

Telemetry logs

  • Telemetry.logs.error / warn / debug over a custom LogStore, deduplicating identical logs into one entry with an occurrence count, draining on the same timer as the logs request type. Removed the OTel TelemetryLogExporter bridge.

Lifecycle & wiring

  • Telemetry owns the app-lifecycle protocol (app-started on init, heartbeat timer, app-closing on shutdown) and is created with the tracer; observers thread network/storage facts back up via neutral protocols.
  • Tracer lifecycle tied to the monitor; network instrumentation made self-contained; tracer re-creation/teardown hardened (shutdown deadlock + swizzling crash fixes).
  • Runtime/device info via RuntimeInfo; correct product-version resolution for the app/framework under test; separate app vs. library versions.

Upload & debug logging

  • Upload outcomes (success/retry/drop, with a failure description) logged from DataUploadWorker.
  • HTTPClient debug mode now logs the request body, response data, and request/response headers — bodies rendered as UTF-8 text with a <binary N bytes> fallback, and DD-API-KEY / DD-APPLICATION-KEY redacted to ****.

Testing

  • EventsExporter and DatadogSDKTesting build from source; telemetry exporter tests pass (metrics, distributions, logs, mixed batches, app-lifecycle).

🤖 Generated with Claude Code

ypopovych and others added 12 commits June 15, 2026 11:27
… test

DDTracer read Bundle.main directly, which is the bare xctest runner for
host-less test bundles, so it reported a wrong/unknown application version.

Add Bundle.productUnderTest, a resolver that picks the bundle representing
the product being tested and reports its version, with priority:
  1. DD_VERSION override (quick-exit: skips all discovery when set)
  2. Bundle.main when it is an .app (app-/UI-hosted tests)
  3. a dynamic framework named after XCODE_SCHEME_NAME, co-located with the
     .xctest bundle (sibling on macOS/simulator/device, or embedded), found
     among loaded frameworks or probed on disk (static frameworks ship in the
     products folder but are never loaded into memory)
  4. the .xctest bundle version (static/SPM-merged products)
  5. "<unknown>"

The selection policy is a pure function over value-type descriptors so it is
fully unit-tested. Adds DD_VERSION as a configuration key.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…tracer on shutdown

Replace the Mach-O LC_BUILD_VERSION compiler-version scanner (CompilerVersion.c/h)
with RuntimeInfo: an Xcode/SPM enum exposing runtimeName, runtimeVersion and
swiftVersion. It maps the XCTest bundle's DTXcode to a Swift version (Xcode 26+),
falling back to a single `swiftc --version` spawn only when needed (avoids the
subprocess on macOS for both Xcode and SPM). XCODE_SCHEME_NAME selects the case.

Thread the runtimeName/runtimeVersion through APIServiceConfig and
TestOptimizationApiService so TelemetryApplication reports them instead of
hardcoding "swift".

Make DDTestMonitor.tracer lazily recreatable: SessionManager.endSession now
shuts the tracer down (which tears down the global OpenTelemetry provider), and
DDTracer.deinit restores the default providers. The next session rebuilds a
tracer that re-registers a live SDK provider, so spans stay SpanSdk instead of
crashing the `as! SpanSdk` casts. The backing store is guarded by Synced to keep
the lazy init/reset race-free.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…refine device/runtime info

- shutdownTracer(): clear the tracer slot under the lock but run shutdown()
  outside it. shutdown() flushes/logs/uploads and stdout/stderr capture routes
  back through DDTestMonitor.tracer, so holding the non-reentrant lock during
  shutdown deadlocked.
- URLSessionInstrumentation: check URLSessionDelegate conformance via the ObjC
  runtime (InstrumentationUtils.classConforms, walking the superclass chain with
  class_conformsToProtocol) instead of a Swift `is` cast. The cast messages every
  class from objc_getClassList() and aborted on pathological internal classes
  (__NSGenericDeallocHandler, __NSAtom, __NSMessageBuilder).
- EnvironmentTests: update to the refactored PlatformUtils API
  (getPlatformVersion) and assert runtime tags against getRuntimeInfo().
- Device/Platform/RuntimeInfo refactor wiring (getDeviceInfo/getKernelInfo).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…umentation self-contained

Move the tracer off the global lazy accessor and onto the DDTestMonitor
instance: it's created in init and torn down with the monitor (stop() flushes
and shuts it down; DDTracer.deinit restores the default OpenTelemetry
providers). Removed the static `tracer` accessor entirely — callers use
`DDTestMonitor.instance?.tracer`, so no tracer is created when no monitor is
installed (this is what caused the accidental backend connection), and there's
no lock to deadlock on during shutdown.

Session-scoped span creators (DDSession/DDModule/DDSuite/DDTest) get the tracer
from SessionConfig.tracer, captured from the monitor when the session
bootstraps. Ambient callers (stdout/stderr capture, crash, XCUI, instrumentation
control) use the optional instance accessor and no-op without a monitor.

DDNetworkInstrumentation is now self-contained: injectHeaders / recordPayload /
disableNetworkCallStack / enableNetworkCallStackSymbolicated are its own
properties passed at init, and the URLSessionInstrumentationConfiguration
callbacks capture self weakly to break a retain cycle (config -> closures ->
self -> urlSessionInstrumentation -> config).

DDCrashes.install takes the tracer and returns CrashInformation? instead of
reaching into DDTestMonitor.instance; the monitor stores the result.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Restore the upload logging path: DataUploadWorker logs each batch's
success/retry/drop with a failure description (DataUploadStatus.failureDescription),
threaded through FeatureStoreAndUpload and the feature exporters. Remove the
now-redundant per-request logs in the API services and the orphaned
clientLibraryVersionHeader call. Drop the obsolete DeviceTests.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Move telemetry metrics and logs off OpenTelemetry into hand-written stores,
mirroring the existing MetricStore rationale (the DD intake wants raw samples /
minimal log values, not OTel-shaped data).

Metrics: drop the Typed* prefix — Counter<Tags>/Distribution<Tags> are now the
only instruments, each paired with a per-metric MetricTags type and a constrained
extension exposing the typed add/record. Removes the parallel untyped Counter/
Distribution/Factory and the one-off wrapper structs.

Logs: add a LogStore + telemetry.logs.error/warn/debug handler on Telemetry,
deduplicating identical logs into one entry with a count and draining on the
shared flush timer as the `logs` request type. Tags reuse the metric rendering.
Delete the now-unused OTel TelemetryLogExporter bridge and its test.

Also send DD-Client-Library-Version on the telemetry request directly, and
document both stores in ARCHITECTURE.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…mode

Capture the uncompressed request body before deflate and log it alongside the
response. Render payloads as UTF-8 text (JSON in nearly all cases), falling back
to a `<binary N bytes>` marker when the bytes aren't valid UTF-8.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…ields

Add request and response headers to the HTTPClient debug log. Capture just the
url, headers, and uncompressed body into a small LogFields value up front (only
when debug is on), so the completion handler no longer retains the full
URLRequest for the request's duration.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
TelemetryMetric.Series.Point.timestamp is now Int64 unix seconds (rounded),
matching the intake's expected format, with a `time:` convenience init that
rounds a TimeInterval. MetricStore stamps points with the rounded clock value.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Replace DD-API-KEY and DD-APPLICATION-KEY values with **** in HTTPClient debug
header logs (names sourced from HTTPHeader.Field, matched case-insensitively).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Flip debugTelemetry to false for release; backend debug processing was only
needed while validating the telemetry pipeline.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The SDK can't instrument its own test run, so report unit-test results by
uploading the JUnit reports xcbeautify produces. Give each scheme/platform a
distinct report filename so the two unit-test schemes no longer overwrite one
junit.xml, then upload build/reports via datadog-ci in the unit-tests workflow
(runs on failure too; skips gracefully when DD_API_KEY is absent, e.g. fork PRs).

Use `npx --yes @datadog/datadog-ci` instead of a global install, here and in the
codeQL SARIF upload.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@ypopovych
ypopovych force-pushed the SDTEST-3830-instrumentation-telemetry branch from c2999a9 to 4a2ed0b Compare June 15, 2026 10:27
The function-name regex used `\S+` for Swift symbols, truncating a backtick-
quoted name with spaces (e.g. `failing parameterized test`(_:)) at the first
space, so the source-line parser missed the function. Match backtick-quoted
segments as a unit, and split the module on the last dot before the first
backtick so a dot inside a quoted name isn't taken as the module separator.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@ypopovych
ypopovych merged commit a145074 into main Jun 15, 2026
11 checks passed
@ypopovych
ypopovych deleted the SDTEST-3830-instrumentation-telemetry branch June 15, 2026 12:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants