Added code formatting style for clang-format

Added code formatting style for clang-format

Use agent logging module

Add benchmarks on open

* Add partial flags when tailing k8s container logs

* parser interface compliance

* Adjust UT

* [logs] Extract parsing from lineHandler logic

* [logs] Adjust UT, minor fixes

* [logs] UT covering previous changes

* [logs] fix async test

* Docker parser handles partial itself

* Rebase

* Address reviews

* Address reviews

* [logs] rebase collateral fixes

Co-authored-by: Jaime Fullaondo <[email protected]>

…ll detectors

PR 49939 had 30 iters, all detector-targeted (bocpd/scanmw/scanwelch
tweaks). The proposer's diversity guideline ("at least 3 distinct
families") doesn't enforce CATEGORY diversity — three families of
detector tweaks still satisfies "diverse families" while staying in
the same structural surface.

Adds a runtime check: if every candidate in the last 10 had only
detector-style target_components (no name containing 'correlator' or
'extractor'), inject a STRUCTURAL DIVERSITY REQUIRED clause demanding
at least one correlator candidate this round. Auto-disables once
correlator candidates appear in recent history.

Cooperates with operator steering (#1+#2) — explicit user directives
still take precedence; this is the autonomous backup when the operator
isn't watching.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

### What does this PR do?

Started as a bug fix and grew into a redesign of the operator-side allow-list contract for rshell, plus a new per-environment path-list feature. Three themes:

#### 1. Bug fixes (original scope)

Fixes three bugs in the intersection layer for `private_action_runner.restricted_shell.{allowed_commands,allowed_paths}`:

- **YAML `[]` was treated as "operator unset."** `GetStringSlice` returns nil for a YAML empty sequence, indistinguishable from "key absent." The kill-switch row of the truth table didn't work.
- **Path intersection used plain string equality.** An operator entry narrower than a backend entry (e.g. `/host/var/log/par-probe.txt` against backend `/host/var/log`) was dropped instead of admitted.
- **Bare operator command entries silently produced empty intersections.** Backend ships `rshell:<name>`; operator-written `cat` never matched. Now warned at startup pointing the operator at the corrected form.

#### 2. Contract redesign — sentinel-default, always-intersect

Replaces the three-way `nil` / `[]` / `[X]` slice contract with a uniform "always intersect" contract using sentinel defaults:

- `allowed_paths` default is `["/"]`. `pathContains("/", X)` is true for any absolute path, so the intersection passes the backend list through when the operator hasn't narrowed.
- `allowed_commands` default is `["rshell:*"]`. The wildcard token is a special-case in the operator-side intersection: when present, every backend entry in the `rshell:` namespace is admitted (scoped via `onlyRshellPrefixedCommands`).

The `IsConfigured` gate and the handler's nil-pass-through bypass are gone. End-user behavior is preserved: unset operator config gets the backend list as-is on both axes, explicit empty list is the kill-switch, explicit non-empty narrows.

#### 3. Per-environment paths (new feature)

`RunCommandInputs.AllowedPaths` is now `map[string][]string` keyed by environment (`default` / `containerized`). `wf-actions-server` ships one task with both keys; the runner picks the relevant slice based on `env.IsContainerized()`, then the slice flows through the same intersection logic. This lets a single Balto rule cover both host-installed agents (which see `/var/log`) and containerized agents (which see `/host/var/log`).

> **⚠️ Wire-format change**: `allowedPaths` is no longer a slice. The runner side ships in this PR; Balto / `wf-actions-server` need to ship the new shape concurrently. Coordinate the rollout — old tasks (slice shape) will fail to deserialize on the new runner.

### Implementation notes

- Pure path utilities live in `helper.go`: `cleanPathList`, `reducePathListToBroadest`, `intersectPathLists`, `commonPath`, `onlyRshellPrefixedCommands`, `backendPathsForEnv`.
- Handler methods (`filterAllowedCommands`, `filterAllowedPaths`) and `Run` stay in `run_command.go`.
- Two advisory warnings at config load surface misconfigurations: backslash entries (forward-slash contract) and non-directory entries (rshell's `os.Root` sandbox is directory-only).

### Motivation

Discovered during end-to-end PAR validation against a prod-connected test drive. Findings and reproduction traces: [experimental/.../rshell-permission-test.md](https://github.com/DataDog/experimental/blob/main/users/jules.macret/q_branch/agent_mcp/k8s_setup/rshell-permission-test.md). Parent PRs:

- #49536 (operator-side intersection)
- DataDog/dd-source#414468 (backend injects allowedPaths into every task)

Confluence: [Rshell permission model (allow lists)](https://datadoghq.atlassian.net/wiki/spaces/agent/pages/6608848267/Rshell+permission+model+allow+lists).

### Describe how you validated your changes

- **Unit tests.** 118 tests pass across `pkg/privateactionrunner/...` and `pkg/config/setup`. Linter clean (`dda inv linter.go` → 0 issues).
  - Helper tests in `helper_test.go` cover `commonPath`, `cleanPathList`, `reducePathListToBroadest` (including idempotence + order-independence properties), `intersectPathLists`, `onlyRshellPrefixedCommands`, and `backendPathsForEnv`.
  - Method tests in `run_command_test.go` pin `filterAllowedCommands` and `filterAllowedPaths` matrices.
  - YAML-backed transform tests exercise the real config-load path (the original bug #1 escaped because earlier tests used `SetWithoutSource`, which doesn't reproduce the nil-from-YAML behavior).
- **End-to-end validation** against image `v109530495-91d30ac1-7-arm64` on a prod-connected test drive. Behavior matched the Confluence contract (v13) on every scenario in this table. The contract has shifted since (sentinel defaults, per-env paths) — a fresh E2E pass on the latest image should follow before merge.

  | Scenario | Works as documented? |
  |---|---|
  | Operator `[]` + `[]` kill-switch | Yes — effective `[]` on both axes |
  | Narrower commands `["rshell:cat"]` | Yes — only `cat` works |
  | File-level path entry | Yes — WARN at load; entry silently dropped by rshell sandbox |
  | Directory sub-path `["/host/var/log/nginx"]` | Yes — narrower wins |
  | Bare command name `["cat"]` | Yes — WARN at load; intersection empty |
  | Disjoint non-empty | Yes — effective `[]`, everything blocked |
  | Prefix-sibling `/var/logger` vs `/var/log` | Yes — separator boundary honored |

### Additional Notes

- The startup warnings (unnamespaced commands, backslash paths, non-directory paths) are advisory. Entries still flow through to the handler; the warnings make silent-failure modes observable in agent logs.
- Confluence will need a follow-up update once the per-env path shape is finalized in Balto.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


Co-authored-by: matt-dz <[email protected]>
Co-authored-by: jules.macret <[email protected]>

Replaces the brittle 'HorizontalLastActions > 0' gate with an explicit
LastScaledTarget tracker on PodAutoscalerInternal. The tracker records
(namespace, name, GVK) every time the horizontal controller successfully
writes `.spec.replicas`, and is cleared after a successful release. The
release path now fires on three triggers, not one:

  - horizontal scaling disabled  (existing)
  - apply mode switched to Preview  (#4 — previously not covered)
  - the DPA's TargetRef was retargeted to a different workload, so the
    OLD target still holds the stale managedFields entry while the new
    target has none  (#7 — previously released against the wrong target)

Release-failure handling also overhauled:

  - On failure, the helper now constructs a ConditionError, emits a
    Warning Event on the DPA, calls UpdateFromHorizontalAction(nil, err),
    increments HorizontalActionErrorInc, and returns the error so the
    workqueue's maxRetry guard caps the loop instead of hot-looping
    invisibly with (Requeue, nil)  (#3 — was previously silent).
  - The three delete branches (remote-owned, profile-managed,
    local-owned) now retry release up to maxRetry attempts via
    c.Workqueue.NumRequeues; on exhaustion they log Errorf and proceed
    with the delete so a permanently broken release (RBAC never granted)
    cannot indefinitely block a user from deleting a DPA  (#1).

Tests:
  - Existing TestHorizontalControllerReleaseOwnershipOnDisable updated
    to seed LastScaledTarget instead of HorizontalLastActions.
  - New TestHorizontalControllerReleaseOwnershipOnPreviewTransition.
  - New TestHorizontalControllerReleaseOwnershipOnTargetRefChange
    asserts release fires against the OLD target, not the spec's
    current target.
  - TestHorizontalControllerReleaseOwnershipOnDisable_FailureRetainsState
    rewritten to assert (Requeue, err) and LastScaledTarget retention.
  - TestLeaderCreateDeleteLocal / TestLeaderCreateDeleteRemote updated
    to seed LastScaledTarget so the delete path actually exercises the
    release call.
  - testScalingDecision now mirrors SetLastScaledTarget after every
    successful scale, matching production semantics.

LastScaledTarget is in-memory only — on cluster-agent restart it resets
and the next successful scale re-populates it. The trade-off is a
narrow window where a DPA disabled across a controller restart with no
subsequent scale would not release; acceptable in exchange for not
needing CRD status schema changes.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Replaces the brittle 'HorizontalLastActions > 0' gate with an explicit
LastScaledTarget tracker on PodAutoscalerInternal. The tracker records
(namespace, name, GVK) every time the horizontal controller successfully
writes `.spec.replicas`, and is cleared after a successful release. The
release path now fires on three triggers, not one:

  - horizontal scaling disabled  (existing)
  - apply mode switched to Preview  (#4 — previously not covered)
  - the DPA's TargetRef was retargeted to a different workload, so the
    OLD target still holds the stale managedFields entry while the new
    target has none  (#7 — previously released against the wrong target)

Release-failure handling also overhauled:

  - On failure, the helper now constructs a ConditionError, emits a
    Warning Event on the DPA, calls UpdateFromHorizontalAction(nil, err),
    increments HorizontalActionErrorInc, and returns the error so the
    workqueue's maxRetry guard caps the loop instead of hot-looping
    invisibly with (Requeue, nil)  (#3 — was previously silent).
  - The three delete branches (remote-owned, profile-managed,
    local-owned) now retry release up to maxRetry attempts via
    c.Workqueue.NumRequeues; on exhaustion they log Errorf and proceed
    with the delete so a permanently broken release (RBAC never granted)
    cannot indefinitely block a user from deleting a DPA  (#1).

Tests:
  - Existing TestHorizontalControllerReleaseOwnershipOnDisable updated
    to seed LastScaledTarget instead of HorizontalLastActions.
  - New TestHorizontalControllerReleaseOwnershipOnPreviewTransition.
  - New TestHorizontalControllerReleaseOwnershipOnTargetRefChange
    asserts release fires against the OLD target, not the spec's
    current target.
  - TestHorizontalControllerReleaseOwnershipOnDisable_FailureRetainsState
    rewritten to assert (Requeue, err) and LastScaledTarget retention.
  - TestLeaderCreateDeleteLocal / TestLeaderCreateDeleteRemote updated
    to seed LastScaledTarget so the delete path actually exercises the
    release call.
  - testScalingDecision now mirrors SetLastScaledTarget after every
    successful scale, matching production semantics.

LastScaledTarget is in-memory only — on cluster-agent restart it resets
and the next successful scale re-populates it. The trade-off is a
narrow window where a DPA disabled across a controller restart with no
subsequent scale would not release; acceptable in exchange for not
needing CRD status schema changes.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Replaces the brittle 'HorizontalLastActions > 0' gate with an explicit
LastScaledTarget tracker on PodAutoscalerInternal. The tracker records
(namespace, name, GVK) every time the horizontal controller successfully
writes `.spec.replicas`, and is cleared after a successful release. The
release path now fires on three triggers, not one:

  - horizontal scaling disabled  (existing)
  - apply mode switched to Preview  (#4 — previously not covered)
  - the DPA's TargetRef was retargeted to a different workload, so the
    OLD target still holds the stale managedFields entry while the new
    target has none  (#7 — previously released against the wrong target)

Release-failure handling also overhauled:

  - On failure, the helper now constructs a ConditionError, emits a
    Warning Event on the DPA, calls UpdateFromHorizontalAction(nil, err),
    increments HorizontalActionErrorInc, and returns the error so the
    workqueue's maxRetry guard caps the loop instead of hot-looping
    invisibly with (Requeue, nil)  (#3 — was previously silent).
  - The three delete branches (remote-owned, profile-managed,
    local-owned) now retry release up to maxRetry attempts via
    c.Workqueue.NumRequeues; on exhaustion they log Errorf and proceed
    with the delete so a permanently broken release (RBAC never granted)
    cannot indefinitely block a user from deleting a DPA  (#1).

Tests:
  - Existing TestHorizontalControllerReleaseOwnershipOnDisable updated
    to seed LastScaledTarget instead of HorizontalLastActions.
  - New TestHorizontalControllerReleaseOwnershipOnPreviewTransition.
  - New TestHorizontalControllerReleaseOwnershipOnTargetRefChange
    asserts release fires against the OLD target, not the spec's
    current target.
  - TestHorizontalControllerReleaseOwnershipOnDisable_FailureRetainsState
    rewritten to assert (Requeue, err) and LastScaledTarget retention.
  - TestLeaderCreateDeleteLocal / TestLeaderCreateDeleteRemote updated
    to seed LastScaledTarget so the delete path actually exercises the
    release call.
  - testScalingDecision now mirrors SetLastScaledTarget after every
    successful scale, matching production semantics.

LastScaledTarget is in-memory only — on cluster-agent restart it resets
and the next successful scale re-populates it. The trade-off is a
narrow window where a DPA disabled across a controller restart with no
subsequent scale would not release; acceptable in exchange for not
needing CRD status schema changes.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

…iant (#52777)

## Intent

A `datadog-fips-agent` host running `start-experiment` would silently pull and run a **non-FIPS** package unless `DD_FIPS_MODE=true` was set by hand. Investigation found several independent breakages in the FIPS package-selection path:

| # | Component | Root cause | Example failure | Fix |
|---|-----------|-----------|----------------|-----|
| 1 | **Build-flavor detection** (`env.FromEnv`) | The code asked "is FIPS crypto currently active in this process?" (`pkg/fips.Enabled()`) instead of "was this binary *compiled* as a FIPS build?" — two different questions. The installer spawns short-lived child processes that haven't initialized the crypto backend yet, so they always answer "no" and pull non-FIPS packages. | `start-experiment` on a `datadog-fips-agent` host downloads the non-FIPS agent variant unless `DD_FIPS_MODE=true` is set manually. | New `pkg/fips.BuiltForFIPS()` reads the binary's embedded build metadata to check for the `requirefips`/`goexperiment.systemcrypto` compile-time flags — a reliable answer regardless of runtime state. |
| 2 | **Daemon env** (`NewDaemon`) | The long-running installer daemon builds its own configuration object from scratch rather than going through the shared `env.FromEnv()` path, so `FIPSMode` was simply never set — even if the binary was a FIPS build. | Even with fix #1 applied, the daemon's OCI download still picked the non-FIPS agent variant because its internal config always had `FIPSMode=false`. | `NewDaemon` now sets `FIPSMode: pkgfips.BuiltForFIPS() || DD_FIPS_MODE=="true"`, matching `FromEnv` exactly. A FIPS-flavor daemon now automatically operates in FIPS mode without any env override. |
| 3 | **FIPS provider setup** (OCI flow) | FIPS-compiled binaries require a one-time per-machine self-test of the OpenSSL FIPS module (`fipsinstall.sh`) before they can run. DEB/RPM packages run this in their post-install script; the OCI code path skipped it entirely. | After downloading the experiment via OCI, the new agent binary panics immediately on startup: `panic: opensslcrypto: FIPS mode requested but not available`. | New `ensureAgentFIPSProvider()` runs `fipsinstall.sh` before re-exec'ing into the agent. Also hardened: clears inherited `OPENSSL_*` env before running the script to prevent cross-tree contamination, and fails explicitly when the previous run left the tree in a partial state. |
| 4 | **Library loading under privilege restriction** (installer.layer bootstrap) | The daemon's systemd unit has `CapabilityBoundingSet=all`. When it exec's the bootstrap installer binary, the kernel sets `AT_SECURE`, causing the dynamic linker to silently drop all `\$ORIGIN`-relative RPATH entries. The build system had just converted every RPATH to `\$ORIGIN`-relative, so the binary loaded the host's system libcrypto (wrong version) and panicked. | `start-experiment` fails with `panic: opensslcrypto: FIPS mode requested… OpenSSL 3.5.6` — the system OpenSSL (3.5.6) is loaded instead of the bundled one (3.5.7), and the FIPS module refuses to run. | `patchelf --add-rpath /opt/datadog-agent/embedded/lib` is applied to the FIPS installer binary in `datadog-agent-finalize.rb` **after** `rpath-edit` (which would otherwise overwrite it). Absolute RPATH entries are always honoured under `AT_SECURE`. The path is version-independent: it is the deb install location directly, and a symlink to the current stable OCI tree on OCI-managed hosts. |
| 5 | **Standalone installer FIPS build** | No FIPS variant of the standalone `datadog-installer` binary existed. Its `BuiltForFIPS()` was always `false`, so it always selected the non-FIPS package variant when used to bootstrap OCI installs. | FIPS OCI-managed installs downloaded the wrong variant even after other fixes were in place. | New `installer-fips-amd64/arm64-oci` CI jobs build a FIPS-flavor standalone installer and publish it as the `Platform.Variant="fips"` layer of the `datadog-installer` OCI image index. |

Additional hardening: `fipsinstall.sh` now derives `INSTALL_DIR` via `realpath` (fixes OCI relocation), rewrites the `.include` path in `openssl.cnf` to the physical directory and verifies the substitution succeeded, and exits non-zero on verify failure. `tasks/omnibus.py`'s `_patch_binary_rpath` is unchanged; the existing unit test is updated.

## Package changes

- New `pkg/fips/fips_buildinfo.go` + `fips_buildinfo_test.go`: runtime `BuiltForFIPS()` via `debug.ReadBuildInfo()`, replacing three build-tag-constrained stubs that were unreliable on some deb builds.
- New `datadog-fips-installer` OCI variant: FIPS-flavor standalone installer published as `Platform.Variant="fips"` in the `datadog-installer` OCI image index.

## Surface changes

- `start-experiment` on a `datadog-fips-agent` host now selects and configures the FIPS package end-to-end without any env override.
- New CI jobs: `installer-fips-amd64-oci` and `installer-fips-arm64-oci` (the non-oci variants are hidden templates).
- `fipsinstall.sh`: stricter — exits non-zero on verify failure or broken intermediate state.
- No CLI, config, or proto changes.

## Automated review results

| Agent | Findings | Fixed | Remaining |
|-------|----------|-------|-----------|
| Go best practices | 1 | 0 | 1 (pre-existing comment typo, out of scope) |
| Go code reviewer (7 sub-reviewers) | 17 | 17 | 0 |

## QA guidelines

Install from pipeline A's deb, then run `start-experiment` from pipeline B's OCI. Both should report `FIPS Mode: enabled`.

```bash
# Pipeline A — install deb
sudo env \
  DD_API_KEY=<key> DD_SITE=datadoghq.com DD_AGENT_MAJOR_VERSION=7 \
  DD_AGENT_FLAVOR=datadog-fips-agent DD_REMOTE_UPDATES=true \
  TESTING_KEYS_URL=apttesting.datad0g.com/test-keys \
  TESTING_APT_URL=s3.amazonaws.com/apttesting.datad0g.com/datadog-agent/pipeline-<A>-a7 \
  TESTING_APT_REPO_VERSION="stable-x86_64 7" \
  DD_INSTALLER_REGISTRY_URL=installtesting.datad0g.com \
  DD_INSTALLER_REGISTRY_URL_AGENT_PACKAGE=installtesting.datad0g.com \
  DD_INSTALLER_DEFAULT_PKG_VERSION_DATADOG_AGENT=pipeline-<A> \
  bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)"

sudo tee /etc/datadog-agent/datadog.yaml <<EOF
api_key: <key>
site: datadoghq.com
remote_updates: true
remote_configuration:
  enabled: true
installer:
  registry:
    url: installtesting.datad0g.com
EOF
sudo chown dd-agent:dd-agent /etc/datadog-agent/datadog.yaml
sudo systemctl restart datadog-agent
sleep 10
sudo datadog-agent status | grep "FIPS Mode"   # → FIPS Mode: enabled

# Pipeline B — start-experiment from a different OCI
BIN=/opt/datadog-agent/embedded/bin/installer
V=<version-from-pipeline-B>
sudo \$BIN daemon set-catalog '{"packages":[{"package":"datadog-agent","version":"'"$V"'","url":"oci://installtesting.datad0g.com/agent-package:pipeline-<B>"}]}'
sudo \$BIN daemon start-experiment datadog-agent "\$V"
sudo /opt/datadog-packages/datadog-agent/experiment/bin/agent/agent status | grep "FIPS Mode"   # → FIPS Mode: enabled
```

Co-authored-by: baptiste.foy <[email protected]>

Fixes bug #1 (concurrent multiplexing). The previous per-connection reassembly
assumed one response at a time: when two responses were multiplexed (their DATA
frames interleaved) on one connection, only the first stream was tracked and the
second was mixed in / lost.

Replace it with a per-connection HTTP/2 frame demuxer (llmConnDemux): walk the
connection's decrypted byte stream frame-by-frame and route each DATA frame's
payload to a per-stream accumulator keyed by the frame's own stream id, so
interleaved responses are separated. Each stream completes independently on
token-usage-seen or END_STREAM (read from the frame flags), then pairs with the
request captured on the same HTTP/2 stream. Includes a resync (skip to a
plausible frame header after a lost/truncated read) and bounds on pending bytes,
per-stream body, and concurrent streams.

Removes the single-response reassembler (llmRespReasm) and the standalone
deframeHTTP2Data helper — the demuxer subsumes both (it de-frames as it walks).

TestResponseDemuxInterleaved is the regression test: two responses with
interleaved DATA frames on one connection, delivered in frame-unaligned reads,
both emit with their correct bodies. All existing reassembly/pairing tests still
pass (updated to feed framed responses, since the demuxer requires real frames).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>