Skip to content

fix(validator): make ExtractResult sidecar-safe by reading 'validator' container explicitly#881

Merged
mchmarny merged 3 commits into
NVIDIA:mainfrom
xdu31:fix/validator-sidecar-safe-result
May 14, 2026
Merged

fix(validator): make ExtractResult sidecar-safe by reading 'validator' container explicitly#881
mchmarny merged 3 commits into
NVIDIA:mainfrom
xdu31:fix/validator-sidecar-safe-result

Conversation

@xdu31

@xdu31 xdu31 commented May 14, 2026

Copy link
Copy Markdown
Contributor

Summary

Makes ExtractResult() and HandleTimeout() sidecar-safe by explicitly reading from the "validator" container by name instead of assuming the first container (index 0) is the validator.

Motivation and Context

Fixes #795

The current implementation assumes the first container in the pod is the validator, which breaks when external controllers inject sidecars for log streaming or result processing. This change enforces the ADR-002 contract that the validator container must be named "validator".

Type of Change

  • Bug fix (non-breaking change that fixes an issue)

Components Affected

  • pkg/validator/job - result extraction logic

Implementation Notes

  • Added findContainerStatus() helper function for name-based container lookup
  • Updated ExtractResult() to find container by name "validator" instead of using index 0
  • Updated HandleTimeout() with same sidecar-safe lookup
  • Pass explicit container name "validator" to GetPodLogs() calls
  • Updated function documentation to document the ADR-002 contract
  • Error messages mention ADR-002 when validator container not found

Testing

  • All unit tests pass with race detector
  • Added 4 new test cases for sidecar scenarios:
    • TestExtractResultWithSidecar - validator + sidecar, reads from validator only
    • TestExtractResultSidecarOnlyNoValidator - sidecar only, fails with ADR-002 violation
    • TestHandleTimeoutWithSidecar - timeout with sidecar present
    • TestHandleTimeoutSidecarOnlyNoValidator - timeout with no validator container
  • Updated all existing tests to set container Name field to "validator"
  • E2E tests: 20 passed, 0 failed

Risk Assessment

Low risk - Defensive change that makes existing functionality more robust. The "validator" container name is already part of the public contract (ADR-002), so this change enforces what should already be true.

Acceptance Criteria

  • ExtractResult() finds container by name "validator" instead of using index 0
  • GetPodLogs() is called with explicit container name "validator"
  • HandleTimeout() uses same sidecar-safe lookup
  • Function documentation explicitly states "validator" contract per ADR-002
  • Unit tests verify behavior with multiple containers (validator + sidecar)
  • Error message when "validator" container not found mentions ADR-002

Checklist

  • Tests pass locally (make test)
  • Linting passes (golangci-lint run)
  • E2E tests pass (make qualify)
  • Documentation updated (godoc comments)

…' container explicitly

Fixes NVIDIA#795

- Add findContainerStatus() helper to find container by name instead of assuming index 0
- Update ExtractResult() to explicitly read from 'validator' container (ADR-002 contract)
- Update HandleTimeout() with same sidecar-safe lookup
- Pass explicit container name 'validator' to GetPodLogs()
- Add comprehensive test coverage for sidecar scenarios
- Document ADR-002 contract in function comments
@xdu31 xdu31 requested a review from a team as a code owner May 14, 2026 05:31
@coderabbitai

coderabbitai Bot commented May 14, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

This PR implements ADR-002 sidecar-safety by refactoring ExtractResult and HandleTimeout in pkg/validator/job/result.go to explicitly select the "validator" container by name instead of assuming the first container. A new findContainerStatus helper locates container status entries by name. Both functions now return an ADR-002 violation message with ExitCode = -1 when the validator container is missing. All existing test fixtures are updated to include the "validator" container name, the negative test is renamed and updated to verify the new error message, and four new test cases verify correct behavior when sidecars are present alongside the validator container.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and specifically describes the main change: making ExtractResult sidecar-safe by explicitly reading the 'validator' container.
Description check ✅ Passed The description comprehensively explains the problem, motivation, implementation details, testing, and acceptance criteria related to the sidecar-safety changes.
Linked Issues check ✅ Passed The PR fully implements all requirements from issue #795: finds container by name, passes explicit container name to GetPodLogs, adds findContainerStatus helper, updates HandleTimeout, documents ADR-002 contract, includes comprehensive tests for sidecar scenarios, and error messages reference ADR-002.
Out of Scope Changes check ✅ Passed All changes are directly aligned with issue #795 objectives. The ValidatorContainerName constant and deployer.go update support the core sidecar-safety improvements with no unrelated modifications.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/validator/job/result_test.go`:
- Around line 498-500: The current test assertion only checks for "container
'validator' not found" and can pass if the ADR-002 reference is removed; update
the test that inspects result.TerminationMsg to explicitly require the ADR-002
wording by asserting strings.Contains(result.TerminationMsg, "ADR-002")
(optionally along with the existing "container 'validator' not found" check) so
the message contract mandated by ADR-002 is enforced.

In `@pkg/validator/job/result.go`:
- Around line 57-59: Replace the repeated literal "validator" with a named
constant (e.g., ValidatorContainerName) and use that constant wherever the
string is used: in the call to
findContainerStatus(jobPod.Status.ContainerStatuses, ...), in any container
lookups, and in related log/error messages inside this file (references around
findContainerStatus, container status extraction, and message formatting).
Declare the constant in the central defaults package (pkg/defaults) per
guidelines and import it into pkg/validator/job/result.go, then update all
occurrences (including places currently using the literal in status checks and
logs) to reference pkg/defaults.ValidatorContainerName to prevent drift/typos.
- Around line 126-131: The timeout branch that checks for the validator
container (using findContainerStatus on jobPod.Status.ContainerStatuses) should
include the ADR-002 reference in the error text to match ExtractResult's
contract: update the TerminationMsg set in the HandleTimeout (where
result.ExitCode = -1 and TerminationMsg currently says "timeout: validator did
not complete within %s (container 'validator' not found)") to include "ADR-002"
(e.g. append or insert "ADR-002" into the parenthetical) so the
missing-validator timeout message explicitly cites ADR-002 while keeping the
same timeout and container details.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 53ab1519-5810-465b-a8ab-322f026b746f

📥 Commits

Reviewing files that changed from the base of the PR and between e83b2ea and f6ed26b.

📒 Files selected for processing (2)
  • pkg/validator/job/result.go
  • pkg/validator/job/result_test.go

Comment thread pkg/validator/job/result_test.go Outdated
Comment thread pkg/validator/job/result.go Outdated
Comment thread pkg/validator/job/result.go Outdated

@mchmarny mchmarny left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Targeted fix, good test coverage of the sidecar-present and sidecar-only cases. CI green.

CodeRabbit already covered the magic-string extraction, the missing ADR-002 wording in HandleTimeout, and the test assertion strengthening — won't repeat those. Three additional notes:

  1. The "ADR-002 contract" / "ADR-002 violation" wording in comments and error messages isn't actually backed by ADR-002 — the doc doesn't codify the validator container name. Either add the contract to ADR-002 in this PR or reword to attribute it to the validator package.
  2. In HandleTimeout, the log fetch happens before the container-existence check — small reorder saves a wasted apiserver call on the missing-validator path.
  3. Producer side (deployer.go:175 .WithName("validator")) should consume the same constant CodeRabbit suggested extracting; pinning it inside pkg/validator/job is simpler than pkg/defaults.

Nothing blocks merge.

Comment thread pkg/validator/job/result.go Outdated
Comment thread pkg/validator/job/result.go Outdated
Comment thread pkg/validator/job/result.go
- Extract 'validator' string to ValidatorContainerName constant
- Update error messages to reference 'validator package contract' instead of ADR-002
- Reorder HandleTimeout to check container existence before fetching logs
- Update deployer.go to use the constant when creating validator container
- Strengthen test assertions to explicitly check for container name and 'not found' in error messages

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/validator/job/result.go`:
- Around line 125-131: The timeout message uses d.entry.Timeout (which may be
zero) instead of the effective timeout; in HandleTimeout call
d.resolvedTimeout() once, store it in a local variable (e.g., resolved :=
d.resolvedTimeout()) and reuse that value when formatting both the
validator-missing message (the block using findContainerStatus and setting
result.TerminationMsg) and the generic timeout branch later, replacing
d.entry.Timeout with the resolved variable so the message shows the actual
timeout; update references around ValidatorContainerName, result, and the
HandleTimeout path accordingly.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 1b5eb5cd-e7ba-420e-9f09-10fce1e99f2c

📥 Commits

Reviewing files that changed from the base of the PR and between f6ed26b and 7c001c3.

📒 Files selected for processing (3)
  • pkg/validator/job/deployer.go
  • pkg/validator/job/result.go
  • pkg/validator/job/result_test.go

Comment on lines +125 to +131
// Check container status from "validator" container first (before fetching logs)
cs, found := findContainerStatus(jobPod.Status.ContainerStatuses, ValidatorContainerName)
if !found {
result.ExitCode = -1
result.TerminationMsg = fmt.Sprintf("timeout: validator did not complete within %s (container %q not found - validator package contract)", d.entry.Timeout, ValidatorContainerName)
return result
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Use resolved timeout value in timeout messages.

Line 129 formats the message with d.entry.Timeout, which can be 0 when the entry relies on default timeout resolution. That can produce misleading timeout text (within 0s). Use d.resolvedTimeout() once in HandleTimeout and reuse it here and in the generic timeout branch on Line 149.

Suggested fix
 func (d *Deployer) HandleTimeout(ctx context.Context) *ctrf.ValidatorResult {
 	result := &ctrf.ValidatorResult{
 		Name:  d.entry.Name,
 		Phase: d.entry.Phase,
 	}
+	timeout := d.resolvedTimeout()
@@
 	cs, found := findContainerStatus(jobPod.Status.ContainerStatuses, ValidatorContainerName)
 	if !found {
 		result.ExitCode = -1
-		result.TerminationMsg = fmt.Sprintf("timeout: validator did not complete within %s (container %q not found - validator package contract)", d.entry.Timeout, ValidatorContainerName)
+		result.TerminationMsg = fmt.Sprintf("timeout: validator did not complete within %s (container %q not found - validator package contract)", timeout, ValidatorContainerName)
 		return result
 	}
@@
 	} else {
 		result.ExitCode = -1
-		result.TerminationMsg = fmt.Sprintf("timeout: validator did not complete within %s", d.entry.Timeout)
+		result.TerminationMsg = fmt.Sprintf("timeout: validator did not complete within %s", timeout)
 	}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/validator/job/result.go` around lines 125 - 131, The timeout message uses
d.entry.Timeout (which may be zero) instead of the effective timeout; in
HandleTimeout call d.resolvedTimeout() once, store it in a local variable (e.g.,
resolved := d.resolvedTimeout()) and reuse that value when formatting both the
validator-missing message (the block using findContainerStatus and setting
result.TerminationMsg) and the generic timeout branch later, replacing
d.entry.Timeout with the resolved variable so the message shows the actual
timeout; update references around ValidatorContainerName, result, and the
HandleTimeout path accordingly.

@xdu31

xdu31 commented May 14, 2026

Copy link
Copy Markdown
Contributor Author

Review comments addressed:

  • Extracted 'validator' string to ValidatorContainerName constant
  • Updated references from 'ADR-002' to 'validator package contract' (per feedback that ADR-002 doesn't document this)
  • Reordered HandleTimeout to check container existence before fetching logs
  • Updated deployer.go to use the constant
  • Strengthened test assertions to explicitly check for container name

@xdu31 xdu31 requested a review from mchmarny May 14, 2026 16:00
@mchmarny mchmarny enabled auto-merge (squash) May 14, 2026 16:07
@mchmarny mchmarny merged commit 3de6085 into NVIDIA:main May 14, 2026
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make ExtractResult sidecar-safe by reading from 'validator' container explicitly

2 participants