Summary

This PR properly fixes issue #2270 where AWS SDK debug logs continue to appear in helmfile output despite the fix in PR #2288.

Two-part solution: This PR works in conjunction with vals PR #893 to provide flexible, configurable AWS SDK logging with secure defaults.

⚠️ Dependency

This PR depends on: helmfile/vals#893

Current status:

• ✅ vals PR helmfile hangs on large diff with --enable-live-output #893 implements clean SetDefaultLogLevel() package-level function
• ✅ Helmfile references vals commit 847469e1d13b via replace directive
• ✅ All tests passing with new vals version
• ⏳ Once vals PR merges, will update to official release version

After vals PR merge:

go get github.com/helmfile/vals@latest
go mod tidy
# Remove replace directive from go.mod

Problem Statement

As reported in PR #2288 comment, users still see AWS SDK debug logs in helmfile 1.2.1:

❯ helmfile diff -l cluster=staging
SDK 2025/11/24 07:54:06 DEBUG Request
PUT /latest/api/token HTTP/1.1
Host: 169.254.169.254

This exposes sensitive information including:

• 🔓 AWS tokens and authorization headers
• 🔓 Request/response bodies containing credentials
• 🔓 Secret metadata from vals providers

Root Cause Analysis

The original fix in PR #2288 added LogOutput: io.Discard to vals.Options, but this only suppressed vals' own logging, not the AWS SDK's debug output.

Why the issue persisted:

1. AWS SDK v2 uses a separate logging system:

   • Controlled by AWS_SDK_GO_LOG_LEVEL environment variable
   • Configured via WithClientLogMode option in AWS config
   • Completely independent from vals' LogOutput

2. AWS SDK clients in remote.go needed attention:

   • Three config.LoadDefaultConfig() calls had no log suppression
   • These clients interact with S3, exposing credentials in logs

3. Vals library defaulted to verbose logging:

   • vals hardcoded aws.LogRetries | aws.LogRequest as default
   • No programmatic way to change this behavior
   • Leaked credentials on every AWS API call

Solution

Two-Part Fix

Part 1: Vals Library Enhancement (PR #893)

Clean implementation without environment variable modification:

• ✅ Added Options.AWSLogLevel field for programmatic control
• ✅ Created package-level SetDefaultLogLevel() function in awsclicompat package
• ✅ Enhanced parseAWSLogLevel() to check package-level variable between env var and parameter
• ✅ Changed default from aws.LogRetries | aws.LogRequest to 0 (no logging)
• ✅ Added preset levels: "off", "minimal", "standard", "verbose"
• ✅ Maintains environment variable precedence (AWS_SDK_GO_LOG_LEVEL always wins)

Key implementation details:

// vals.go - Calls SetDefaultLogLevel() when Options.AWSLogLevel is provided
func New(opts Options) (*Runtime, error) {
    // Configure AWS SDK logging if AWSLogLevel is provided
    // This provides programmatic control without modifying environment variables
    if opts.AWSLogLevel != "" {
        awsclicompat.SetDefaultLogLevel(opts.AWSLogLevel)
    }
    // ...
}

// pkg/awsclicompat/session.go - Package-level configuration
var defaultLogLevel string

func SetDefaultLogLevel(level string) {
    defaultLogLevel = level
}

func parseAWSLogLevel(paramDefault string) aws.ClientLogMode {
    // Priority: AWS_SDK_GO_LOG_LEVEL > defaultLogLevel > paramDefault > secure default
    logLevel := strings.TrimSpace(os.Getenv("AWS_SDK_GO_LOG_LEVEL"))
    
    if logLevel == "" && defaultLogLevel != "" {
        logLevel = defaultLogLevel
    }
    
    if logLevel == "" {
        logLevel = paramDefault
    }
    
    if logLevel == "" {
        return 0  // Secure default
    }
    // ... preset handling
}

Part 2: Helmfile Changes (This PR)

1. Added Flexible Configuration (pkg/envvar/const.go)

// HELMFILE_AWS_SDK_LOG_LEVEL controls AWS SDK logging level
// Valid values: "off" (default), "minimal", "standard", "verbose", or custom
AWSSDKLogLevel = "HELMFILE_AWS_SDK_LOG_LEVEL";

Preset levels:

• "off" (default) - No AWS SDK logging (secure, prevents credential leakage)
• "minimal" - Log retries only (minimal debugging)
• "standard" - Log retries + requests (previous default behavior)
• "verbose" - Log everything (requests, responses, bodies, signing)
• Custom - Comma-separated: "request,response,signing"

2. Enhanced Vals Configuration (pkg/plugins/vals.go)

// Read log level from HELMFILE_AWS_SDK_LOG_LEVEL
logLevel := strings.TrimSpace(os.Getenv(envvar.AWSSDKLogLevel))

// Default to "off" for security if not specified
if logLevel == "" {
    logLevel = "off"
}

// Pass to vals library using new Options.AWSLogLevel field
// vals will call awsclicompat.SetDefaultLogLevel() internally
opts.AWSLogLevel = logLevel

// Also suppress vals' own logging when set to "off"
if logLevel == "off" {
    opts.LogOutput = io.Discard
}

3. Conditional AWS SDK Log Suppression (pkg/remote/remote.go)

Made WithClientLogMode(0) conditional in all 3 AWS config locations:

• S3Getter.Get() - S3 chart downloads
• S3Getter.S3FileExists() - S3 file checks (region config)
• S3Getter.S3FileExists() - S3 file checks (default config)

// Only suppress if explicitly set to "off" (default)
if awsSDKLogLevel == "off" {
    configOpts = append(configOpts, config.WithClientLogMode(0))
}
// For other values, AWS SDK respects AWS_SDK_GO_LOG_LEVEL env var

4. Comprehensive Unit Tests

Added extensive test coverage to verify AWS SDK logging behavior:

pkg/plugins/vals_test.go:

• ✅ TestAWSSDKLogLevelConfiguration - Tests all preset levels + custom

  • Verifies default ("off"), explicit off, minimal, standard, verbose, custom
  • Validates LogOutput = io.Discard only when level is "off"
  • 6 test cases covering all configuration scenarios

• ✅ TestEnvironmentVariableReading - Tests env var parsing

  • Empty defaults to "off"
  • Whitespace trimming
  • Value preservation
  • 3 test cases

pkg/remote/remote_test.go:

• ✅ TestAWSSDKLogLevelInit - Tests init() logic
  • Default behavior (empty → "off")
  • All preset values (off, minimal, standard, verbose)
  • Whitespace trimming
  • 6 test cases

Test Results:

✅ pkg/plugins: PASS (all 3 test suites, 9 new test cases)
✅ pkg/remote: PASS (all test suites, 6 new test cases)
✅ Linting: 0 issues (golangci-lint)

Usage Examples

🔒 Production (Default - Secure)

helmfile diff
# ✅ No AWS SDK debug output
# ✅ No vals debug output
# ✅ Credentials protected

🔍 Minimal Debugging (Retries Only)

HELMFILE_AWS_SDK_LOG_LEVEL=minimal helmfile diff
# Shows only retry attempts
# Minimal overhead, good for troubleshooting connectivity

📊 Standard Debugging (Previous Default)

HELMFILE_AWS_SDK_LOG_LEVEL=standard helmfile diff
# Shows retries + request URLs
# Good for debugging auth/permissions issues

🔬 Verbose Debugging (Full Details)

HELMFILE_AWS_SDK_LOG_LEVEL=verbose helmfile diff
# Shows everything: requests, responses, bodies, signing
# Use for deep debugging only (exposes credentials!)

🎯 Custom Granular Control

HELMFILE_AWS_SDK_LOG_LEVEL=request,response helmfile diff
# Fine-grained control over specific log components

🌐 Environment Variable Priority

# AWS_SDK_GO_LOG_LEVEL always overrides HELMFILE_AWS_SDK_LOG_LEVEL
AWS_SDK_GO_LOG_LEVEL=verbose helmfile diff
# This takes precedence over any helmfile setting

Configuration Priority

The logging configuration follows this priority order (highest to lowest):

1. AWS_SDK_GO_LOG_LEVEL environment variable (highest priority)
2. HELMFILE_AWS_SDK_LOG_LEVEL environment variable → Options.AWSLogLevel → awsclicompat.SetDefaultLogLevel()
3. Secure default ("off" - no logging)

This ensures:

• Users can override helmfile's settings when needed
• AWS SDK's standard environment variable always works
• Secure by default when nothing is configured
• Clean implementation without global env var modification

Approach

This PR takes a two-tier approach:

1. Fix the root cause in vals library (PR helmfile hangs on large diff with --enable-live-output #893)

   • Adds programmatic control over AWS SDK logging using clean package-level function
   • Uses SetDefaultLogLevel() instead of modifying global environment variables
   • Changes default to secure (no logging)
   • Benefits all vals users, not just helmfile

2. Leverage vals improvements in helmfile (this PR)

   • Uses new Options.AWSLogLevel field
   • Adds flexible configuration via environment variable
   • Addresses AWS SDK usage in remote S3 operations
   • Provides comprehensive test coverage

This approach ensures the fix is thorough and maintainable, with secure defaults that prevent credential leakage while still allowing users to enable debugging when needed.

Security Impact

🔒 Before (helmfile ≤ 1.2.1)

• AWS SDK logged by default: aws.LogRetries | aws.LogRequest
• Credentials exposed in logs:

  SDK 2025/11/24 07:54:06 DEBUG Request
  PUT /latest/api/token HTTP/1.1
  Host: 169.254.169.254
  X-aws-ec2-metadata-token: REDACTED_BUT_VISIBLE_IN_LOGS
  

🔐 After (This PR)

• Default: No AWS SDK logging (0)
• Credentials protected unless user explicitly enables debugging
• Opt-in security model: Users must consciously enable sensitive logging
• Clean implementation: No global environment variable modification

Security principles:

• ✅ Secure by default - No logging unless needed
• ✅ Principle of least privilege - Minimal logging even when enabled
• ✅ Defense in depth - Prevents accidental credential exposure
• ✅ Explicit opt-in - Users understand security implications when enabling
• ✅ Clean architecture - Package-level configuration, no global state pollution

Testing

All code compiles, passes linting, and comprehensive test coverage:

Unit Tests:

✅ go test ./pkg/plugins/... -v
   - TestValsInstance: PASS
   - TestAWSSDKLogLevelConfiguration: PASS (6 test cases)
   - TestEnvironmentVariableReading: PASS (3 test cases)

✅ go test ./pkg/remote/... -v  
   - TestAWSSDKLogLevelInit: PASS (6 test cases)
   - All existing remote tests: PASS

✅ golangci-lint run ./pkg/plugins/... ./pkg/remote/... ./pkg/envvar/...
   - 0 issues

Test Coverage:

• New test cases: 15 (across 3 test functions)
• Code paths tested:
  • Default behavior (no env var)
  • All preset levels (off, minimal, standard, verbose)
  • Custom log level values
  • Whitespace trimming
  • LogOutput configuration
  • Environment variable precedence

Files Changed

Helmfile Changes (This PR)

Total: 7 files changed, 271 insertions(+), 17 deletions(-)

Vals Changes (PR #893)

Total: 3 files changed, 99 insertions(+), 24 deletions(-)

Migration Path

For Vals Dependency Update (After PR #893 Merges)

# In helmfile repository
# Remove replace directive from go.mod
go get github.com/helmfile/vals@latest
go mod tidy
go build ./cmd/helmfile

For Users Upgrading Helmfile

No action required! The default is secure (no logging).

If you need debugging:

# Minimal debugging (recommended for troubleshooting)
HELMFILE_AWS_SDK_LOG_LEVEL=minimal helmfile diff

# Standard debugging (previous default behavior)
HELMFILE_AWS_SDK_LOG_LEVEL=standard helmfile diff

# Verbose debugging (for deep investigation)
HELMFILE_AWS_SDK_LOG_LEVEL=verbose helmfile diff

Related Issues & PRs

• Fixes: Helmfile 1.1.6+ uses Debug log level when making requests via the AWS Go SDK #2270 (AWS SDK debug logging exposing credentials)
• Depends on: Add configurable AWS SDK logging with secure defaults vals#893 (vals library enhancement with clean SetDefaultLogLevel())
• Related: Fix four critical bugs: array merging (#2281), AWS SDK logging (#2270), helmDefaults skip flags (#2269), and OCI chart versions (#2247) #2288 (initial fix attempt - addressed vals logging but not AWS SDK)
• Related: disable AWS GO SDK logging #2289 (alternative approach to the same issue)

Checklist

• Code changes are focused and minimal
• Comprehensive unit tests added (15 new test cases)
• All tests pass (pkg/plugins, pkg/remote)
• All linting passes (golangci-lint)
• No breaking changes introduced
• Secure by default (no logging)
• Flexible configuration (4 presets + custom)
• Follows existing code patterns
• Clean git history with signed commits
• Security implications documented
• Backward compatible via "standard" preset
• Comprehensive documentation in comments
• Dependency clearly documented (vals PR helmfile hangs on large diff with --enable-live-output #893)
• Environment variable precedence maintained
• All AWS SDK client creation points addressed
• Test coverage for all configuration scenarios
• Clean implementation without global environment variable modification
• Vals reference updated to commit 847469e1d13b