CSI TLS verification lacks caching, causing performance degradation #1182

Expected Behavior

The TLS configuration verification in resolveTLSConfig should efficiently verify CA certificates during connection handshakes while maintaining the ability to detect and reload CA certificate rotations without impacting performance.

Actual Behavior

The CA certificate verification logic in resolveTLSConfig currently reads the CA trust bundle from disk on every TLS connection handshake. This causes redundant disk I/O operations on each connection, which may impact performance and scalability.

The current implementation is acceptable for the initial implementation but has been identified as needing optimization before general availability (GA).

Steps to Reproduce the Problem

1. Deploy CSI driver controller with TLS enabled via Pod Identity (PR identity: Secure CSI Driver TCP Connections via Pod Identity #956)
2. Configure multiple ate-api instances making requests to the CSI controller
3. Monitor disk I/O and performance metrics during sustained load
4. Observe that CA certificate pool is re-read from disk on each TLS handshake

Specifications

• Version: From PR identity: Secure CSI Driver TCP Connections via Pod Identity #956 (identity: Secure CSI Driver TCP Connections via Pod Identity)
• Component: CSI Driver TLS Configuration
• Related Files:
  • internal/volume/csi/plugin.go (function: resolveTLSConfig)
• Related PR: identity: Secure CSI Driver TCP Connections via Pod Identity #956
• Target Milestone: M2 (pre-GA)

Additional Context

The resolveTLSConfig function uses VerifyConnection to dynamically verify server certificates against the latest CA bundle read from disk on each handshake. While this ensures CA rotations are detected without process restart, it currently lacks a caching mechanism.

Implementation should:

• Cache the CA certificate pool to reduce disk I/O
• Maintain dynamic reload capability to detect CA certificate rotations
• Reference implementation: https://github.com/ahmedtd/tinycert/blob/main/lib/spiffefsd/spiffefsd.go