Component

API Server / GraphQL

Infrahub version

4.7.3 (Enterprise, medium), deployed on EKS

Current Behavior

When deploying with S3 artifact storage on EKS using IRSA (IAM Roles for
Service Accounts) — the standard EKS pattern where pods assume an IAM role via
a projected service account token, with no static AWS_ACCESS_KEY_ID /
AWS_SECRET_ACCESS_KEY set — S3 access fails:

botocore.exceptions.ClientError: AuthorizationHeaderMalformed —
a non-empty Access Key (AKID) must be provided

Root cause: S3StorageSettings in backend/infrahub/config.py defines
access_key_id and secret_access_key with default="", so they can never be
None. InfrahubS3ObjectStorage (backend/infrahub/storage.py) inherits from
fastapi_storages.S3Storage, whose __init__ passes these straight through:

self._s3 = boto3.resource(
    "s3",
    endpoint_url=self._url,
    aws_access_key_id=self.AWS_ACCESS_KEY_ID,        # ""
    aws_secret_access_key=self.AWS_SECRET_ACCESS_KEY, # ""
)

When an empty string is passed explicitly, boto3 uses it verbatim instead of
falling back to the default credential provider chain (IRSA, instance profiles,
ECS task roles, etc.). This restricts S3 storage to static keys only.

Expected Behavior

When no static AWS credentials are configured, Infrahub should let boto3 fall
back to the default AWS credential provider chain, so S3 storage works with
IRSA, EC2 instance profiles, ECS task roles, and any standard AWS credential
method — not only static access keys.

Steps to Reproduce

1. Deploy Infrahub on EKS with S3 artifact storage configured (bucket,
   endpoint), but without AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY.
2. Attach an IAM role to the pod via IRSA (AWS_WEB_IDENTITY_TOKEN_FILE +
   AWS_ROLE_ARN present in the environment).
3. Trigger any S3 read/write (e.g. generate or fetch an artifact).
4. Observe AuthorizationHeaderMalformed — the empty access key is sent
   instead of credentials being resolved via IRSA.

Additional Information

Verified that passing None (vs "") restores the credential-chain fallback:

• aws_access_key_id="" → sends empty AKID → fails
• aws_access_key_id=None → falls back to credential chain → works

Suggested direction (for the implementer to evaluate): when the credential
fields are empty, omit them from the boto3.resource(...) call rather than
passing empty strings — e.g. override the resource construction in
InfrahubS3ObjectStorage so the kwargs are only included when an access key is
set. This would cover IRSA, instance profiles, and any standard AWS credential
method. The fix likely needs to live in Infrahub's InfrahubS3ObjectStorage
subclass rather than relying on the upstream fastapi_storages library.

Working workaround (no static keys, no custom image)

We unblocked ourselves by injecting a sitecustomize.py into site-packages via
a Kubernetes ConfigMap volume mount. Python loads sitecustomize.py at
interpreter startup, before application code, and it flips the empty-string
credentials to None:

import fastapi_storages.s3

_orig_init = fastapi_storages.s3.S3Storage.__init__

def _patched_init(self):
    if not self.AWS_ACCESS_KEY_ID:
        self.AWS_ACCESS_KEY_ID = None
    if not self.AWS_SECRET_ACCESS_KEY:
        self.AWS_SECRET_ACCESS_KEY = None
    _orig_init(self)

fastapi_storages.s3.S3Storage.__init__ = _patched_init

Deployment:

1. ConfigMap infrahub-irsa-patch holds the patch script.
2. A Flux HelmRelease postRenderer mounts it into the server and task-worker
   pods at /.venv/lib/python3.13/site-packages/sitecustomize.py via subPath.
3. On startup Python loads the patch → empty credentials become None → boto3
   falls back to the default credential provider chain → picks up the IRSA
   token from AWS_WEB_IDENTITY_TOKEN_FILE.

Result: artifact generation flows complete successfully with S3 storage via
IRSA — no static keys, no custom images, no init containers. Once the upstream
fix ships, the ConfigMap and volume mount are simply removed.