Skip to content

bug: Background flows hang against TLS-only Redis (INFRAHUB_CACHE_TLS_* not honored by Prefect result-storage block) #9217

Description

@PhillSimonds

Component

API Server / GraphQL

Infrahub version

1.9.3 (present in earlier versions too — defect was introduced when the Prefect Redis result-storage block was added).

Current Behavior

When Infrahub is configured to use an external TLS-only Redis (INFRAHUB_CACHE_TLS_ENABLED=true, plus optionally INFRAHUB_CACHE_TLS_INSECURE and INFRAHUB_CACHE_TLS_CA_FILE), background flows that persist results through Prefect — for example git_repositories_sync, clean-up-deadlocks, and schema_validate_migrations — start, run their internal task runs, and then hang in RUNNING indefinitely. The flow never reaches a terminal state.

Startup itself looks healthy, because RedisStorageContainer.asave(...) only writes the block document to Prefect's metadata DB; it does not connect to Redis. The failure only surfaces later, when a flow first tries to persist a result through the block.

Cache (InfrahubCache) and distributed-lock (InfrahubLock) traffic continue to work, because those code paths construct redis.Redis(..., ssl=..., ssl_cert_reqs=..., ssl_check_hostname=..., ssl_ca_certs=...) directly and honor INFRAHUB_CACHE_TLS_*. The Prefect result-storage block is the only Redis client in the codebase that does not.

Expected Behavior

INFRAHUB_CACHE_TLS_ENABLED, INFRAHUB_CACHE_TLS_INSECURE, and INFRAHUB_CACHE_TLS_CA_FILE should apply uniformly to every Redis client Infrahub creates — including the Prefect result-storage block — so that flows complete successfully against a TLS-only Redis.

Steps to Reproduce

  1. Provision a TLS-only Redis reachable to Infrahub (a managed TLS-only Redis, or a local stunnel/testcontainer wrapper in front of a plaintext Redis).
  2. Set on the Infrahub backend processes:
    • INFRAHUB_CACHE_ADDRESS=<tls-redis-host>
    • INFRAHUB_CACHE_TLS_ENABLED=true
    • PREFECT_RESULTS_PERSIST_BY_DEFAULT=true
  3. Start Infrahub and trigger any flow that persists results (e.g. wait for clean-up-deadlocks on its schedule, or run git_repositories_sync).
  4. Observe in Prefect: the flow's task runs complete, but the parent flow run stays in RUNNING indefinitely.

Workaround that confirms the diagnosis: patch the saved block document in Prefect's DB to use connection_string: rediss://... instead of host/port/db/username/password. All affected flows complete immediately afterwards.

Additional Information

Root cause. backend/infrahub/workflows/initialization.py::setup_blocks() builds the block via RedisStorageContainer.from_host(...). Prefect's from_host has no TLS parameter, and the internal client path used in that branch (_client()) calls aioredis.Redis(host=..., port=..., username=..., password=..., db=...) with no ssl= argument. The only TLS-capable path through RedisStorageContainer is connection_string, which is then resolved via redis.Redis.from_url(...).

Sites that already honor the setting (for parity reference):

  • backend/infrahub/lock.pyredis.Redis(..., ssl=config.SETTINGS.cache.tls_enabled, ssl_cert_reqs=..., ssl_check_hostname=..., ssl_ca_certs=...)
  • backend/infrahub/services/adapters/cache/redis.py — same pattern

Proposed fix. Always construct a connection_string URL in setup_blocks():

  • Scheme switches between redis:// and rediss:// based on tls_enabled.
  • tls_insecure=true encodes as ?ssl_cert_reqs=none&ssl_check_hostname=False query params.
  • tls_ca_file encodes as ?ssl_ca_certs=<path> query param.

redis.Redis.from_url parses these query params and applies them to the underlying SSLConnection, so all three TLS knobs become deterministic across every Redis client in the codebase — no upstream Prefect change needed and no separate "this flag doesn't apply here" warning.

PR to follow.

Metadata

Metadata

Assignees

Labels

type/bugSomething isn't working as expected

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions