Summary

Adds pool_connections and pool_maxsize configuration options to the REST emitter to allow tuning of HTTP connection pool settings. This helps prevent "Connection pool is full" warnings when running high-concurrency ingestion workloads.

Background

The hardcoded pool_maxsize=100 was originally added in October 2021 (PR #3431) to support parallel metadata writes. However, it was never made configurable. Users running with max_threads > 100 or experiencing slow requests (30+ second timeouts) could exhaust the connection pool, resulting in warnings like:

Connection pool is full, discarding connection: localhost. Connection pool size: 100

Changes

• Added pool_connections and pool_maxsize fields to DatahubClientConfig
• Added environment variables for system-wide defaults:
  • DATAHUB_REST_EMITTER_DEFAULT_POOL_CONNECTIONS (default: 100)
  • DATAHUB_REST_EMITTER_DEFAULT_POOL_MAXSIZE (default: 100)
• Made pool settings configurable in both ingestion sink and Python SDK (DataHubGraph)
• Maintains backward compatibility (defaults to 100 for both)

Configuration Options

Priority order:

1. Explicit YAML/Python config
2. Environment variables
3. Defaults (100)

Example YAML config:

sink:
  type: datahub-rest
  config:
    server: "https://your-datahub.io"
    pool_maxsize: 200  # Increase for high concurrency
    max_threads: 150

Example Python SDK:

from datahub.ingestion.graph.client import DataHubGraph, DatahubClientConfig

config = DatahubClientConfig(
    server="https://your-datahub.io",
    pool_maxsize=200
)
graph = DataHubGraph(config)

Testing

• Linting passes (./gradlew :metadata-ingestion:lintFix)
• Backward compatible (defaults match previous hardcoded values)
• Configuration flows through all layers (config → emitter → session)

Checklist

• PR conforms to DataHub's Contributing Guideline (PR title format)
• Links to related issue: Addresses connection pool exhaustion in high-concurrency scenarios
• Tests: Existing tests cover the code paths, new config fields use standard pydantic validation
• Docs: Configuration is self-documenting via docstrings and type hints

🤖 Generated with Claude Code