Skip to content

refactor: introduce shared NetworkPool to manage and reuse HTTP transports across downloaders#410

Merged
SuperCoolPencil merged 2 commits intomainfrom
hndwtn-unified-worker-pool
Apr 24, 2026
Merged

refactor: introduce shared NetworkPool to manage and reuse HTTP transports across downloaders#410
SuperCoolPencil merged 2 commits intomainfrom
hndwtn-unified-worker-pool

Conversation

@SuperCoolPencil
Copy link
Copy Markdown
Member

@SuperCoolPencil SuperCoolPencil commented Apr 23, 2026

Greptile Summary

This PR replaces the per-downloader transport construction (and the old per-config sync.Map cache in the single downloader) with a centralized NetworkPool that ref-counts leases and evicts idle transports after 10 seconds. Bootstrap and download clients in the concurrent downloader now share the same pool entry (both use PoolMaxConnsPerHost), resolving the previous key-mismatch concern. The probe layer drops its LRU client cache and acquires from the same pool per call.

Confidence Score: 5/5

Safe to merge; the one remaining finding is a minor test consistency issue that does not affect correctness.

All previously raised P0/P1 concerns (IPv4 preference, test race, bootstrap/download key mismatch) are addressed or intentionally resolved. The only new finding is a P2 pool-key inconsistency in the prewarm test that doesn't affect production behavior.

internal/engine/concurrent/prewarm_reuse_test.go — pool key should match the one used in Download().

Important Files Changed

Filename Overview
internal/engine/network.go New file — implements NetworkPool with ref-counted leases, idle eviction via time.AfterFunc, and a timerGen guard to avoid double-eviction. Logic is correct and thread-safe.
internal/engine/network_test.go New tests cover reuse, idle cleanup, and config isolation; TestNetworkPool_IdleCleanup reads lease fields without the pool mutex (noted in previous threads).
internal/engine/concurrent/downloader.go Removes newConcurrentClient; both bootstrap and download clients now share the same pool transport (PoolMaxConnsPerHost), fixing the prior key mismatch concern.
internal/engine/concurrent/prewarm_reuse_test.go Updated to use NetworkPool directly; acquires transport with GetMaxConnectionsPerHost() (64) while Download() uses PoolMaxConnsPerHost (512) — different pool key, so the test exercises a different transport config than production.
internal/engine/single/downloader.go Removes the per-instance singleTransportCache; transport is now acquired from the shared NetworkPool per Download() call with a deferred release.
internal/processing/probe.go Replaces the LRU probe client cache with a per-call NetworkPool acquire/release; CheckRedirect logic is preserved verbatim.
internal/engine/types/config.go Adds PoolMaxIdleConns (512), PoolMaxIdleConnsPerHost (128), and PoolMaxConnsPerHost (512) constants for the new shared pool.
internal/engine/single/downloader_test.go Transport reuse and isolation tests updated to acquire directly from NetworkPool; assertions are equivalent to the removed test.
Prompt To Fix All With AI
This is a comment left during a code review.
Path: internal/engine/concurrent/prewarm_reuse_test.go
Line: 27-29

Comment:
**Pool key mismatch between test and production code**

`AcquireTransport` is called here with `runtime.GetMaxConnectionsPerHost()` (64 by default), but `Download()` always passes `types.PoolMaxConnsPerHost` (512). These resolve to different `poolKey` entries, so the transport acquired in the test has `MaxConnsPerHost = 64` while the one used in production has `MaxConnsPerHost = 512`. The prewarm→reuse assertion still passes because both use the same client object, but the test is no longer exercising the transport configuration that actually runs in production. Consider passing `types.PoolMaxConnsPerHost` here to stay in sync with `Download()`.

How can I resolve this? If you propose a fix, please make it concise.

Reviews (5): Last reviewed commit: "refactor: standardize network transport ..." | Re-trigger Greptile

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Apr 23, 2026

Binary Size Analysis

⚠️ Size Increased

Version Human Readable Raw Bytes
Main 19.21 MB 20144420
PR 19.22 MB 20148516
Difference 4.00 KB 4096

Comment thread internal/processing/probe.go Outdated
Comment thread internal/engine/network_test.go
Comment thread internal/engine/concurrent/downloader.go Outdated
@SuperCoolPencil SuperCoolPencil force-pushed the hndwtn-unified-worker-pool branch from f0192b1 to dc48652 Compare April 23, 2026 14:20
@SuperCoolPencil SuperCoolPencil force-pushed the hndwtn-unified-worker-pool branch from c45df60 to 0aa6f73 Compare April 23, 2026 18:19
@SuperCoolPencil SuperCoolPencil merged commit 418fcdf into main Apr 24, 2026
16 checks passed
@SuperCoolPencil SuperCoolPencil deleted the hndwtn-unified-worker-pool branch April 24, 2026 14:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature Request: Worker Pool Reuse and Persistent Connection Management

1 participant