-
Notifications
You must be signed in to change notification settings - Fork 8.3k
S3 http_keep_alive_timeout, use ConnectionPool #50217
Copy link
Copy link
Closed
Labels
Description
Use http_keep_alive_timeout and ConnectionPool for S3
Almost done here #49732
Well. After #49732 we found several points to grow. PoolBase does not meet the goals. We need mode specific ConnectionPool with particular features.
- spread connections over all the resolved addresses (addressed in HTTP connections pool #58845)
- use resolved addresses from different resolves with ttl in case when resolved each time only one different address (addressed in HTTP connections pool #58845)
- do not return a stale connection (keep_alive_timeout has passed), find the fresh one (addressed in HTTP connections pool #58845)
- do not iterate through all connections when you need one, different containers for vacant and in-use. (addressed in HTTP connections pool #58845)
- track all connections, even those which is under soft limit, they are useful too and can be reused (we see cases, when it is above the limits, sessions from prefetches are not reused in the following reads) (addressed in HTTP connections pool #58845)
- when ConnectionPool has more connections than a soft limit, evict free connection faster than keep_alive_timeout but still leave a chances to reuse them (addressed in HTTP connections pool #58845)
- the code which do reconnect for connection has to be altered in order to return one connection (if it is broken than it is just destroyed) and take already open one from the ConnectionPool (addressed in HTTP connections pool #58845)
ReadBufferFromS3PreservedSessionsis broken, it shows only explicit reset of session. (addressed in HTTP connections pool #58845)- sessions are only reused when they are marked with
HTTPSessionReuseTag. Only few pieces of code do that. For example,fileExistscloses session after. (addressed in HTTP connections pool #58845) - limits work for the groups of connections: disks, storages, http(the rest). (addressed in HTTP connections pool #58845)
- proactive create some count session, but for some time after last usage. Forgotten instances should not do anything eventually.
Reactions are currently unavailable