@Akazz can you advise what's wrong with linker which doesn't know where's attachTo?

Note that DB::CurrentThread::detachQueryIfNotDetached() cannot be found as well. I guess it is a matter of where you placed this attach/detach logics. Currently as it stands there is a dependency Interpreters -> common_io (DB::CurrentThread::attachTo() resides in ThreadStatusExt.cpp, which belongs to Interpreters), and here we'd need the reverse dependency. This would create cyclic links.

For a bigger picture I guess we'll have to look at how such thread management is done elsewhere in the code.

Due to some dependencies in the code it looks like it is better to simply ignore extra thread usage for the s3 writes (and fix this separately)

@Akazz what do you think?

@azat I was thinking the same thing - for s3 writes we can temporarily ignore these threads be unaccounted for.
@excitoon But better put a huge TODO not to forget this issue.

Why this pr stucked, we need it now :) @excitoon @azat @Akazz @alexey-milovidov

@Akazz can you please advise how to resolve dependency loop when I need to access thread pools from IO?

Performance comparison between old version and new. Let's try to upload test.hits to some s3 installation and check the query duration and speed. Also I am interested in dstat statistics.

I can't show you significant performance boost because it is about 2-3% and experiment would require a constant TCP retransmission rate in order to reliably prove it. I shall add retries to WriteBufferFromS3 (which is a good thing anyway) and show it on simple HTTP connection drops.

Turned out S3 client makes retries automatically, even without 64d7577875c61993e82c01c43a83ac96dd2ba724 .

I have no idea why I don't get good performance boost, possibly there is some bottleneck before WriteBufferFromS3, I shall check that.

Also, I would check it in Yandex.Cloud :)

Also we talked a bit with @KochetovNicolai, and decided this feature is better to be implemented through Processors (e.g parallelized with pipeline). So, you can create multiple sources or whatever.

@nikitamikhaylov Are you sure you were not talking about partitioned writes? Actually they fit processors in much more natural way.

@nikitamikhaylov This PR is accomplishable through processors as well.

I've made an experiment with downloading from S3, same region with 10% packet drop which shows that multiple threads work very well in poor network conditions.

All committers have signed the CLA.

@excitoon Let's revive this PR.

I was asked to rewrite multi-threaded code on processors here.

@excitoon Please resolve conflicts.

Continued in #33291.