Add setting `send_profile_events` to reduce network traffic by SpencerTorres · Pull Request #80761 · ClickHouse/ClickHouse

SpencerTorres · 2025-05-23T23:16:41Z

Changelog category (leave one):

New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Added send_profile_events setting which allows clients to reduce network traffic when profile events are unused.

Documentation entry for user-facing changes

Documentation is written (mandatory for new features)

For users who are doing many small async inserts over Native, it's possible that more data will be received from the server than sent by the client. The ProfileEvents packet was identified to be a large portion of this. To help reduce traffic, I have added a boolean setting called send_profile_events that defaults to 1. Clients can choose to disable profile events to reduce traffic. In the Go client for example, this data isn't used. It's parsed and immediately ignored. This can be applied per-query.

See sample Go client packet log using this new setting:

Packet log with `ProfileEvents` disabled

Settings: map[string]any{
	"send_profile_events": 0,
},

2025/05/23 19:00:28 [clickhouse][conn=1][[::1]:9000] [handshake] ->  0.0.0
2025/05/23 19:00:28 [clickhouse][conn=1][[::1]:9000] [handshake] <- ClickHouse (xxx) server version 25.6.1 revision 54477 (timezone America/New_York)
2025/05/23 19:00:28 [clickhouse][conn=1][[::1]:9000] [send query] compression="none" INSERT INTO function null('x UInt64') FORMAT Native
2025/05/23 19:00:28 [clickhouse][conn=1][[::1]:9000] [send data] compression="none"
2025/05/23 19:00:28 [clickhouse][conn=1][[::1]:9000] [read data] compression="none". block: columns=1, rows=0
2025/05/23 19:00:28 [clickhouse][conn=1][[::1]:9000] [send data] compression="none"
2025/05/23 19:00:28 [clickhouse][conn=1][[::1]:9000] [end of stream]
2025/05/23 19:00:28 [clickhouse][conn=1][[::1]:9000] [released] connection [1]
2025/05/23 19:00:28 [clickhouse][conn=1][[::1]:9000] [send query] compression="none" SELECT 1
2025/05/23 19:00:28 [clickhouse][conn=1][[::1]:9000] [send data] compression="none"
2025/05/23 19:00:28 [clickhouse][conn=1][[::1]:9000] [read data] compression="none". block: columns=1, rows=0
2025/05/23 19:00:28 [clickhouse][conn=1][[::1]:9000] [read data] compression="none". block: columns=1, rows=1
2025/05/23 19:00:28 [clickhouse][conn=1][[::1]:9000] [profile info] rows=1, bytes=4104, blocks=1, rows before limit=0, applied limit=false, calculated rows before limit=true
2025/05/23 19:00:28 [clickhouse][conn=1][[::1]:9000] [progress] rows=1, bytes=1, total rows=1, wrote rows=0 wrote bytes=0 elapsed=487.041µs
2025/05/23 19:00:28 [clickhouse][conn=1][[::1]:9000] [progress] rows=1, bytes=1, total rows=1, wrote rows=0 wrote bytes=0 elapsed=487.041µs
2025/05/23 19:00:28 [clickhouse][conn=1][[::1]:9000] [read data] compression="none". block: columns=0, rows=0
2025/05/23 19:00:28 [clickhouse][conn=1][[::1]:9000] [progress] rows=0, bytes=0, total rows=0, wrote rows=0 wrote bytes=0 elapsed=7.224µs
2025/05/23 19:00:28 [clickhouse][conn=1][[::1]:9000] [progress] rows=0, bytes=0, total rows=0, wrote rows=0 wrote bytes=0 elapsed=7.224µs
2025/05/23 19:00:28 [clickhouse][conn=1][[::1]:9000] [end of stream]
2025/05/23 19:00:28 [clickhouse][conn=1][[::1]:9000] [released] connection [1]

Packet log with `ProfileEvents` enabled (default, current behavior)

Settings: map[string]any{
	"send_profile_events": 1,
},

2025/05/23 19:01:18 [clickhouse][conn=1][[::1]:9000] [handshake] ->  0.0.0
2025/05/23 19:01:18 [clickhouse][conn=1][[::1]:9000] [handshake] <- ClickHouse (xxx) server version 25.6.1 revision 54477 (timezone America/New_York)
2025/05/23 19:01:18 [clickhouse][conn=1][[::1]:9000] [send query] compression="none" INSERT INTO function null('x UInt64') FORMAT Native
2025/05/23 19:01:18 [clickhouse][conn=1][[::1]:9000] [send data] compression="none"
2025/05/23 19:01:18 [clickhouse][conn=1][[::1]:9000] [read data] compression="none". block: columns=1, rows=0
2025/05/23 19:01:18 [clickhouse][conn=1][[::1]:9000] [send data] compression="none"
2025/05/23 19:01:18 [clickhouse][conn=1][[::1]:9000] [read data] compression="none". block: columns=6, rows=18
2025/05/23 19:01:18 [clickhouse][conn=1][[::1]:9000] [profile events] rows=18
2025/05/23 19:01:18 [clickhouse][conn=1][[::1]:9000] [end of stream]
2025/05/23 19:01:18 [clickhouse][conn=1][[::1]:9000] [released] connection [1]
2025/05/23 19:01:18 [clickhouse][conn=1][[::1]:9000] [send query] compression="none" SELECT 1
2025/05/23 19:01:18 [clickhouse][conn=1][[::1]:9000] [send data] compression="none"
2025/05/23 19:01:18 [clickhouse][conn=1][[::1]:9000] [read data] compression="none". block: columns=1, rows=0
2025/05/23 19:01:18 [clickhouse][conn=1][[::1]:9000] [read data] compression="none". block: columns=1, rows=1
2025/05/23 19:01:18 [clickhouse][conn=1][[::1]:9000] [profile info] rows=1, bytes=4104, blocks=1, rows before limit=0, applied limit=false, calculated rows before limit=true
2025/05/23 19:01:18 [clickhouse][conn=1][[::1]:9000] [progress] rows=1, bytes=1, total rows=1, wrote rows=0 wrote bytes=0 elapsed=485.818µs
2025/05/23 19:01:18 [clickhouse][conn=1][[::1]:9000] [progress] rows=1, bytes=1, total rows=1, wrote rows=0 wrote bytes=0 elapsed=485.818µs
2025/05/23 19:01:18 [clickhouse][conn=1][[::1]:9000] [read data] compression="none". block: columns=6, rows=23
2025/05/23 19:01:18 [clickhouse][conn=1][[::1]:9000] [profile events] rows=23
2025/05/23 19:01:18 [clickhouse][conn=1][[::1]:9000] [read data] compression="none". block: columns=0, rows=0
2025/05/23 19:01:18 [clickhouse][conn=1][[::1]:9000] [progress] rows=0, bytes=0, total rows=0, wrote rows=0 wrote bytes=0 elapsed=25.218µs
2025/05/23 19:01:18 [clickhouse][conn=1][[::1]:9000] [progress] rows=0, bytes=0, total rows=0, wrote rows=0 wrote bytes=0 elapsed=25.218µs
2025/05/23 19:01:18 [clickhouse][conn=1][[::1]:9000] [end of stream]
2025/05/23 19:01:18 [clickhouse][conn=1][[::1]:9000] [released] connection [1]

Note the difference, these lines are excluded since the server never sent them:

2025/05/23 19:01:18 [clickhouse][conn=1][[::1]:9000] [profile events] rows=18
2025/05/23 19:01:18 [clickhouse][conn=1][[::1]:9000] [profile events] rows=23

This change does not affect the ProfileInfo packet. Setting docs should be auto-generated, let me know if I need to manually add anything. Thanks!

clickhouse-gh · 2025-05-23T23:17:05Z

Workflow [PR], commit [c001cdb]

azat · 2025-05-24T22:24:41Z

src/Server/TCPHandler.cpp


 void TCPHandler::sendProfileEvents(QueryState & state)
 {
+    if (!state.query_context->getSettingsRef()[Setting::send_profile_events])


I was worrying about this for a long time - #28364 (comment)

We cannot simply just not send them now, since some clients relies on that they are there, i.e. python driver here - https://github.com/mymarilyn/clickhouse-driver/blob/8a4e7c5b99b532df2b015651d893a6f36288a22c/clickhouse_driver/client.py#L725-L757

And using a setting for this does not looks correct because such settings will not work if they are specified in the query directly (i.e. select 1 settings) only in protocol, and we need to clarify this to users. And in general setting that controls the protocol looks odd.

So I guess we need to add one more revision, and let client decide does it need profile events or not (in hello addendum), and only those who can handle this packets will set this flag.

@alexey-milovidov I remember you wanted to avoid the complexity, but there is already a timer for how long does it take to send them... - #62752

@novikd @alexey-milovidov @Algunenano what do you think?

My concern is that the new setting will most likely be disabled in the Cloud and non-modifiable, because ProfileEvents may be used for accounting, and the fact that they are send between servers in the distributed query execution is also important.

Also, I've recently added the compression of ProfileEvents, which makes this feature less relevant.

PS.

We cannot simply just not send them now, since some clients relies on that they are there

It is a mistake in the Python client if it relies on this specific order of packets. However, it is unlikely to be a problem - for it to happen, the user will have to explicitly disable the profile events.

because such settings will not work if they are specified in the query directly (i.e. select 1 settings) only in protocol

We should make it in a way that the server stops sending ProfileEvents as soon as it applies this setting for a query.

alexey-milovidov · 2025-05-25T01:15:06Z

@azat @SpencerTorres Let's ensure that ProfileEvents are not sent more frequently than interactive_delay (similarly to the progress packets). If the server receives ProfileEvents from many other servers, it can aggregate them for the interactive_delay period before sending to the client. This should solve the problem with no extra settings.

azat · 2025-05-25T07:32:40Z

Let's ensure that ProfileEvents are not sent more frequently than interactive_delay (similarly to the progress packets). If the server receives ProfileEvents from many other servers, it can aggregate them for the interactive_delay period before sending to the client. This should solve the problem with no extra settings.

This change will break the mentioned driver, but perhaps that's acceptable.

Regarding your suggestion - are you proposing not to send ProfileEvents at all if interactive_delay hasn’t elapsed? Or to send the first one immediately and then delay subsequent ones until the interactive_delay interval passes?

The second option might still cause issues. I'm not entirely sure what was causing the original problem - it could be related to the waiting phase during async INSERTs, where ProfileEvents might be received, if so, then this should work.

alexey-milovidov · 2025-05-25T13:17:31Z

Or to send the first one immediately and then delay subsequent ones until the interactive_delay interval passes?

Yes, the first and the last one can always be sent, and everything in between not frequently than the interactive_delay.

nikitamikhaylov · 2025-05-28T03:23:35Z

Maybe we can change the protocol itself making all the clients to explicitly say whether they want profile events back or not? We can put that in the receiveAddendum function in the TCPHander.

SpencerTorres · 2025-05-28T16:30:24Z

Maybe we can change the protocol itself making all the clients to explicitly say whether they want profile events back or not? We can put that in the receiveAddendum function in the TCPHander.

This would be useful; @azat suggested the same. If we're going to make a new revision, are there other packets we should conditionally exclude? We could do a boolean to toggle ProfileEvents, or a 1 byte bit set to determine which optional packets to receive (this would be more future-proof and save space if we ever wanted to disable other packets in the future).

I see what @alexey-milovidov said about the interactive_delay, but in this case the client doesn't use these packets at all and it would be better if they were never sent. If there's an issue with how frequently ProfileEvents are sent then we can put that in a separate issue.

@alexey-milovidov How do you feel about the suggested revision? Is it reasonable for clients to decide not to receive these packets? By disabling them we save some CPU/network resources on both client and server.

azat · 2025-05-28T17:47:15Z

If we're going to make a new revision, are there other packets we should conditionally exclude?

AFAIR - no (logs are already under a setting)

We could do a boolean to toggle ProfileEvents, or a 1 byte bit set to determine which optional packets to receive

I like the idea of using bit mask, but, we still will have to bump the revision I guess, and I don't expect too much changes here, so maybe the complexity does not worth it

andreev-io · 2025-05-29T10:33:51Z

Loosely related to #75719.

alexey-milovidov · 2025-06-01T17:39:04Z

Related to #63065

alexey-milovidov · 2025-06-24T23:36:24Z

Yes, the first and the last one can always be sent, and everything in between not frequently than the interactive_delay.

Ok, this is not applicable because metrics from different servers have different hostname fields, so they cannot be squashed without losing this info.

alexey-milovidov · 2025-06-24T23:41:25Z

I think this solution is good enough, and it is flexible enough.
It's not a problem if send_profile_events cannot be disabled entirely in the SETTINGS clause.
It's not a problem if some clients break if sending profile events is disabled on the profile level.

clickhouse-gh · 2025-06-24T23:42:14Z

Workflow [PR], commit [358c3a8]

Summary: ❌
15 failures out of 109 shown:

job_name	test_name	status
Docs check		dropped
Fast test		failure
	02995_new_settings_history	FAIL
Build (amd_debug)		dropped
Build (amd_release)		dropped
Build (amd_asan)		dropped
Build (amd_tsan)		dropped
Build (amd_msan)		dropped
Build (amd_ubsan)		dropped
Build (amd_binary)		dropped
Build (arm_release)		dropped
Build (arm_asan)		dropped
Build (arm_coverage)		dropped
Build (arm_binary)		dropped
Build (amd_darwin)		dropped
Build (arm_darwin)		dropped

alexey-milovidov

And everything looks good.

SpencerTorres · 2025-06-24T23:53:44Z

No need to rush this change, we can go with the other solution of adding a revision with a boolean in the client Hello. I updated my local code to do this, but I need to update the related tests/docs. I can close this PR and submit the other when its ready.

I want to make sure we enable this ability the proper way, if a setting isn't the right approach then let's wait to add it as a revision to the Hello packet. Let me know what the preferred approach is

alexey-milovidov · 2025-06-25T00:13:51Z

Ok.

clickhouse-gh · 2025-07-29T13:22:42Z

Dear @azat, @alexey-milovidov, this PR hasn't been updated for a while. You will be unassigned. Will you continue working on it? If so, please feel free to reassign yourself.

clickhouse-gh · 2025-10-21T13:21:20Z

Dear @azat, this PR hasn't been updated for a while. You will be unassigned. Will you continue working on it? If so, please feel free to reassign yourself.

erezrokah · 2025-10-29T10:35:47Z

Related to ClickHouse/clickhouse-go#1685 and ClickHouse/clickhouse-go#1686. Skipping the consumption of profile events on the client side can reduce memory allocations by over 3x.

It would be great to be able to disable these at the session level. This is especially important for async inserts where profile events take up a bigger chunk of the data that is streamed from the server.

alexey-milovidov · 2025-11-05T18:03:47Z

See #89588.

add setting send_profile_events

c001cdb

clickhouse-gh bot added the pr-feature Pull request with new product feature label May 23, 2025

azat self-assigned this May 24, 2025

azat reviewed May 24, 2025

View reviewed changes

azat mentioned this pull request Jun 24, 2025

When working with a large cluster, client uses too much network for metrics information #63065

Closed

Merge branch 'master' into setting_send_profile_events

358c3a8

alexey-milovidov approved these changes Jun 24, 2025

View reviewed changes

alexey-milovidov self-assigned this Jun 24, 2025

clickhouse-gh bot unassigned azat and alexey-milovidov Jul 29, 2025

azat self-assigned this Sep 17, 2025

clickhouse-gh bot unassigned azat Oct 21, 2025

erezrokah mentioned this pull request Oct 29, 2025

fix: Don't scan profile events if listener is not set ClickHouse/clickhouse-go#1686

Merged

2 tasks

kavirajk mentioned this pull request Nov 5, 2025

Support send_profile_events settings to reduce network flow #89588

Merged

1 task

alexey-milovidov closed this Nov 5, 2025

kavirajk mentioned this pull request Nov 13, 2025

Support send_profile_events settings to reduce network flow #89972

Merged

1 task

kavirajk self-assigned this Nov 25, 2025

Conversation

SpencerTorres commented May 23, 2025

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Documentation entry for user-facing changes

Packet log with ProfileEvents disabled

Packet log with ProfileEvents enabled (default, current behavior)

Uh oh!

clickhouse-gh bot commented May 23, 2025

Uh oh!

azat May 24, 2025

Choose a reason for hiding this comment

Uh oh!

alexey-milovidov Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

alexey-milovidov commented May 25, 2025

Uh oh!

azat commented May 25, 2025

Uh oh!

alexey-milovidov commented May 25, 2025

Uh oh!

nikitamikhaylov commented May 28, 2025

Uh oh!

SpencerTorres commented May 28, 2025

Uh oh!

azat commented May 28, 2025

Uh oh!

andreev-io commented May 29, 2025

Uh oh!

alexey-milovidov commented Jun 1, 2025

Uh oh!

alexey-milovidov commented Jun 24, 2025

Uh oh!

alexey-milovidov commented Jun 24, 2025

Uh oh!

clickhouse-gh bot commented Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alexey-milovidov left a comment

Choose a reason for hiding this comment

Uh oh!

SpencerTorres commented Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alexey-milovidov commented Jun 25, 2025

Uh oh!

clickhouse-gh bot commented Jul 29, 2025

Uh oh!

clickhouse-gh bot commented Oct 21, 2025

Uh oh!

erezrokah commented Oct 29, 2025

Uh oh!

alexey-milovidov commented Nov 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Packet log with `ProfileEvents` disabled

Packet log with `ProfileEvents` enabled (default, current behavior)

clickhouse-gh bot commented Jun 24, 2025 •

edited

Loading

SpencerTorres commented Jun 24, 2025 •

edited

Loading