prefer_localhost_replica has a bit of a weird behaviour which makes it's usability questionable.

• it is ignored if default database is set (

  ClickHouse/src/Interpreters/Cluster.cpp

  Lines 33 to 46 in 468089a

  	inline bool isLocalImpl(const Cluster::Address & address, const Poco::Net::SocketAddress & resolved_address, UInt16 clickhouse_port)
  	{
  	/// If there is replica, for which:
  	/// - its port is the same that the server is listening;
  	/// - its host is resolved to set of addresses, one of which is the same as one of addresses of network interfaces of the server machine*;
  	/// then we must go to this shard without any inter-process communication.
  	///
  	/// * - this criteria is somewhat approximate.
  	///
  	/// Also, replica is considered non-local, if it has default database set
  	/// (only reason is to avoid query rewrite).
  	return address.default_database.empty() && isLocalAddress(resolved_address, clickhouse_port);
  	}

  )
• second query is not logged in query_log which causes 2 minor issues:
  • metrics for both distributed query and local queries are reported together
  • query is sent with initial user limits instead of a the custom user set in cluster config which might have different limits
• there is no consistent way to check if query is reading from a distributed table or from a local table:
  • QueryProcessingStage::WithMergeableState check doesn't work when distributed_group_by_no_merge is enabled
  • ClientInfo::QueryKind::SECONDARY_QUERY doesn't work as ClientInfo::QueryKind is not updated for "local" query

Relates to #5732, where I want to set a different max_execution_speed_bytes setting for "leaf query".