Skip to content

Optimizations of ranges selection while reading with enabled optimize_read_in_order.#6299

Merged
alexey-milovidov merged 14 commits intoClickHouse:masterfrom
CurtizJ:order-by-efficient
Aug 15, 2019
Merged

Optimizations of ranges selection while reading with enabled optimize_read_in_order.#6299
alexey-milovidov merged 14 commits intoClickHouse:masterfrom
CurtizJ:order-by-efficient

Conversation

@CurtizJ
Copy link
Copy Markdown
Member

@CurtizJ CurtizJ commented Aug 2, 2019

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

For changelog. Remove if this is non-significant change.

Category (leave one):

  • Performance Improvement

Short description (up to few sentences):

Now less data will be read with enabled optimize_read_in_order and small limit.
Part of #6054. Do not include to changelog.

@alexey-milovidov
Copy link
Copy Markdown
Member

TSan report looks relevant.

@CurtizJ CurtizJ force-pushed the order-by-efficient branch from e4827a4 to cfe32ad Compare August 12, 2019 23:15
@CurtizJ CurtizJ changed the title Optimizations of ranges selection while reading with enabled optimize_read_in_order. [WIP] Optimizations of ranges selection while reading with enabled optimize_read_in_order. Aug 13, 2019
@CurtizJ CurtizJ force-pushed the order-by-efficient branch from 9094cc7 to 16ddc6b Compare August 13, 2019 13:52
@CurtizJ CurtizJ force-pushed the order-by-efficient branch from 16ddc6b to 7ae73be Compare August 13, 2019 14:07
@CurtizJ CurtizJ changed the title [WIP] Optimizations of ranges selection while reading with enabled optimize_read_in_order. Optimizations of ranges selection while reading with enabled optimize_read_in_order. Aug 13, 2019
@CurtizJ CurtizJ changed the title Optimizations of ranges selection while reading with enabled optimize_read_in_order. [WIP] Optimizations of ranges selection while reading with enabled optimize_read_in_order. Aug 13, 2019
@CurtizJ
Copy link
Copy Markdown
Member Author

CurtizJ commented Aug 14, 2019

I disabled read in order optimization with joins as it works incorrectly now, because after join UnionBlockInputStream is called.

@CurtizJ CurtizJ changed the title [WIP] Optimizations of ranges selection while reading with enabled optimize_read_in_order. Optimizations of ranges selection while reading with enabled optimize_read_in_order. Aug 15, 2019
@alexey-milovidov alexey-milovidov merged commit a466c7a into ClickHouse:master Aug 15, 2019
@MahmoudGoda0
Copy link
Copy Markdown

Hi Guys,
I think this is related to "Optimization of ORDER BY with respect to the ORDER key in MergeTree tables", I installed the new stable version (19.13.2.19) today, and i see very bad results comparing with version "19.13.1.961" for the below query:

select TOP 1000 ext_rec_num,xdr_id,xdr_grp,xdr_type,xdr_subtype,xdr_direction,xdr_location, start_time,stop_time, transaction_duration,response_time,protocol,chunk_count,dpc, opc,first_link_id,last_dpc,last_opc,last_link_id,first_back_opc, first_back_link_id,calling_ssn,called_ssn,called_sccp_address, calling_party_address,response_calling_address,root_end_code, root_cause_code,root_cause_pl,root_failure,root_equip, map_end_code,map_cause_code,map_cause_pl,sip_end_code, sip_cause_code,sip_cause_pl,isup_end_code,isup_cause_code, isup_cause_pl,inap_end_code,inap_cause_code,inap_cause_pl, subs_id,mccmnc,imsi,msisdn,tac,imei,msc_number,vlr_number, hlr_number,service_key_1,camel_phases,camel_cpbt_handling, gsm_scf_address,location_number,old_lac,mcc,mnc,lac, cell_id,sms_length,service_center_address,calling_number, dialled_number,connect_timestamp,setup_time, conversation_time,cic,call_connected,release, forwarded,a_nature,a_num_plan,b_nature,b_num_plan, c_number,c_nature,c_type,d_number,d_type,e_number,e_type, teleservice,leg1_1_event,leg1_2_event,leg1_3_event, leg2_1_event,leg2_2_event,leg2_3_event,leg3_1_event, leg3_2_event,leg3_3_event,sgsn_number,sgsn_address,apn_1, apn_2,apn_3,apn_4,apn_5,call_id,calling_ip_address, calling_port,calling_id_user,calling_id_host, calling_id_name,calling_realm,calling_user_agent, original_called_id_user,original_called_id_host, original_called_id_name,called_id_user,called_id_host, called_id_name,request_uri,from_uri,to_uri,media_format from ss7_table where start_time > 971128806382 and start_time < 971172006000 and msisdn = '33610379835' ORDER BY start_time DESC

@alexey-milovidov
Copy link
Copy Markdown
Member

No, it should not be included in any stable version yet.

@alexey-milovidov
Copy link
Copy Markdown
Member

Your testing version 19.13.1.961 had the optimization for ORDER BY, but it wasn't released in any stable version. After you installed 19.13.2.19-stable, you get lost this optimization.

Resolution: install the latest testing from https://repo.yandex.ru/clickhouse/deb/testing/main/
It is v19.13.1.1088-testing.

PS. Actually it should be named 19.14 instead.

@MahmoudGoda0
Copy link
Copy Markdown

Thank you very much!
2 questions please:
1- Now version 19.13.1.1088-testing including both requests "5042" & "6299" ? I'm asking this question as i see the enhancement done in "5042" effected negative when i used 1088-testing.
2- How I know that a specific version has been released to the stable version ?

@MahmoudGoda0
Copy link
Copy Markdown

@alexey-milovidov I really appreciate if you answer my above 2 questions.

@CurtizJ CurtizJ added pr-improvement Pull request with some product improvements pr-performance Pull request with some performance improvements labels Sep 6, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-improvement Pull request with some product improvements pr-performance Pull request with some performance improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants