Repository navigation
Commit 03bd2a3
authored
[enhancement](Nereids) Enable parse sql from sql cache (apache#33262)
Before this pr, the query must pass through parser, analyzer, rewriter, optimizer and translator, then we can check whether this query can use sql cache, if the query is too long, or the number of join tables too big, the plan time usually >= 500ms.
This pr reduce this time by skip the fashion plan path, because we can reuse the previous physical plan and query result if no any changed. In some cases we should not parse sql from sql cache, e.g. table structure changed, data changed, user policies changed, privileges changed, contains non-deterministic functions, and user variables changed.
In my test case: query a view which has lots of join and union, and the tables has empty partition, the query latency is about 3ms. if not parse sql from sql cache, the plan time is about 550ms
## Features
1. use Config.sql_cache_manage_num to control how many sql cache be reused in on fe
2. if explain plan appear some plans contains `LogicalSqlCache` or `PhysicalSqlCache`, it means the query can use sql cache, like this:
```sql
mysql> set enable_sql_cache=true;
Query OK, 0 rows affected (0.00 sec)
mysql> explain physical plan select * from test.t;
+----------------------------------------------------------------------------------+
| Explain String(Nereids Planner) |
+----------------------------------------------------------------------------------+
| cost = 3.135 |
| PhysicalResultSink[53] ( outputExprs=[c1#0, c2#1] ) |
| +--PhysicalDistribute[50]@0 ( stats=3, distributionSpec=DistributionSpecGather ) |
| +--PhysicalOlapScan[t]@0 ( stats=3 ) |
+----------------------------------------------------------------------------------+
4 rows in set (0.02 sec)
mysql> select * from test.t;
+------+------+
| c1 | c2 |
+------+------+
| 1 | 2 |
| -2 | -2 |
| NULL | 30 |
+------+------+
3 rows in set (0.05 sec)
mysql> explain physical plan select * from test.t;
+-------------------------------------------------------------------------------------------+
| Explain String(Nereids Planner) |
+-------------------------------------------------------------------------------------------+
| cost = 0.0 |
| PhysicalSqlCache[2] ( queryId=78511f515cda466b-95385d892d6c68d0, backend=127.0.0.1:9050 ) |
| +--PhysicalResultSink[52] ( outputExprs=[c1#0, c2#1] ) |
| +--PhysicalDistribute[49]@0 ( stats=3, distributionSpec=DistributionSpecGather ) |
| +--PhysicalOlapScan[t]@0 ( stats=3 ) |
+-------------------------------------------------------------------------------------------+
5 rows in set (0.01 sec)
```1 parent c5a81e9 commit 03bd2a3
54 files changed
Lines changed: 3025 additions & 427 deletions
File tree
- fe
- fe-common/src/main/java/org/apache/doris/common
- fe-core/src
- main/java/org/apache/doris
- blockrule
- catalog
- common
- profile
- nereids
- parser
- rules
- analysis
- expression/rules
- rewrite
- trees
- expressions
- functions
- scalar
- table
- udf
- literal
- plans
- algebra
- logical
- physical
- visitor
- qe
- cache
- test/java/org/apache/doris/nereids/rules/analysis
- regression-test
- framework/src/main/groovy/org/apache/doris/regression
- action
- suite
- plugins
- suites
- nereids_p0/cache
- query_p0/cache
- schema_change_p0
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
20 | | - | |
21 | | - | |
22 | 20 | | |
23 | 21 | | |
24 | 22 | | |
| |||
2004 | 2002 | | |
2005 | 2003 | | |
2006 | 2004 | | |
| 2005 | + | |
| 2006 | + | |
| 2007 | + | |
| 2008 | + | |
| 2009 | + | |
| 2010 | + | |
| 2011 | + | |
| 2012 | + | |
2007 | 2013 | | |
2008 | 2014 | | |
2009 | 2015 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
43 | 43 | | |
44 | 44 | | |
45 | 45 | | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
46 | 50 | | |
47 | 51 | | |
48 | 52 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
269 | 269 | | |
270 | 270 | | |
271 | 271 | | |
272 | | - | |
273 | | - | |
274 | | - | |
275 | | - | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
276 | 276 | | |
277 | 277 | | |
278 | 278 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
105 | 105 | | |
106 | 106 | | |
107 | 107 | | |
| 108 | + | |
108 | 109 | | |
109 | 110 | | |
110 | 111 | | |
| |||
530 | 531 | | |
531 | 532 | | |
532 | 533 | | |
| 534 | + | |
| 535 | + | |
533 | 536 | | |
534 | 537 | | |
535 | 538 | | |
| |||
766 | 769 | | |
767 | 770 | | |
768 | 771 | | |
| 772 | + | |
| 773 | + | |
| 774 | + | |
769 | 775 | | |
770 | 776 | | |
771 | 777 | | |
| |||
6092 | 6098 | | |
6093 | 6099 | | |
6094 | 6100 | | |
| 6101 | + | |
| 6102 | + | |
| 6103 | + | |
| 6104 | + | |
6095 | 6105 | | |
6096 | 6106 | | |
6097 | 6107 | | |
| |||
0 commit comments