Skip to content

[opt](nereids) optimize small sql #43546

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 26, 2025
Merged

Conversation

924060929
Copy link
Contributor

@924060929 924060929 commented Nov 10, 2024

What problem does this PR solve?

Problem Summary:

this pr improve QPS for small sql, by this points:

  1. try best use Visitor rewriter to traverse plan, instead of Stack rewriter
  2. fast skip traverse plan tree, if the tree not contains the specify type. for example, if not contains LogicalApply, we will not traverse for the rules of Subquery unnesting topic
  3. use RoaringBitmap to check whether the slot is exist, instead of Set<Slot>
  4. skip prune column for logicalFilter(logicalOlapScan), or else need more rules to optimize it. for example, logicalProject(logicalFilter(localcalOlapScan)) → ColumnPruning → logicalProject(logicalFilter(logicalProject(localcalOlapScan))) → PushDownFilterThroughProject → logicalProject(logicalProject(logicalFilter(localcalOlapScan))) → MergeProjects → logicalProject(logicalFilter(localcalOlapScan)))
  5. use for loop instead of collection's stream api
  6. faster process string, like format, split
  7. try to process project in one rule as possible, for example, project(project(oneRowRelatoin)) → oneRowRelation
  8. fast compute some hashCode, e.g. LogicalOlapScan only use relationId as hash code
  9. reuse ConnectContext to generate next exprId without invoke ConnectContext.get() multiple times
  10. don't find table multiple times for compute lots of ColumnStatistic which have same table

this sql can optimize from 21000 QPS to 32000 QPS

explain
select date
from test.tbl

this sql can optimize from 16000 QPS to 23000 QPS

explain
select col1, col2, ..., col16
from test.tbl2
where col1 = 2 and col2 = 'xxx'
  and col3 >= 1721480971 and col3 <= 1721507345
  and col4>=20240720 and col4<=20240721 ;

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@924060929 924060929 marked this pull request as draft November 10, 2024 15:52
@924060929
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.91% (9868/26027)
Line Coverage: 29.09% (82265/282774)
Region Coverage: 28.25% (42355/149928)
Branch Coverage: 24.80% (21445/86488)
Coverage Report: http://coverage.selectdb-in.cc/coverage/33b99da88218451608a4dbb50653bcf6b4564ea3_33b99da88218451608a4dbb50653bcf6b4564ea3/report/index.html

@924060929
Copy link
Contributor Author

run buildal

@924060929
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.91% (9868/26028)
Line Coverage: 29.10% (82299/282785)
Region Coverage: 28.25% (42355/149939)
Branch Coverage: 24.79% (21446/86494)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f45db3a4af7fa53fc12d5dce3ca228928832d91d_f45db3a4af7fa53fc12d5dce3ca228928832d91d/report/index.html

@924060929
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@924060929
Copy link
Contributor Author

run buildall

@924060929
Copy link
Contributor Author

run buildall

4 similar comments
@924060929
Copy link
Contributor Author

run buildall

@924060929
Copy link
Contributor Author

run buildall

@924060929
Copy link
Contributor Author

run buildall

@924060929
Copy link
Contributor Author

run buildall

@924060929
Copy link
Contributor Author

run buildall

1 similar comment
@924060929
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37880 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6bbc5132fa6e0fa14b2c923bed15da6eee739246, data reload: false

------ Round 1 ----------------------------------
region	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:27:24	NULL	utf-8	NULL	NULL	
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	23838	3014	3061	3014
q2	2060	392	294	294
q3	10154	2732	2250	2250
q4	10282	1746	1349	1349
q5	8122	4531	4681	4531
q6	187	163	129	129
q7	q8	9413	2563	2427	2427
q9	q10	6901	3176	2851	2851
q11	826	699	673	673
q12	693	735	573	573
q13	17775	3701	3050	3050
q14	265	282	255	255
q15	543	516	487	487
q16	694	703	666	666
q17	1724	1594	1252	1252
q18	7521	7145	7133	7133
q19	1179	826	1037	826
q20	q21	5890	5267	5109	5109
q22	1035	1013	1011	1011
Total cold run time: 109102 ms
Total hot run time: 37880 ms

----- Round 2, with runtime_filter_mode=off -----
region	Doris	NULL	NULL	5	240	1201	NULL	147	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:27:24	NULL	utf-8	NULL	NULL	
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	3922	3028	3035	3028
q2	239	313	225	225
q3	2189	2651	2291	2291
q4	1439	1781	1402	1402
q5	4222	4140	4110	4110
q6	208	156	117	117
q7	q8	2323	2259	2324	2259
q9	q10	3092	3202	2848	2848
q11	768	697	669	669
q12	641	717	572	572
q13	3385	3675	3071	3071
q14	265	298	252	252
q15	521	477	476	476
q16	672	685	648	648
q17	1085	1546	1296	1296
q18	7336	7088	7072	7072
q19	815	843	902	843
q20	q21	5699	5226	5069	5069
q22	1056	1015	943	943
Total cold run time: 39877 ms
Total hot run time: 37191 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 133675 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6bbc5132fa6e0fa14b2c923bed15da6eee739246, data reload: false

============================================
show table status;
Name	Engine	Version	Row_format	Rows	Avg_row_length	Data_length	Max_data_length	Index_length	Data_free	Auto_increment	Create_time	Update_time	Check_time	Collation	Checksum	Create_options	Comment
catalog_sales	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:57	2023-12-26 22:55:02	NULL	utf-8	NULL	NULL	
item	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:58	2023-12-26 22:44:05	NULL	utf-8	NULL	NULL	
date_dim	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:56	2023-12-26 22:44:01	NULL	utf-8	NULL	NULL	
call_center	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:57	2023-12-26 22:44:01	NULL	utf-8	NULL	NULL	
web_sales	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:58	2023-12-26 22:53:25	NULL	utf-8	NULL	NULL	
inventory	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:57	2023-12-26 22:50:32	NULL	utf-8	NULL	NULL	
store	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:59	2023-12-26 22:44:01	NULL	utf-8	NULL	NULL	
catalog_returns	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:57	2023-12-26 22:46:25	NULL	utf-8	NULL	NULL	
time_dim	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:59	2023-12-26 22:44:03	NULL	utf-8	NULL	NULL	
household_demographics	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:58	2023-12-26 22:44:01	NULL	utf-8	NULL	NULL	
store_sales	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:59	2023-12-26 22:59:20	NULL	utf-8	NULL	NULL	
ship_mode	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:44:00	2023-12-26 22:44:05	NULL	utf-8	NULL	NULL	
customer_address	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:58	2023-12-26 22:44:41	NULL	utf-8	NULL	NULL	
customer	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:44:00	2023-12-26 22:44:53	NULL	utf-8	NULL	NULL	
web_returns	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:58	2023-12-26 22:45:43	NULL	utf-8	NULL	NULL	
warehouse	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:56	2023-12-26 22:44:01	NULL	utf-8	NULL	NULL	
web_site	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:58	2023-12-26 22:44:01	NULL	utf-8	NULL	NULL	
customer_demographics	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:56	2023-12-26 22:44:34	NULL	utf-8	NULL	NULL	
promotion	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:58	2023-12-26 22:44:01	NULL	utf-8	NULL	NULL	
web_page	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:59	2023-12-26 22:44:01	NULL	utf-8	NULL	NULL	
store_returns	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:59	2023-12-26 22:47:09	NULL	utf-8	NULL	NULL	
income_band	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:58	2023-12-26 22:44:01	NULL	utf-8	NULL	NULL	
dbgen_version	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:44:00	2023-12-26 22:44:01	NULL	utf-8	NULL	NULL	
catalog_page	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:58	2023-12-26 22:44:01	NULL	utf-8	NULL	NULL	
reason	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:56	2023-12-26 22:44:01	NULL	utf-8	NULL	NULL	
============================================
query1	query2	query3	9749	765	711	711
query4	query5	query6	5425	1206	1209	1206
query7	15826	2092	1821	1821
query8	4921	1071	883	883
query9	5074	2502	2485	2485
query10	6981	1528	1131	1131
query11	query12	2028	306	266	266
query13	7932	1378	1334	1334
query14	query15	4510	928	681	681
query16	12744	2943	2689	2689
query17	5842	3145	3081	3081
query18	4321	1596	1304	1304
query19	1510	1584	1278	1278
query20	2116	528	421	421
query21	4350	1815	1385	1385
query22	4842	4641	4784	4641
query23	query24	query25	2913	2870	2869	2869
query26	1088	1244	902	902
query27	2093	2077	2153	2077
query28	2492	2294	2216	2216
query29	2657	3109	2692	2692
query30	query31	query32	948	401	358	358
query33	query34	2250	2083	1800	1800
query35	1616	1818	1591	1591
query36	1716	1915	1610	1610
query37	722	1056	669	669
query38	4760	5279	4743	4743
query39	query40	762	919	734	734
query41	51	47	44	44
query42	712	874	699	699
query43	982	1132	961	961
query44	1152	778	786	778
query45	648	627	514	514
query46	2666	3233	2769	2769
query47	query48	1130	1200	1006	1006
query49	12876	1959	1885	1885
query50	1833	1909	1680	1680
query51	4874	4917	4799	4799
query52	740	782	722	722
query53	1045	1053	995	995
query54	query55	696	686	678	678
query56	query57	query58	1354	1445	1293	1293
query59	query60	query61	314	302	267	267
query62	1341	780	661	661
query63	994	1195	996	996
query64	query65	5643	5660	5596	5596
query66	query67	16521	17053	16038	16038
query68	19259	3031	2519	2519
query69	query70	3313	1912	1886	1886
query71	8784	3736	3216	3216
query72	6916	6728	6759	6728
query73	1466	1942	1374	1374
query74	query75	query76	904	1143	715	715
query77	query78	query79	2633	2915	2495	2495
query80	query81	query82	869	1219	807	807
query83	242	279	240	240
query84	460	170	151	151
query85	942	615	384	384
query86	417	420	407	407
query87	5006	5517	5053	5053
query88	7707	7635	7411	7411
query89	2712	1423	1221	1221
query90	730	546	554	546
query91	395	269	237	237
query92	418	266	208	208
query93	query94	1427	1343	1178	1178
query95	query96	2298	1190	887	887
query97	3952	4516	3998	3998
query98	838	874	788	788
query99	2314	1509	1267	1267
Total cold run time: 251762 ms
Total hot run time: 133675 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.24 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6bbc5132fa6e0fa14b2c923bed15da6eee739246, data reload: false

query1	0.04	0.04	0.03
query2	0.13	0.11	0.10
query3	0.24	0.19	0.19
query4	1.59	0.18	0.20
query5	0.55	0.53	0.55
query6	1.16	0.72	0.73
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.60	0.51	0.54
query10	0.56	0.61	0.57
query11	0.16	0.11	0.10
query12	0.15	0.11	0.11
query13	0.61	0.61	0.59
query14	2.80	2.69	2.81
query15	0.92	0.86	0.83
query16	0.38	0.37	0.39
query17	1.01	1.02	1.04
query18	1.05	1.03	1.02
query19	1.92	1.75	1.81
query20	0.01	0.01	0.02
query21	15.35	0.93	0.54
query22	0.74	1.15	0.66
query23	15.01	1.42	0.66
query24	16.32	1.51	0.56
query25	2.13	0.33	0.17
query26	0.16	0.16	0.16
query27	0.17	0.17	0.18
query28	1.63	1.07	0.92
query29	12.58	3.90	3.24
query30	0.24	0.09	0.06
query31	2.89	0.61	0.39
query32	3.23	0.55	0.47
query33	3.06	3.05	3.14
query34	15.74	5.10	4.53
query35	4.56	4.52	4.51
query36	0.66	0.49	0.47
query37	0.07	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.08	0.03	0.02
query42	0.03	0.02	0.03
query43	0.03	0.03	0.03
Total cold run time: 108.86 s
Total hot run time: 32.24 s

@924060929
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 34016 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ad1364ca4eb4ea44a921fd093e38018448fd74d8, data reload: false

------ Round 1 ----------------------------------
q1	26238	5098	5075	5075
q2	2076	287	184	184
q3	10501	1285	697	697
q4	10278	1002	528	528
q5	8536	2463	2283	2283
q6	193	163	131	131
q7	917	748	579	579
q8	9298	1285	1052	1052
q9	6857	5176	5130	5130
q10	6808	2297	1881	1881
q11	455	277	258	258
q12	335	352	216	216
q13	17774	3714	3052	3052
q14	220	221	206	206
q15	527	475	483	475
q16	690	630	591	591
q17	588	844	365	365
q18	8000	7262	7079	7079
q19	1234	955	568	568
q20	325	325	220	220
q21	4262	3423	2474	2474
q22	1081	1037	972	972
Total cold run time: 117193 ms
Total hot run time: 34016 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5131	5124	5103	5103
q2	240	317	219	219
q3	2146	2685	2296	2296
q4	1454	1881	1532	1532
q5	4608	4470	4378	4378
q6	218	164	122	122
q7	1957	1918	1750	1750
q8	2529	2532	2492	2492
q9	7174	7176	7257	7176
q10	2997	3219	2755	2755
q11	575	518	502	502
q12	664	740	595	595
q13	3495	3911	3274	3274
q14	274	291	266	266
q15	509	477	469	469
q16	647	719	677	677
q17	1161	1551	1363	1363
q18	7745	7560	7464	7464
q19	799	815	783	783
q20	1985	2007	1806	1806
q21	5218	4759	4659	4659
q22	1085	1045	987	987
Total cold run time: 52611 ms
Total hot run time: 50668 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184588 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 01870400aa7ee8c6b3bb7b562e96c5e5970a5526, data reload: false

query1	1020	405	379	379
query2	6522	1648	1638	1638
query3	6746	216	217	216
query4	26354	23398	22938	22938
query5	4322	579	422	422
query6	302	212	191	191
query7	4623	480	285	285
query8	260	213	208	208
query9	8613	2637	2652	2637
query10	481	321	267	267
query11	15831	14966	14775	14775
query12	155	110	106	106
query13	1655	535	429	429
query14	9284	5889	6024	5889
query15	207	186	173	173
query16	7338	618	495	495
query17	1197	708	594	594
query18	1990	412	314	314
query19	210	199	161	161
query20	124	115	114	114
query21	215	124	110	110
query22	4056	4223	4069	4069
query23	34240	33229	33076	33076
query24	8432	2335	2340	2335
query25	520	481	391	391
query26	1233	270	150	150
query27	2751	510	342	342
query28	4271	2112	2091	2091
query29	759	560	430	430
query30	284	216	185	185
query31	921	851	734	734
query32	72	61	56	56
query33	553	370	296	296
query34	786	836	514	514
query35	782	805	754	754
query36	933	974	877	877
query37	113	99	74	74
query38	4055	4077	4007	4007
query39	1488	1425	1406	1406
query40	212	117	104	104
query41	60	71	55	55
query42	122	110	105	105
query43	488	500	474	474
query44	1312	812	826	812
query45	178	169	165	165
query46	833	1003	613	613
query47	1756	1808	1716	1716
query48	395	417	297	297
query49	724	472	394	394
query50	633	697	405	405
query51	4072	4137	4089	4089
query52	108	104	97	97
query53	241	251	177	177
query54	570	566	490	490
query55	81	83	80	80
query56	288	296	295	295
query57	1191	1195	1122	1122
query58	255	249	254	249
query59	2519	2615	2548	2548
query60	323	315	293	293
query61	124	121	116	116
query62	781	719	656	656
query63	218	183	182	182
query64	4357	1018	650	650
query65	4298	4185	4193	4185
query66	1143	417	359	359
query67	15689	15641	15337	15337
query68	8618	889	525	525
query69	485	298	269	269
query70	1210	1150	1090	1090
query71	448	322	305	305
query72	5635	4723	4791	4723
query73	714	626	349	349
query74	9167	9022	8953	8953
query75	4074	3162	2698	2698
query76	3709	1138	718	718
query77	788	372	285	285
query78	10662	10189	9301	9301
query79	2275	827	593	593
query80	620	497	440	440
query81	466	257	224	224
query82	415	122	94	94
query83	282	254	228	228
query84	281	105	85	85
query85	801	358	391	358
query86	333	314	290	290
query87	4398	4340	4364	4340
query88	3002	2271	2283	2271
query89	448	320	282	282
query90	1946	211	204	204
query91	138	141	110	110
query92	71	57	52	52
query93	1349	971	594	594
query94	660	379	316	316
query95	361	281	279	279
query96	492	572	279	279
query97	2757	2748	2649	2649
query98	230	204	212	204
query99	1436	1391	1287	1287
Total cold run time: 274940 ms
Total hot run time: 184588 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.16 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 01870400aa7ee8c6b3bb7b562e96c5e5970a5526, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.24	0.08	0.07
query4	1.60	0.11	0.11
query5	0.44	0.43	0.41
query6	1.16	0.65	0.67
query7	0.02	0.02	0.02
query8	0.04	0.04	0.04
query9	0.60	0.53	0.53
query10	0.56	0.56	0.55
query11	0.16	0.11	0.12
query12	0.15	0.12	0.12
query13	0.63	0.62	0.61
query14	0.80	0.82	0.88
query15	0.91	0.90	0.87
query16	0.39	0.40	0.39
query17	1.11	1.09	1.06
query18	0.22	0.21	0.21
query19	1.98	1.83	1.81
query20	0.02	0.01	0.01
query21	15.40	0.87	0.54
query22	0.75	1.15	0.64
query23	15.02	1.42	0.62
query24	9.23	0.67	0.38
query25	0.42	0.18	0.06
query26	0.64	0.18	0.15
query27	0.07	0.05	0.05
query28	8.67	0.90	0.45
query29	12.57	4.06	3.35
query30	0.26	0.10	0.06
query31	2.83	0.63	0.39
query32	3.23	0.57	0.48
query33	3.04	3.05	3.16
query34	16.18	5.43	4.80
query35	4.84	4.91	4.87
query36	0.69	0.50	0.49
query37	0.09	0.07	0.06
query38	0.07	0.03	0.03
query39	0.03	0.02	0.02
query40	0.17	0.15	0.14
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.05	0.03	0.03
Total cold run time: 105.51 s
Total hot run time: 29.16 s

@924060929
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33950 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0d589fdbc3a10f1b4933a121399e997a94f9036b, data reload: false

------ Round 1 ----------------------------------
q1	17575	5121	5185	5121
q2	1939	280	185	185
q3	10338	1389	692	692
q4	10228	1031	499	499
q5	7529	2310	2347	2310
q6	173	160	124	124
q7	891	760	597	597
q8	9313	1259	1102	1102
q9	6815	5072	5172	5072
q10	6891	2357	1946	1946
q11	469	285	275	275
q12	344	340	206	206
q13	17764	3718	3100	3100
q14	229	225	210	210
q15	560	473	480	473
q16	419	444	387	387
q17	631	840	386	386
q18	7522	7234	7183	7183
q19	1289	943	538	538
q20	329	347	227	227
q21	3716	3197	2353	2353
q22	1076	1041	964	964
Total cold run time: 106040 ms
Total hot run time: 33950 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5082	5060	5048	5048
q2	237	317	224	224
q3	2195	2666	2312	2312
q4	1369	1772	1355	1355
q5	4210	4110	4272	4110
q6	210	164	124	124
q7	1959	1928	1781	1781
q8	2586	2508	2509	2508
q9	7124	7092	7030	7030
q10	3108	3268	2781	2781
q11	573	504	492	492
q12	689	753	630	630
q13	3488	3957	3356	3356
q14	274	304	272	272
q15	513	476	474	474
q16	434	483	438	438
q17	1156	1530	1401	1401
q18	7594	7282	7120	7120
q19	762	789	902	789
q20	1915	1955	1810	1810
q21	4834	4375	4254	4254
q22	1083	1026	996	996
Total cold run time: 51395 ms
Total hot run time: 49305 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185404 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0d589fdbc3a10f1b4933a121399e997a94f9036b, data reload: false

query1	997	389	381	381
query2	6544	1738	1745	1738
query3	6743	224	225	224
query4	26513	23235	23294	23235
query5	4331	584	439	439
query6	320	221	195	195
query7	4621	510	297	297
query8	258	226	212	212
query9	8618	2601	2626	2601
query10	443	312	275	275
query11	15748	15030	14839	14839
query12	165	115	104	104
query13	1652	547	425	425
query14	9517	5869	6070	5869
query15	202	189	168	168
query16	7603	627	438	438
query17	1174	720	612	612
query18	2021	396	297	297
query19	188	190	152	152
query20	122	120	111	111
query21	209	117	101	101
query22	4131	4165	4106	4106
query23	33834	32932	33106	32932
query24	8361	2363	2397	2363
query25	538	475	389	389
query26	714	265	152	152
query27	2723	503	348	348
query28	4256	2121	2100	2100
query29	650	575	434	434
query30	286	217	185	185
query31	909	831	744	744
query32	70	64	62	62
query33	546	364	320	320
query34	805	849	523	523
query35	779	819	747	747
query36	941	972	885	885
query37	109	100	82	82
query38	4071	4072	4133	4072
query39	1571	1402	1563	1402
query40	222	121	109	109
query41	55	55	52	52
query42	123	110	101	101
query43	485	505	470	470
query44	1369	835	835	835
query45	175	173	166	166
query46	855	1029	644	644
query47	1748	1804	1711	1711
query48	388	441	326	326
query49	695	495	385	385
query50	687	694	428	428
query51	4190	4149	4269	4149
query52	114	105	102	102
query53	233	260	188	188
query54	582	574	502	502
query55	83	77	90	77
query56	306	317	301	301
query57	1178	1189	1117	1117
query58	262	262	288	262
query59	2632	2610	2588	2588
query60	341	326	308	308
query61	122	121	129	121
query62	814	812	650	650
query63	225	192	192	192
query64	3093	994	673	673
query65	4271	4178	4168	4168
query66	906	410	316	316
query67	15613	15570	15251	15251
query68	7843	908	534	534
query69	512	300	277	277
query70	1191	1092	1091	1091
query71	429	325	308	308
query72	5656	4792	4886	4792
query73	687	652	353	353
query74	8962	8929	8960	8929
query75	3495	3205	2687	2687
query76	3225	1152	738	738
query77	538	402	291	291
query78	9944	10110	9393	9393
query79	1886	843	591	591
query80	616	521	449	449
query81	508	261	218	218
query82	183	122	96	96
query83	255	255	237	237
query84	246	111	83	83
query85	828	436	399	399
query86	371	312	298	298
query87	4322	4544	4273	4273
query88	3497	2304	2379	2304
query89	379	326	284	284
query90	1936	212	210	210
query91	135	138	113	113
query92	82	60	57	57
query93	1869	986	613	613
query94	675	426	301	301
query95	385	296	290	290
query96	494	569	284	284
query97	2723	2783	2691	2691
query98	239	203	197	197
query99	1304	1368	1272	1272
Total cold run time: 270465 ms
Total hot run time: 185404 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.58 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0d589fdbc3a10f1b4933a121399e997a94f9036b, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.24	0.07	0.07
query4	1.63	0.11	0.10
query5	0.44	0.44	0.43
query6	1.17	0.66	0.65
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.60	0.51	0.52
query10	0.58	0.58	0.56
query11	0.16	0.10	0.11
query12	0.14	0.12	0.11
query13	0.63	0.61	0.61
query14	0.79	0.82	0.81
query15	0.92	0.89	0.89
query16	0.40	0.38	0.41
query17	1.05	1.10	1.02
query18	0.22	0.22	0.21
query19	1.96	1.85	1.88
query20	0.01	0.02	0.01
query21	15.43	0.89	0.55
query22	0.78	1.16	0.71
query23	14.83	1.38	0.59
query24	7.27	1.48	0.62
query25	0.45	0.21	0.19
query26	0.64	0.17	0.14
query27	0.06	0.06	0.06
query28	9.23	0.91	0.45
query29	12.63	3.98	3.34
query30	0.27	0.09	0.07
query31	2.84	0.60	0.40
query32	3.22	0.55	0.50
query33	3.04	3.05	3.23
query34	16.12	5.42	4.78
query35	4.82	4.85	4.89
query36	0.72	0.50	0.50
query37	0.09	0.07	0.07
query38	0.05	0.05	0.04
query39	0.04	0.02	0.03
query40	0.17	0.15	0.14
query41	0.09	0.02	0.03
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 103.99 s
Total hot run time: 29.58 s

@hello-stephen
Copy link
Contributor

FE UT Coverage Report

Increment line coverage 80.24% (1811/2257) 🎉
Increment coverage report
Complete coverage report

@924060929
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 34114 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 977ece2a5cb50de2c10e70cbfa508c4a47741dcc, data reload: false

------ Round 1 ----------------------------------
q1	17606	5172	5046	5046
q2	1977	269	178	178
q3	10293	1309	726	726
q4	10219	1040	534	534
q5	7544	2367	2374	2367
q6	177	161	128	128
q7	904	744	598	598
q8	9311	1254	1090	1090
q9	6836	5129	5095	5095
q10	6874	2406	1968	1968
q11	479	283	283	283
q12	356	356	215	215
q13	17762	3655	3133	3133
q14	235	228	210	210
q15	552	487	478	478
q16	430	449	373	373
q17	614	860	363	363
q18	7657	7260	7220	7220
q19	1449	951	547	547
q20	330	353	224	224
q21	4008	2607	2372	2372
q22	1060	1001	966	966
Total cold run time: 106673 ms
Total hot run time: 34114 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5147	5094	5085	5085
q2	249	326	221	221
q3	2202	2609	2293	2293
q4	1400	1814	1343	1343
q5	4190	4323	4347	4323
q6	222	175	125	125
q7	1998	1969	1767	1767
q8	2601	2603	2632	2603
q9	7139	7194	7244	7194
q10	3079	3225	2792	2792
q11	588	521	480	480
q12	675	740	646	646
q13	3555	3925	3337	3337
q14	290	323	278	278
q15	538	487	470	470
q16	446	474	445	445
q17	1161	1599	1322	1322
q18	7404	7207	7144	7144
q19	781	750	818	750
q20	1945	2044	1807	1807
q21	4890	4291	4277	4277
q22	1075	1045	973	973
Total cold run time: 51575 ms
Total hot run time: 49675 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184251 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 977ece2a5cb50de2c10e70cbfa508c4a47741dcc, data reload: false

query1	1002	393	412	393
query2	6537	1706	1690	1690
query3	6754	216	221	216
query4	26047	23774	23057	23057
query5	4363	607	436	436
query6	325	213	207	207
query7	4640	502	304	304
query8	276	221	219	219
query9	8613	2592	2601	2592
query10	479	338	268	268
query11	15330	15070	14852	14852
query12	163	107	108	107
query13	1644	531	414	414
query14	9138	6131	5822	5822
query15	204	187	176	176
query16	7531	619	461	461
query17	1209	721	601	601
query18	2099	402	324	324
query19	204	196	159	159
query20	117	116	111	111
query21	209	131	108	108
query22	4073	4135	3995	3995
query23	34021	33164	32943	32943
query24	8274	2381	2357	2357
query25	535	473	382	382
query26	1236	270	152	152
query27	2696	507	345	345
query28	4297	2120	2101	2101
query29	775	579	443	443
query30	282	213	187	187
query31	896	825	760	760
query32	70	61	60	60
query33	550	358	331	331
query34	803	822	530	530
query35	791	813	724	724
query36	973	980	898	898
query37	115	97	76	76
query38	4168	4100	4061	4061
query39	1501	1407	1433	1407
query40	212	114	104	104
query41	58	60	53	53
query42	120	110	102	102
query43	483	532	464	464
query44	1325	828	817	817
query45	178	170	161	161
query46	831	990	614	614
query47	1745	1800	1710	1710
query48	391	410	331	331
query49	731	475	377	377
query50	627	696	428	428
query51	4097	4110	4047	4047
query52	109	103	99	99
query53	221	240	186	186
query54	574	565	507	507
query55	80	82	83	82
query56	293	274	281	274
query57	1191	1199	1114	1114
query58	265	257	256	256
query59	2582	2678	2606	2606
query60	337	322	317	317
query61	122	117	120	117
query62	778	727	638	638
query63	225	183	182	182
query64	4314	1037	659	659
query65	4266	4209	4134	4134
query66	1094	406	314	314
query67	15677	15542	15198	15198
query68	8536	889	529	529
query69	480	303	258	258
query70	1190	1067	1089	1067
query71	491	317	297	297
query72	5621	4737	4576	4576
query73	690	562	348	348
query74	8934	8889	8887	8887
query75	3896	3254	2699	2699
query76	3749	1139	729	729
query77	799	368	300	300
query78	10017	10286	9368	9368
query79	2178	820	579	579
query80	585	506	449	449
query81	472	266	223	223
query82	474	129	95	95
query83	288	252	232	232
query84	294	108	85	85
query85	781	403	314	314
query86	328	277	282	277
query87	4414	4452	4294	4294
query88	2943	2247	2250	2247
query89	397	312	286	286
query90	1921	212	201	201
query91	140	140	110	110
query92	71	62	57	57
query93	1138	938	599	599
query94	684	414	306	306
query95	375	292	290	290
query96	491	564	278	278
query97	2715	2714	2661	2661
query98	227	215	203	203
query99	1427	1396	1247	1247
Total cold run time: 272612 ms
Total hot run time: 184251 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.82 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 977ece2a5cb50de2c10e70cbfa508c4a47741dcc, data reload: false

query1	0.05	0.04	0.03
query2	0.08	0.04	0.04
query3	0.24	0.07	0.07
query4	1.61	0.12	0.11
query5	0.44	0.46	0.41
query6	1.17	0.66	0.66
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.61	0.53	0.52
query10	0.56	0.58	0.57
query11	0.17	0.11	0.11
query12	0.15	0.12	0.12
query13	0.63	0.61	0.61
query14	0.81	0.82	0.81
query15	0.91	0.87	0.91
query16	0.39	0.39	0.40
query17	1.09	1.10	1.05
query18	0.23	0.22	0.22
query19	2.02	1.87	1.83
query20	0.02	0.01	0.01
query21	15.40	0.93	0.54
query22	0.76	1.18	0.73
query23	14.85	1.39	0.63
query24	6.59	1.59	0.94
query25	0.50	0.09	0.09
query26	0.60	0.18	0.15
query27	0.06	0.06	0.05
query28	10.09	0.91	0.46
query29	12.57	4.03	3.31
query30	0.25	0.09	0.06
query31	2.84	0.62	0.39
query32	3.23	0.56	0.47
query33	3.16	3.10	3.13
query34	16.03	5.41	4.76
query35	4.83	4.88	4.84
query36	0.71	0.52	0.50
query37	0.09	0.06	0.07
query38	0.05	0.04	0.03
query39	0.04	0.02	0.03
query40	0.18	0.15	0.14
query41	0.09	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 104.25 s
Total hot run time: 29.82 s

@hello-stephen
Copy link
Contributor

FE UT Coverage Report

Increment line coverage 80.24% (1811/2257) 🎉
Increment coverage report
Complete coverage report

@924060929
Copy link
Contributor Author

run buildall

@hello-stephen
Copy link
Contributor

FE UT Coverage Report

Increment line coverage 80.24% (1811/2257) 🎉
Increment coverage report
Complete coverage report

@doris-robot
Copy link

TPC-H: Total hot run time: 33989 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 17ae1a19a96d6cdd6c56bea4cda7e887f526448c, data reload: false

------ Round 1 ----------------------------------
q1	17593	5229	5112	5112
q2	1960	280	194	194
q3	10291	1256	708	708
q4	10205	1028	515	515
q5	7536	2349	2384	2349
q6	181	162	128	128
q7	913	743	602	602
q8	9326	1235	1058	1058
q9	6859	5093	5116	5093
q10	6875	2383	1986	1986
q11	493	282	280	280
q12	346	358	218	218
q13	17764	3677	3088	3088
q14	242	219	222	219
q15	538	483	483	483
q16	441	428	379	379
q17	613	861	362	362
q18	7538	7104	7162	7104
q19	1207	960	555	555
q20	327	356	224	224
q21	3750	3215	2410	2410
q22	1066	1036	922	922
Total cold run time: 106064 ms
Total hot run time: 33989 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5165	5152	5144	5144
q2	250	330	219	219
q3	2156	2712	2303	2303
q4	1410	1795	1349	1349
q5	4214	4368	4300	4300
q6	218	173	136	136
q7	2008	1913	1786	1786
q8	2624	2577	2519	2519
q9	7179	7143	7183	7143
q10	3065	3433	2837	2837
q11	569	521	494	494
q12	670	761	616	616
q13	3555	3845	3317	3317
q14	277	304	273	273
q15	525	482	466	466
q16	456	499	451	451
q17	1148	1567	1359	1359
q18	7495	7071	6957	6957
q19	773	807	990	807
q20	1939	1996	1812	1812
q21	4811	4348	4336	4336
q22	1076	1043	980	980
Total cold run time: 51583 ms
Total hot run time: 49604 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186103 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 17ae1a19a96d6cdd6c56bea4cda7e887f526448c, data reload: false

query1	1023	412	436	412
query2	6543	1733	1698	1698
query3	6746	219	231	219
query4	26575	24035	23072	23072
query5	4745	589	419	419
query6	313	230	205	205
query7	4642	494	295	295
query8	279	230	208	208
query9	8656	2706	2723	2706
query10	462	330	269	269
query11	15721	15105	14801	14801
query12	159	106	106	106
query13	1655	574	433	433
query14	9425	5828	5831	5828
query15	208	189	175	175
query16	7666	651	468	468
query17	1174	719	567	567
query18	2016	406	291	291
query19	218	191	159	159
query20	123	117	112	112
query21	207	125	104	104
query22	4073	4143	4101	4101
query23	34138	33285	33128	33128
query24	8427	2385	2435	2385
query25	558	461	402	402
query26	1224	271	155	155
query27	2745	510	354	354
query28	4310	2212	2180	2180
query29	718	564	435	435
query30	283	218	191	191
query31	946	848	751	751
query32	77	66	64	64
query33	565	356	318	318
query34	806	844	528	528
query35	779	823	798	798
query36	977	1007	920	920
query37	106	101	81	81
query38	4263	4233	4114	4114
query39	1524	1435	1409	1409
query40	215	123	104	104
query41	55	54	52	52
query42	127	112	109	109
query43	495	532	506	506
query44	1310	831	847	831
query45	201	185	167	167
query46	851	1008	645	645
query47	1775	1810	1765	1765
query48	393	413	303	303
query49	742	488	397	397
query50	642	695	414	414
query51	4132	4249	4038	4038
query52	113	107	101	101
query53	234	257	187	187
query54	574	576	518	518
query55	90	79	76	76
query56	313	308	281	281
query57	1201	1196	1113	1113
query58	263	258	258	258
query59	2638	2728	2573	2573
query60	326	323	302	302
query61	124	125	123	123
query62	854	715	641	641
query63	227	192	193	192
query64	4198	1008	759	759
query65	4297	4185	4222	4185
query66	1074	429	319	319
query67	15861	15515	15473	15473
query68	7836	914	540	540
query69	465	310	280	280
query70	1229	1160	1110	1110
query71	468	336	318	318
query72	5689	4761	4784	4761
query73	715	665	365	365
query74	9188	9039	8915	8915
query75	3846	3189	2766	2766
query76	3553	1141	753	753
query77	810	376	293	293
query78	9949	10180	9332	9332
query79	2278	837	602	602
query80	575	514	453	453
query81	491	269	226	226
query82	452	126	99	99
query83	266	248	227	227
query84	245	119	82	82
query85	865	435	398	398
query86	382	393	307	307
query87	4463	4443	4394	4394
query88	3759	2348	2311	2311
query89	394	315	289	289
query90	1844	207	198	198
query91	150	144	116	116
query92	72	60	56	56
query93	1833	964	604	604
query94	673	369	304	304
query95	377	290	287	287
query96	504	589	286	286
query97	2746	2814	2655	2655
query98	235	216	204	204
query99	1618	1426	1279	1279
Total cold run time: 276370 ms
Total hot run time: 186103 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.78 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 17ae1a19a96d6cdd6c56bea4cda7e887f526448c, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.05	0.04
query3	0.24	0.07	0.07
query4	1.61	0.11	0.12
query5	0.45	0.46	0.44
query6	1.15	0.66	0.65
query7	0.03	0.02	0.02
query8	0.05	0.04	0.04
query9	0.61	0.55	0.52
query10	0.58	0.58	0.58
query11	0.16	0.11	0.11
query12	0.15	0.12	0.11
query13	0.63	0.60	0.60
query14	0.80	0.83	0.79
query15	0.91	0.90	0.86
query16	0.39	0.38	0.38
query17	1.06	1.10	1.06
query18	0.23	0.22	0.21
query19	1.99	1.85	1.83
query20	0.02	0.01	0.01
query21	15.38	0.91	0.56
query22	0.76	1.15	0.62
query23	15.01	1.40	0.63
query24	6.81	1.46	1.01
query25	0.43	0.09	0.06
query26	0.57	0.18	0.14
query27	0.06	0.05	0.05
query28	9.76	0.92	0.47
query29	12.58	4.02	3.34
query30	0.25	0.09	0.06
query31	2.82	0.60	0.39
query32	3.27	0.56	0.47
query33	3.05	3.07	3.17
query34	16.02	5.37	4.76
query35	4.84	4.90	4.86
query36	0.71	0.52	0.49
query37	0.10	0.07	0.06
query38	0.06	0.04	0.04
query39	0.03	0.02	0.02
query40	0.18	0.15	0.15
query41	0.08	0.03	0.02
query42	0.04	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 104.03 s
Total hot run time: 29.78 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 26, 2025
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@924060929 924060929 merged commit ceebf3f into apache:master Jun 26, 2025
25 of 27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/3.1.x dev/3.1.x-conflict reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants