Skip to content

[Enhancement] implement createRepositoryCommand in nereids #51161

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

[Enhancement] implement createRepositoryCommand in nereids #51161

wants to merge 2 commits into from

Conversation

codeDing18
Copy link
Contributor

What problem does this PR solve?

Issue Number: close #51003

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@codeDing18
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 34036 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c351f6644ab487e9ac1fe7e5a1b13adbdc4d74e4, data reload: false

------ Round 1 ----------------------------------
q1	26297	5082	5007	5007
q2	2081	282	183	183
q3	10397	1244	708	708
q4	10231	993	531	531
q5	8052	2404	2377	2377
q6	279	165	134	134
q7	938	743	632	632
q8	9337	1396	1116	1116
q9	6873	5123	5117	5117
q10	6868	2318	1897	1897
q11	486	307	274	274
q12	341	352	205	205
q13	17771	3632	3134	3134
q14	242	257	209	209
q15	538	507	494	494
q16	422	429	390	390
q17	628	863	378	378
q18	7532	7107	7109	7107
q19	1289	952	552	552
q20	349	360	226	226
q21	4187	3197	2388	2388
q22	1089	1007	977	977
Total cold run time: 116227 ms
Total hot run time: 34036 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5107	5139	5129	5129
q2	235	325	226	226
q3	2166	2663	2303	2303
q4	1331	1772	1347	1347
q5	4697	4472	4334	4334
q6	215	173	130	130
q7	1960	1914	1709	1709
q8	2578	2509	2515	2509
q9	7247	7084	7140	7084
q10	2980	3210	2760	2760
q11	587	526	513	513
q12	680	765	591	591
q13	3495	3925	3337	3337
q14	286	295	267	267
q15	547	507	504	504
q16	443	509	455	455
q17	1154	1614	1325	1325
q18	7790	7557	7347	7347
q19	807	825	853	825
q20	2000	2007	1879	1879
q21	4827	4427	4350	4350
q22	1045	1051	993	993
Total cold run time: 52177 ms
Total hot run time: 49917 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 183250 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c351f6644ab487e9ac1fe7e5a1b13adbdc4d74e4, data reload: false

query1	1008	473	480	473
query2	6570	1899	1835	1835
query3	6762	219	222	219
query4	26804	23717	23473	23473
query5	4341	613	476	476
query6	295	212	182	182
query7	4611	492	282	282
query8	288	252	245	245
query9	8624	2627	2646	2627
query10	450	323	270	270
query11	15652	15049	15045	15045
query12	153	113	106	106
query13	1664	526	430	430
query14	9183	6297	6043	6043
query15	203	196	172	172
query16	7166	638	488	488
query17	1200	723	581	581
query18	1980	405	299	299
query19	198	195	162	162
query20	122	123	121	121
query21	214	129	111	111
query22	4141	4159	3970	3970
query23	33833	32779	33021	32779
query24	8425	2352	2374	2352
query25	535	450	393	393
query26	1235	279	146	146
query27	2758	506	344	344
query28	4333	2141	2119	2119
query29	784	533	442	442
query30	281	215	189	189
query31	948	834	743	743
query32	70	63	62	62
query33	557	378	311	311
query34	809	850	514	514
query35	783	793	723	723
query36	987	985	854	854
query37	112	102	77	77
query38	4211	4173	4033	4033
query39	1468	1397	1401	1397
query40	210	119	105	105
query41	55	55	55	55
query42	120	104	107	104
query43	504	514	486	486
query44	1322	822	833	822
query45	181	172	165	165
query46	835	1021	637	637
query47	1715	1799	1706	1706
query48	388	421	308	308
query49	776	516	436	436
query50	638	686	412	412
query51	4131	4190	4153	4153
query52	109	112	107	107
query53	222	256	194	194
query54	583	573	512	512
query55	95	83	81	81
query56	311	299	303	299
query57	1149	1138	1089	1089
query58	276	255	252	252
query59	2582	2805	2600	2600
query60	341	312	303	303
query61	125	124	123	123
query62	801	727	679	679
query63	225	195	185	185
query64	4365	1032	674	674
query65	4322	4222	4286	4222
query66	1161	408	306	306
query67	15784	15566	15324	15324
query68	7985	880	511	511
query69	478	309	262	262
query70	1199	1143	1094	1094
query71	443	349	300	300
query72	5573	2348	5080	2348
query73	763	765	355	355
query74	8979	9182	8723	8723
query75	3802	3182	2674	2674
query76	3629	1186	764	764
query77	782	368	286	286
query78	10030	10227	9384	9384
query79	2496	787	584	584
query80	628	520	427	427
query81	472	269	221	221
query82	422	126	104	104
query83	275	252	237	237
query84	290	121	87	87
query85	782	355	306	306
query86	339	318	313	313
query87	4450	4428	4388	4388
query88	3314	2247	2253	2247
query89	392	317	293	293
query90	1936	216	201	201
query91	138	142	115	115
query92	76	61	58	58
query93	1507	935	577	577
query94	678	409	294	294
query95	371	292	292	292
query96	496	577	279	279
query97	2709	2734	2635	2635
query98	244	206	202	202
query99	1447	1408	1262	1262
Total cold run time: 274209 ms
Total hot run time: 183250 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.65 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c351f6644ab487e9ac1fe7e5a1b13adbdc4d74e4, data reload: false

query1	0.04	0.04	0.04
query2	0.12	0.11	0.10
query3	0.25	0.19	0.19
query4	1.60	0.19	0.11
query5	0.45	0.41	0.44
query6	1.16	0.66	0.66
query7	0.03	0.01	0.01
query8	0.04	0.03	0.04
query9	0.58	0.52	0.51
query10	0.57	0.57	0.56
query11	0.16	0.11	0.10
query12	0.15	0.11	0.12
query13	0.61	0.60	0.60
query14	0.79	0.82	0.80
query15	0.90	0.87	0.88
query16	0.38	0.39	0.39
query17	1.04	1.07	1.04
query18	0.22	0.20	0.21
query19	1.91	1.85	1.86
query20	0.01	0.01	0.01
query21	15.40	0.86	0.54
query22	0.75	1.31	0.71
query23	14.75	1.38	0.62
query24	7.21	1.10	0.27
query25	0.39	0.29	0.08
query26	0.52	0.16	0.15
query27	0.06	0.05	0.05
query28	9.60	0.94	0.45
query29	12.59	4.16	3.42
query30	0.25	0.08	0.07
query31	2.82	0.58	0.38
query32	3.22	0.55	0.46
query33	2.97	3.12	3.08
query34	15.83	5.10	4.50
query35	4.53	4.51	4.46
query36	0.66	0.49	0.48
query37	0.08	0.07	0.06
query38	0.05	0.04	0.04
query39	0.02	0.02	0.03
query40	0.16	0.14	0.14
query41	0.08	0.03	0.03
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 103.02 s
Total hot run time: 28.65 s

@codeDing18
Copy link
Contributor Author

run feut

@codeDing18
Copy link
Contributor Author

run p0

@codeDing18 codeDing18 force-pushed the create-repository-nereids branch from c351f66 to 402301b Compare May 22, 2025 14:39
@codeDing18
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33853 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 402301b3d97285130fde0fbb1bf75538ef83b419, data reload: false

------ Round 1 ----------------------------------
q1	26224	5056	5027	5027
q2	2082	275	185	185
q3	10396	1218	691	691
q4	10217	990	537	537
q5	7552	2399	2308	2308
q6	183	166	133	133
q7	935	742	609	609
q8	9306	1291	1045	1045
q9	6810	5131	5127	5127
q10	6946	2300	1932	1932
q11	481	286	274	274
q12	347	355	225	225
q13	18262	3703	3085	3085
q14	231	239	213	213
q15	542	482	489	482
q16	418	436	388	388
q17	598	855	363	363
q18	7412	7102	7132	7102
q19	1562	946	544	544
q20	342	348	226	226
q21	3631	3174	2404	2404
q22	1043	1001	953	953
Total cold run time: 115520 ms
Total hot run time: 33853 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5189	5071	5085	5071
q2	239	317	230	230
q3	2123	2623	2290	2290
q4	1339	1750	1369	1369
q5	4381	4350	4379	4350
q6	216	167	129	129
q7	2025	1943	1804	1804
q8	2588	2460	2592	2460
q9	7292	7226	6893	6893
q10	3052	3205	2785	2785
q11	580	533	520	520
q12	664	766	614	614
q13	3524	3851	3298	3298
q14	278	324	269	269
q15	524	487	476	476
q16	463	518	426	426
q17	1120	1592	1343	1343
q18	7714	7686	7408	7408
q19	841	890	1030	890
q20	1981	2044	1860	1860
q21	4746	4410	4500	4410
q22	1093	1046	1017	1017
Total cold run time: 51972 ms
Total hot run time: 49912 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192610 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 402301b3d97285130fde0fbb1bf75538ef83b419, data reload: false

query1	1412	1117	1064	1064
query2	6322	1861	1863	1861
query3	11021	4527	4346	4346
query4	54676	25713	23245	23245
query5	5093	587	479	479
query6	335	215	227	215
query7	4896	515	295	295
query8	317	253	244	244
query9	5481	2632	2625	2625
query10	406	314	262	262
query11	15024	14913	14734	14734
query12	164	109	106	106
query13	1043	509	418	418
query14	10119	6326	6405	6326
query15	212	192	179	179
query16	7070	685	513	513
query17	1086	753	607	607
query18	1548	429	321	321
query19	208	198	172	172
query20	132	138	127	127
query21	208	132	115	115
query22	4505	4490	4347	4347
query23	34446	33681	33534	33534
query24	6551	2472	2436	2436
query25	471	467	414	414
query26	721	287	158	158
query27	2382	509	343	343
query28	2950	2229	2168	2168
query29	591	596	441	441
query30	298	223	193	193
query31	851	867	754	754
query32	69	63	68	63
query33	457	365	347	347
query34	783	899	529	529
query35	815	837	779	779
query36	948	1009	923	923
query37	115	105	80	80
query38	4306	4319	4221	4221
query39	1554	1446	1446	1446
query40	221	119	109	109
query41	57	58	60	58
query42	130	119	110	110
query43	529	514	497	497
query44	1385	859	843	843
query45	185	173	168	168
query46	864	1034	675	675
query47	1845	1853	1793	1793
query48	419	450	340	340
query49	713	534	437	437
query50	688	706	420	420
query51	4296	4451	4253	4253
query52	120	117	100	100
query53	231	278	187	187
query54	596	576	522	522
query55	84	95	87	87
query56	334	317	304	304
query57	1175	1182	1124	1124
query58	264	263	259	259
query59	2709	2902	2767	2767
query60	360	330	314	314
query61	127	140	128	128
query62	732	731	669	669
query63	221	182	194	182
query64	1832	1029	726	726
query65	4293	4251	4248	4248
query66	726	457	304	304
query67	15991	15809	15419	15419
query68	6707	911	530	530
query69	524	333	268	268
query70	1216	1110	1125	1110
query71	506	326	299	299
query72	5773	4692	4662	4662
query73	1449	610	350	350
query74	9204	9112	8770	8770
query75	4102	3214	2720	2720
query76	4333	1195	770	770
query77	807	394	283	283
query78	10234	10164	9302	9302
query79	4503	790	562	562
query80	631	522	449	449
query81	485	260	215	215
query82	522	135	98	98
query83	294	259	231	231
query84	294	103	88	88
query85	777	413	314	314
query86	377	292	300	292
query87	4352	4404	4260	4260
query88	3612	2306	2298	2298
query89	434	319	284	284
query90	1951	217	216	216
query91	142	161	111	111
query92	77	60	55	55
query93	2819	937	576	576
query94	663	420	309	309
query95	373	305	285	285
query96	493	582	286	286
query97	2785	2794	2679	2679
query98	231	214	198	198
query99	1415	1389	1267	1267
Total cold run time: 302798 ms
Total hot run time: 192610 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.1 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 402301b3d97285130fde0fbb1bf75538ef83b419, data reload: false

query1	0.04	0.04	0.03
query2	0.13	0.11	0.12
query3	0.27	0.20	0.20
query4	1.59	0.20	0.10
query5	0.44	0.43	0.43
query6	1.15	0.65	0.65
query7	0.02	0.01	0.02
query8	0.04	0.03	0.04
query9	0.57	0.52	0.51
query10	0.56	0.58	0.58
query11	0.17	0.11	0.11
query12	0.14	0.11	0.12
query13	0.62	0.61	0.60
query14	0.78	0.81	0.81
query15	0.87	0.86	0.87
query16	0.40	0.38	0.38
query17	1.06	1.08	1.03
query18	0.22	0.21	0.22
query19	1.88	1.82	1.84
query20	0.02	0.01	0.01
query21	15.39	0.92	0.56
query22	0.76	1.16	0.90
query23	14.74	1.38	0.64
query24	7.15	1.41	0.59
query25	0.47	0.27	0.06
query26	0.63	0.16	0.15
query27	0.05	0.06	0.05
query28	9.75	0.95	0.44
query29	12.53	3.97	3.31
query30	0.26	0.10	0.06
query31	2.81	0.59	0.38
query32	3.23	0.54	0.47
query33	3.04	3.07	3.10
query34	15.74	5.08	4.52
query35	4.51	4.57	4.51
query36	0.67	0.49	0.49
query37	0.08	0.07	0.06
query38	0.05	0.04	0.03
query39	0.02	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 103.16 s
Total hot run time: 29.1 s

@codeDing18
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33688 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e0e2bd05be13e097025ccb41e56b8f00d0166b77, data reload: false

------ Round 1 ----------------------------------
q1	25874	5134	5012	5012
q2	2067	272	175	175
q3	10405	1291	670	670
q4	10236	1001	510	510
q5	7960	2443	2290	2290
q6	181	163	134	134
q7	935	737	609	609
q8	9302	1251	1122	1122
q9	7098	5050	5103	5050
q10	6871	2325	1882	1882
q11	501	301	288	288
q12	344	343	210	210
q13	17780	3698	3056	3056
q14	226	221	212	212
q15	528	477	490	477
q16	424	425	374	374
q17	576	844	377	377
q18	7566	7154	7159	7154
q19	1420	928	529	529
q20	336	347	224	224
q21	3560	3130	2363	2363
q22	1054	1042	970	970
Total cold run time: 115244 ms
Total hot run time: 33688 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5096	5121	5076	5076
q2	236	325	231	231
q3	2165	2636	2268	2268
q4	1332	1779	1342	1342
q5	4408	4305	4399	4305
q6	211	168	130	130
q7	1991	1906	1794	1794
q8	2595	2553	2511	2511
q9	7233	7204	7012	7012
q10	3025	3221	2759	2759
q11	559	527	496	496
q12	657	751	623	623
q13	3526	3835	3276	3276
q14	284	292	294	292
q15	522	473	489	473
q16	461	485	423	423
q17	1127	1531	1367	1367
q18	7671	7595	7458	7458
q19	772	787	979	787
q20	1998	2095	1864	1864
q21	4790	4458	4484	4458
q22	1108	1075	1028	1028
Total cold run time: 51767 ms
Total hot run time: 49973 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192934 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e0e2bd05be13e097025ccb41e56b8f00d0166b77, data reload: false

query1	1412	1104	1052	1052
query2	6540	1848	1869	1848
query3	11141	4614	4634	4614
query4	25580	23625	23114	23114
query5	4311	655	473	473
query6	318	216	209	209
query7	4003	498	292	292
query8	297	259	246	246
query9	8498	2617	2638	2617
query10	491	340	263	263
query11	15566	15460	14926	14926
query12	172	108	107	107
query13	1724	536	417	417
query14	10347	6334	6322	6322
query15	202	195	172	172
query16	7590	645	522	522
query17	1412	771	593	593
query18	2010	405	293	293
query19	188	204	170	170
query20	132	126	116	116
query21	203	125	109	109
query22	4564	4523	4336	4336
query23	34528	33727	33856	33727
query24	8172	2475	2431	2431
query25	521	477	399	399
query26	1253	278	153	153
query27	2784	502	342	342
query28	4966	2172	2226	2172
query29	705	593	457	457
query30	277	225	188	188
query31	933	863	794	794
query32	74	62	64	62
query33	524	374	314	314
query34	799	880	559	559
query35	757	808	740	740
query36	963	989	916	916
query37	115	97	77	77
query38	4264	4346	4303	4303
query39	1515	1480	1482	1480
query40	209	120	113	113
query41	59	54	54	54
query42	124	114	114	114
query43	517	519	491	491
query44	1376	858	833	833
query45	187	173	167	167
query46	838	1035	690	690
query47	1913	1861	1806	1806
query48	402	436	322	322
query49	754	524	434	434
query50	661	696	411	411
query51	4240	4296	4183	4183
query52	113	107	103	103
query53	224	269	186	186
query54	595	598	517	517
query55	88	86	87	86
query56	343	309	323	309
query57	1176	1186	1169	1169
query58	268	260	257	257
query59	2773	2788	2741	2741
query60	328	331	315	315
query61	128	132	135	132
query62	765	732	714	714
query63	225	202	200	200
query64	4120	1014	715	715
query65	4496	4497	4231	4231
query66	947	397	303	303
query67	15989	15533	15257	15257
query68	8980	874	520	520
query69	460	309	267	267
query70	1244	1063	1094	1063
query71	460	324	292	292
query72	5796	4797	4869	4797
query73	726	665	358	358
query74	9011	8854	8708	8708
query75	3972	3191	2635	2635
query76	3488	1174	753	753
query77	743	379	285	285
query78	10003	10354	9396	9396
query79	1915	777	581	581
query80	592	512	469	469
query81	476	258	219	219
query82	434	124	95	95
query83	253	253	235	235
query84	237	111	88	88
query85	805	348	311	311
query86	328	299	302	299
query87	4436	4439	4261	4261
query88	3396	2263	2256	2256
query89	408	309	290	290
query90	1883	212	217	212
query91	144	140	117	117
query92	73	57	55	55
query93	1267	950	593	593
query94	662	419	310	310
query95	374	293	285	285
query96	507	555	284	284
query97	2712	2791	2650	2650
query98	221	205	202	202
query99	1420	1362	1270	1270
Total cold run time: 281135 ms
Total hot run time: 192934 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.96 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e0e2bd05be13e097025ccb41e56b8f00d0166b77, data reload: false

query1	0.04	0.03	0.04
query2	0.13	0.10	0.12
query3	0.25	0.19	0.19
query4	1.60	0.18	0.20
query5	0.44	0.45	0.45
query6	1.15	0.66	0.66
query7	0.02	0.02	0.02
query8	0.05	0.04	0.04
query9	0.58	0.51	0.55
query10	0.56	0.58	0.58
query11	0.15	0.11	0.10
query12	0.15	0.12	0.12
query13	0.60	0.60	0.60
query14	0.79	0.80	0.81
query15	0.88	0.86	0.86
query16	0.38	0.38	0.38
query17	1.08	1.04	1.08
query18	0.22	0.21	0.21
query19	1.96	1.82	1.83
query20	0.01	0.01	0.01
query21	15.42	0.93	0.55
query22	0.76	1.11	0.64
query23	15.01	1.34	0.61
query24	6.68	1.67	0.58
query25	0.57	0.20	0.12
query26	0.63	0.16	0.13
query27	0.05	0.05	0.04
query28	9.64	0.91	0.43
query29	12.52	3.92	3.25
query30	0.26	0.09	0.06
query31	2.83	0.58	0.39
query32	3.23	0.55	0.46
query33	3.00	3.13	3.14
query34	15.70	5.17	4.55
query35	4.55	4.61	4.51
query36	0.68	0.50	0.48
query37	0.09	0.07	0.07
query38	0.05	0.04	0.04
query39	0.03	0.03	0.03
query40	0.17	0.14	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 103.06 s
Total hot run time: 28.96 s

@codeDing18
Copy link
Contributor Author

run feut

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Enhancement] implement createRepository in nereids
3 participants
@codeDing18 @hello-stephen @doris-robot