Skip to content

[fix](inverted index) create empty idx file when creating a index on variant-type column (#50937) #51181

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
May 27, 2025

Conversation

csun5285
Copy link
Contributor

pick from master #50937

csun5285 added 2 commits May 23, 2025 11:16
…variant-type column (apache#50937)

create empty idx file when creating a index on variant-type column
@csun5285 csun5285 requested a review from dataroaring as a code owner May 23, 2025 04:05
@Thearas
Copy link
Contributor

Thearas commented May 23, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@csun5285
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39613 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f7e6589a40189f097c6501723128c9d002117604, data reload: false

------ Round 1 ----------------------------------
q1	17579	6715	6557	6557
q2	2068	178	158	158
q3	10629	1101	1198	1101
q4	10533	740	724	724
q5	7756	2860	2852	2852
q6	218	133	134	133
q7	945	619	600	600
q8	9349	1937	2062	1937
q9	6662	6376	6394	6376
q10	7046	2314	2268	2268
q11	471	258	259	258
q12	391	211	204	204
q13	17769	2958	2968	2958
q14	235	200	219	200
q15	507	460	451	451
q16	464	371	376	371
q17	974	641	500	500
q18	7124	6707	6571	6571
q19	1438	1101	1015	1015
q20	462	206	200	200
q21	3956	3231	3206	3206
q22	1087	1007	973	973
Total cold run time: 107663 ms
Total hot run time: 39613 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6608	6640	6630	6630
q2	328	240	247	240
q3	2971	2884	2930	2884
q4	2079	1843	1834	1834
q5	5821	5773	5724	5724
q6	221	127	132	127
q7	2177	1800	1792	1792
q8	3358	3566	3471	3471
q9	8953	8679	8869	8679
q10	3535	3519	3487	3487
q11	597	475	506	475
q12	788	590	594	590
q13	9788	3085	3162	3085
q14	297	268	260	260
q15	512	463	470	463
q16	483	461	439	439
q17	1808	1627	1592	1592
q18	8213	7768	7662	7662
q19	1679	1536	1576	1536
q20	2130	1841	1803	1803
q21	5499	5271	5238	5238
q22	1095	1067	1016	1016
Total cold run time: 68940 ms
Total hot run time: 59027 ms

@csun5285
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40601 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 41fb483cd584c467bd0d3b86a80cf991ba223926, data reload: false

------ Round 1 ----------------------------------
q1	17587	6884	6686	6686
q2	2049	166	156	156
q3	10565	1197	1212	1197
q4	10223	808	727	727
q5	7734	2896	2856	2856
q6	216	135	131	131
q7	979	627	609	609
q8	9368	2018	2064	2018
q9	6728	6452	6494	6452
q10	7031	2278	2323	2278
q11	453	267	268	267
q12	398	219	218	218
q13	17779	2959	2997	2959
q14	240	201	207	201
q15	507	469	464	464
q16	461	381	383	381
q17	984	579	596	579
q18	7535	6729	6719	6719
q19	1392	1139	1138	1138
q20	492	221	203	203
q21	4145	3379	3457	3379
q22	1095	989	983	983
Total cold run time: 107961 ms
Total hot run time: 40601 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6742	6673	6632	6632
q2	327	231	228	228
q3	3012	2883	2863	2863
q4	1976	1759	1755	1755
q5	5504	5550	5499	5499
q6	201	125	124	124
q7	2129	1742	1733	1733
q8	3317	3441	3443	3441
q9	8697	8611	8622	8611
q10	3509	3473	3469	3469
q11	581	500	485	485
q12	796	572	594	572
q13	12489	3015	3018	3015
q14	287	259	257	257
q15	508	459	455	455
q16	483	420	424	420
q17	1829	1601	1569	1569
q18	7866	7617	7443	7443
q19	1673	1608	1608	1608
q20	2037	1787	1803	1787
q21	5491	5242	5260	5242
q22	1120	995	985	985
Total cold run time: 70574 ms
Total hot run time: 58193 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 189524 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 41fb483cd584c467bd0d3b86a80cf991ba223926, data reload: false

query1	949	372	359	359
query2	6406	1931	1914	1914
query3	6412	213	233	213
query4	33385	23336	23382	23336
query5	3903	460	461	460
query6	254	171	196	171
query7	4059	313	307	307
query8	282	223	215	215
query9	8729	2592	2599	2592
query10	496	264	254	254
query11	18176	15146	15244	15146
query12	166	100	101	100
query13	1782	411	398	398
query14	9787	7171	6636	6636
query15	250	169	172	169
query16	8024	474	487	474
query17	1631	579	562	562
query18	2147	312	316	312
query19	337	154	154	154
query20	114	105	108	105
query21	206	105	104	104
query22	4334	4056	4139	4056
query23	34625	33522	33595	33522
query24	11186	2840	2857	2840
query25	639	407	403	403
query26	1122	169	173	169
query27	2646	349	370	349
query28	7748	2123	2114	2114
query29	758	449	438	438
query30	352	160	155	155
query31	1029	784	803	784
query32	100	61	57	57
query33	776	289	303	289
query34	935	504	515	504
query35	875	705	723	705
query36	1114	926	983	926
query37	124	72	69	69
query38	3982	3809	3793	3793
query39	1494	1430	1430	1430
query40	214	103	102	102
query41	54	50	52	50
query42	111	100	102	100
query43	510	482	478	478
query44	1288	798	796	796
query45	181	168	174	168
query46	1125	698	709	698
query47	1934	1804	1794	1794
query48	476	374	392	374
query49	1197	417	418	417
query50	800	412	419	412
query51	7266	7189	7266	7189
query52	101	97	90	90
query53	260	183	185	183
query54	1532	485	487	485
query55	80	81	79	79
query56	268	254	248	248
query57	1249	1089	1099	1089
query58	239	214	221	214
query59	3046	2985	2962	2962
query60	301	279	276	276
query61	136	136	134	134
query62	870	665	665	665
query63	221	199	204	199
query64	5625	630	626	626
query65	3273	3168	3185	3168
query66	1401	312	297	297
query67	15951	15517	15403	15403
query68	4941	577	568	568
query69	441	273	266	266
query70	1181	1149	1076	1076
query71	445	255	265	255
query72	6385	4103	4001	4001
query73	755	343	342	342
query74	10633	9216	8942	8942
query75	3388	2628	2638	2628
query76	2857	1047	1064	1047
query77	437	271	268	268
query78	10420	9615	9521	9521
query79	1055	586	592	586
query80	669	430	436	430
query81	507	221	219	219
query82	464	90	89	89
query83	239	146	148	146
query84	236	84	82	82
query85	1118	306	290	290
query86	316	301	276	276
query87	4448	4166	4171	4166
query88	3597	2409	2355	2355
query89	387	315	309	309
query90	2166	191	190	190
query91	180	152	146	146
query92	62	50	52	50
query93	1063	559	554	554
query94	939	295	297	295
query95	365	257	261	257
query96	605	281	282	281
query97	3255	3167	3137	3137
query98	209	217	193	193
query99	1528	1316	1279	1279
Total cold run time: 296334 ms
Total hot run time: 189524 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.21 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 41fb483cd584c467bd0d3b86a80cf991ba223926, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.03
query3	0.24	0.08	0.07
query4	1.63	0.11	0.11
query5	0.51	0.52	0.52
query6	1.14	0.74	0.72
query7	0.02	0.01	0.01
query8	0.04	0.03	0.04
query9	0.56	0.50	0.52
query10	0.54	0.54	0.54
query11	0.14	0.10	0.10
query12	0.14	0.11	0.13
query13	0.59	0.59	0.59
query14	0.79	0.80	0.81
query15	0.85	0.82	0.82
query16	0.39	0.39	0.38
query17	1.01	1.03	1.06
query18	0.23	0.21	0.22
query19	1.93	1.86	1.88
query20	0.02	0.01	0.01
query21	15.40	0.60	0.57
query22	2.37	1.83	1.69
query23	17.19	0.91	0.81
query24	3.01	1.71	0.99
query25	0.29	0.21	0.15
query26	0.21	0.14	0.13
query27	0.05	0.04	0.03
query28	9.83	0.53	0.52
query29	12.60	3.24	3.24
query30	0.24	0.06	0.07
query31	2.85	0.40	0.37
query32	3.26	0.45	0.45
query33	2.97	3.02	3.01
query34	17.16	4.45	4.46
query35	4.50	4.49	4.55
query36	0.67	0.47	0.47
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.02
query40	0.16	0.12	0.13
query41	0.07	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.93 s
Total hot run time: 30.21 s

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 100.00% (4/4) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 41.07% (10859/26443)
Line Coverage 31.87% (92696/290869)
Region Coverage 30.95% (47806/154462)
Branch Coverage 27.42% (24479/89258)

Copy link
Member

@eldenmoon eldenmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 27, 2025
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 6d8a113 into apache:branch-3.0 May 27, 2025
21 of 23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants