Skip to content

[chore](info) Record data_footprint and total_data_size in SegmentFooterPB and ColumnMetaPB #51001

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 1 commit into
base: master
Choose a base branch
from
Draft

[chore](info) Record data_footprint and total_data_size in SegmentFooterPB and ColumnMetaPB #51001

wants to merge 1 commit into from
+265 −16

Conversation

wyxxxcat
Copy link
Contributor

@wyxxxcat wyxxxcat commented May 19, 2025 •
edited
Loading

What problem does this PR solve?

  1. estimate every column data size in ColumnMetaPB and total column data size in segmentFooterPb when writing
  2. increase when rs compaction

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented May 19, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@wyxxxcat wyxxxcat force-pushed the total_column_data_columnMeta branch from 6f0a488 to 4220956 Compare May 19, 2025 02:27
zhannngchen
zhannngchen reviewed May 19, 2025
View reviewed changes
zhannngchen
zhannngchen reviewed May 19, 2025
View reviewed changes
@wyxxxcat wyxxxcat force-pushed the total_column_data_columnMeta branch 2 times, most recently from eca5573 to 7f503aa Compare May 19, 2025 09:47
@wyxxxcat wyxxxcat marked this pull request as ready for review May 19, 2025 09:47
@wyxxxcat wyxxxcat requested a review from zhannngchen May 19, 2025 10:01
@wyxxxcat
Copy link
Contributor Author

run buildall

@wyxxxcat wyxxxcat force-pushed the total_column_data_columnMeta branch from 7f503aa to c8f569a Compare May 19, 2025 10:02
@wyxxxcat
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 83.31% (1113/1336)
Line Coverage 66.20% (18660/28187)
Region Coverage 65.79% (9260/14076)
Branch Coverage 55.58% (4982/8964)

@doris-robot
Copy link

TPC-H: Total hot run time: 34139 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c8f569a243eb8819215910c5a5b9adb7d23fa7c7, data reload: false

------ Round 1 ----------------------------------
q1	26493	5075	4956	4956
q2	2083	280	193	193
q3	10556	1276	708	708
q4	10231	1031	545	545
q5	7771	2351	2386	2351
q6	189	166	133	133
q7	908	755	632	632
q8	9334	1372	1255	1255
q9	6832	5064	5091	5064
q10	6882	2334	1932	1932
q11	503	298	279	279
q12	352	356	217	217
q13	17764	3694	3097	3097
q14	234	226	214	214
q15	533	486	488	486
q16	440	442	372	372
q17	608	861	365	365
q18	7515	7269	7218	7218
q19	1472	958	551	551
q20	334	340	218	218
q21	3960	3173	2380	2380
q22	1089	983	973	973
Total cold run time: 116083 ms
Total hot run time: 34139 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5155	5045	5097	5045
q2	243	340	233	233
q3	2210	2648	2312	2312
q4	1333	1790	1403	1403
q5	4442	4417	4437	4417
q6	209	174	128	128
q7	2038	1977	1806	1806
q8	2637	2743	2534	2534
q9	7186	7162	7050	7050
q10	3071	3187	2756	2756
q11	595	519	493	493
q12	667	777	601	601
q13	3499	3928	3303	3303
q14	295	319	302	302
q15	527	492	480	480
q16	446	491	454	454
q17	1190	1545	1448	1448
q18	7782	7678	7490	7490
q19	836	949	1107	949
q20	2015	2067	1850	1850
q21	4752	4447	4447	4447
q22	1117	1067	1054	1054
Total cold run time: 52245 ms
Total hot run time: 50555 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193869 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c8f569a243eb8819215910c5a5b9adb7d23fa7c7, data reload: false

query1	1384	1110	1046	1046
query2	6201	1803	1785	1785
query3	11057	4537	4504	4504
query4	55373	25596	23161	23161
query5	5054	581	485	485
query6	344	244	195	195
query7	4920	519	305	305
query8	308	272	231	231
query9	5329	2673	2673	2673
query10	438	341	280	280
query11	15101	14973	14899	14899
query12	162	114	103	103
query13	1016	557	404	404
query14	10291	6435	6627	6435
query15	206	194	177	177
query16	7114	672	496	496
query17	1099	719	618	618
query18	1596	416	321	321
query19	204	202	181	181
query20	136	123	116	116
query21	205	135	113	113
query22	4487	4519	4335	4335
query23	34295	33774	33634	33634
query24	6467	2499	2522	2499
query25	535	487	421	421
query26	680	284	156	156
query27	2232	522	366	366
query28	3277	2169	2185	2169
query29	607	578	474	474
query30	287	220	196	196
query31	843	884	788	788
query32	74	69	72	69
query33	450	386	323	323
query34	820	929	560	560
query35	828	837	737	737
query36	963	1014	901	901
query37	119	105	82	82
query38	4234	4325	4262	4262
query39	1533	1457	1501	1457
query40	217	121	149	121
query41	59	66	53	53
query42	132	110	113	110
query43	505	520	481	481
query44	1435	885	869	869
query45	182	175	170	170
query46	891	1062	674	674
query47	1793	1870	1783	1783
query48	419	463	358	358
query49	677	541	448	448
query50	685	697	412	412
query51	4287	4268	4262	4262
query52	114	118	104	104
query53	241	268	195	195
query54	604	603	535	535
query55	90	87	92	87
query56	334	317	325	317
query57	1201	1199	1129	1129
query58	280	283	266	266
query59	2682	2765	2681	2681
query60	386	350	344	344
query61	149	164	122	122
query62	711	736	654	654
query63	233	190	187	187
query64	1486	1020	714	714
query65	4323	4252	4285	4252
query66	705	401	311	311
query67	16130	15762	15230	15230
query68	6535	910	547	547
query69	550	324	286	286
query70	1173	1112	1116	1112
query71	463	344	305	305
query72	5915	4924	5067	4924
query73	1341	739	367	367
query74	9309	9042	8911	8911
query75	3247	3207	2733	2733
query76	3803	1198	780	780
query77	530	376	292	292
query78	10206	10156	9373	9373
query79	2950	835	599	599
query80	818	527	461	461
query81	491	261	221	221
query82	544	127	99	99
query83	330	258	245	245
query84	306	110	90	90
query85	775	366	316	316
query86	384	297	287	287
query87	4538	4449	4421	4421
query88	3451	2389	2369	2369
query89	406	332	285	285
query90	1700	214	206	206
query91	138	141	123	123
query92	76	59	61	59
query93	2542	969	587	587
query94	728	402	276	276
query95	372	299	280	280
query96	494	579	290	290
query97	2761	2777	2694	2694
query98	239	208	202	202
query99	1430	1408	1303	1303
Total cold run time: 299765 ms
Total hot run time: 193869 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.99 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c8f569a243eb8819215910c5a5b9adb7d23fa7c7, data reload: false

query1	0.04	0.04	0.03
query2	0.15	0.10	0.11
query3	0.35	0.20	0.21
query4	1.60	0.21	0.21
query5	0.48	0.46	0.46
query6	1.17	0.65	0.66
query7	0.02	0.01	0.02
query8	0.06	0.06	0.05
query9	0.62	0.52	0.54
query10	0.59	0.59	0.58
query11	0.26	0.13	0.12
query12	0.26	0.13	0.14
query13	0.64	0.62	0.63
query14	0.80	0.81	0.84
query15	0.98	0.90	0.87
query16	0.38	0.37	0.38
query17	1.04	1.07	1.06
query18	0.19	0.19	0.19
query19	1.99	1.91	1.84
query20	0.02	0.01	0.01
query21	15.50	0.97	0.68
query22	0.94	1.03	0.84
query23	14.69	1.62	0.77
query24	5.41	0.57	0.28
query25	0.17	0.10	0.08
query26	0.55	0.23	0.19
query27	0.09	0.08	0.09
query28	11.01	1.22	0.59
query29	12.54	4.06	3.42
query30	0.29	0.09	0.07
query31	2.84	0.63	0.44
query32	3.23	0.60	0.51
query33	3.14	3.13	3.19
query34	16.34	5.13	4.46
query35	4.44	4.50	4.48
query36	0.63	0.51	0.49
query37	0.20	0.17	0.16
query38	0.17	0.16	0.15
query39	0.05	0.05	0.04
query40	0.20	0.16	0.15
query41	0.10	0.06	0.05
query42	0.06	0.06	0.05
query43	0.06	0.05	0.05
Total cold run time: 104.29 s
Total hot run time: 29.99 s

@wyxxxcat wyxxxcat force-pushed the total_column_data_columnMeta branch from c8f569a to 46aefdf Compare May 20, 2025 03:11
@wyxxxcat wyxxxcat changed the title [chore](info) Record data_footprint and total_data_size in SegmentFooterPB and ColumnMeaPB [chore](info) Record data_footprint and total_data_size in SegmentFooterPB and ColumnMetaPB May 20, 2025
@wyxxxcat
Copy link
Contributor Author

run buildall

@hello-stephen
Copy link
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 83.31% (1113/1336)
Line Coverage 66.28% (18681/28187)
Region Coverage 65.88% (9273/14076)
Branch Coverage 55.70% (4993/8964)

@doris-robot
Copy link

TPC-H: Total hot run time: 34182 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 46aefdf676b089c7a32d30db93252cf71fefda34, data reload: false

------ Round 1 ----------------------------------
q1	26665	5062	5032	5032
q2	2093	289	188	188
q3	10499	1286	733	733
q4	10239	1039	548	548
q5	7627	2331	2363	2331
q6	191	167	138	138
q7	958	750	632	632
q8	9333	1306	1126	1126
q9	6750	5096	5108	5096
q10	6915	2351	1922	1922
q11	491	293	279	279
q12	360	362	221	221
q13	17766	3735	3139	3139
q14	242	223	215	215
q15	548	480	484	480
q16	436	434	384	384
q17	598	850	379	379
q18	7861	7176	7169	7169
q19	1826	972	564	564
q20	344	334	221	221
q21	4153	3197	2405	2405
q22	1073	1016	980	980
Total cold run time: 116968 ms
Total hot run time: 34182 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5234	5184	5103	5103
q2	246	336	232	232
q3	2183	2689	2324	2324
q4	1381	1879	1467	1467
q5	4483	4500	4413	4413
q6	221	173	126	126
q7	2037	1960	1778	1778
q8	2611	2564	2619	2564
q9	7300	7003	7145	7003
q10	3047	3264	2781	2781
q11	576	516	511	511
q12	700	767	626	626
q13	3585	3917	3385	3385
q14	277	307	271	271
q15	546	517	497	497
q16	443	484	445	445
q17	1176	1494	1422	1422
q18	7863	7728	7436	7436
q19	846	891	928	891
q20	1958	1977	1858	1858
q21	4957	4530	4502	4502
q22	1086	1059	1046	1046
Total cold run time: 52756 ms
Total hot run time: 50681 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193411 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 46aefdf676b089c7a32d30db93252cf71fefda34, data reload: false

query1	1418	1095	1058	1058
query2	6383	1808	1783	1783
query3	11050	4563	4349	4349
query4	52462	24876	23402	23402
query5	5020	546	448	448
query6	338	202	195	195
query7	4869	506	301	301
query8	298	243	228	228
query9	5520	2658	2635	2635
query10	422	328	273	273
query11	15084	15050	15179	15050
query12	156	115	102	102
query13	1027	522	400	400
query14	10176	6288	6390	6288
query15	225	203	178	178
query16	7107	645	507	507
query17	1110	795	616	616
query18	1582	418	324	324
query19	207	194	182	182
query20	139	135	121	121
query21	216	133	115	115
query22	4386	4408	4265	4265
query23	34590	33716	33531	33531
query24	6517	2445	2418	2418
query25	481	491	408	408
query26	685	279	161	161
query27	2325	519	343	343
query28	3158	2153	2156	2153
query29	596	614	443	443
query30	275	224	190	190
query31	882	887	788	788
query32	76	71	68	68
query33	450	366	334	334
query34	781	881	569	569
query35	798	865	753	753
query36	974	999	902	902
query37	109	98	76	76
query38	4224	4352	4264	4264
query39	1548	1504	1453	1453
query40	211	126	111	111
query41	58	58	55	55
query42	129	117	120	117
query43	495	496	491	491
query44	1414	874	866	866
query45	205	177	182	177
query46	858	1035	660	660
query47	1829	1881	1826	1826
query48	403	446	350	350
query49	715	505	443	443
query50	688	701	416	416
query51	4227	4323	4138	4138
query52	119	120	103	103
query53	240	270	193	193
query54	583	593	524	524
query55	94	92	88	88
query56	317	332	321	321
query57	1175	1234	1162	1162
query58	282	281	269	269
query59	2656	2811	2621	2621
query60	372	345	345	345
query61	168	150	144	144
query62	776	767	705	705
query63	239	202	201	201
query64	1521	1054	672	672
query65	4325	4241	4214	4214
query66	723	405	302	302
query67	15963	15682	15662	15662
query68	7257	903	530	530
query69	559	331	284	284
query70	1168	1126	1121	1121
query71	507	331	304	304
query72	5503	4706	4644	4644
query73	1456	592	357	357
query74	8972	8950	9084	8950
query75	3814	3180	2672	2672
query76	4232	1210	770	770
query77	615	390	299	299
query78	10106	10149	9347	9347
query79	2181	836	577	577
query80	591	510	533	510
query81	485	251	222	222
query82	438	127	101	101
query83	254	252	236	236
query84	307	107	88	88
query85	801	351	309	309
query86	392	292	267	267
query87	4495	4484	4434	4434
query88	3543	2336	2323	2323
query89	414	323	292	292
query90	1824	210	205	205
query91	139	146	114	114
query92	72	61	59	59
query93	1731	963	596	596
query94	675	415	305	305
query95	373	304	281	281
query96	503	581	284	284
query97	2772	2773	2608	2608
query98	228	235	202	202
query99	1526	1382	1270	1270
Total cold run time: 296382 ms
Total hot run time: 193411 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.74 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 46aefdf676b089c7a32d30db93252cf71fefda34, data reload: false

query1	0.04	0.03	0.03
query2	0.15	0.10	0.11
query3	0.35	0.21	0.20
query4	1.59	0.21	0.09
query5	0.43	0.42	0.42
query6	1.17	0.67	0.66
query7	0.02	0.02	0.02
query8	0.05	0.05	0.04
query9	0.62	0.52	0.51
query10	0.57	0.58	0.57
query11	0.26	0.13	0.14
query12	0.25	0.14	0.13
query13	0.64	0.62	0.62
query14	0.80	0.82	0.84
query15	0.98	0.89	0.88
query16	0.38	0.38	0.38
query17	1.07	1.06	1.06
query18	0.18	0.18	0.17
query19	1.93	1.79	1.80
query20	0.01	0.02	0.01
query21	15.40	0.97	0.67
query22	0.93	1.06	0.84
query23	14.70	1.53	0.78
query24	5.52	0.55	0.28
query25	0.17	0.09	0.08
query26	0.55	0.22	0.19
query27	0.09	0.08	0.08
query28	11.00	1.20	0.58
query29	12.52	4.14	3.46
query30	0.28	0.09	0.06
query31	2.80	0.62	0.43
query32	3.23	0.59	0.50
query33	3.10	3.13	3.10
query34	16.46	5.10	4.44
query35	4.49	4.46	4.50
query36	0.64	0.51	0.51
query37	0.21	0.18	0.17
query38	0.17	0.15	0.15
query39	0.05	0.05	0.04
query40	0.22	0.16	0.15
query41	0.11	0.05	0.06
query42	0.06	0.05	0.05
query43	0.05	0.06	0.04
Total cold run time: 104.24 s
Total hot run time: 29.74 s

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 100.00% (20/20) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 55.95% (14924/26674)
Line Coverage 44.77% (132423/295773)
Region Coverage 43.86% (66615/151878)
Branch Coverage 38.45% (34133/88778)

@wyxxxcat
Copy link
Contributor Author

run p0

@wyxxxcat
Copy link
Contributor Author

run cloud_p0

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (20/20) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 79.39% (20847/26258)
Line Coverage 72.59% (214712/295770)
Region Coverage 70.76% (126262/178430)
Branch Coverage 64.51% (65421/101412)

@wyxxxcat wyxxxcat force-pushed the total_column_data_columnMeta branch from 46aefdf to c0472c7 Compare May 20, 2025 10:27
@wyxxxcat
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 83.31% (1113/1336)
Line Coverage 66.25% (18674/28187)
Region Coverage 65.85% (9269/14076)
Branch Coverage 55.61% (4985/8964)

@doris-robot
Copy link

TPC-H: Total hot run time: 33983 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c0472c79d10037f66e33aacedc6f1313f96ac2a8, data reload: false

------ Round 1 ----------------------------------
q1	26175	5062	5003	5003
q2	2089	311	203	203
q3	10563	1261	715	715
q4	10247	996	511	511
q5	7966	2423	2339	2339
q6	189	169	140	140
q7	926	744	629	629
q8	9332	1282	1151	1151
q9	6876	5101	5117	5101
q10	6877	2337	1890	1890
q11	491	290	272	272
q12	355	352	218	218
q13	17779	3623	3105	3105
q14	232	227	210	210
q15	536	490	470	470
q16	427	443	384	384
q17	625	878	368	368
q18	7824	7223	7143	7143
q19	1800	951	549	549
q20	331	348	225	225
q21	3961	2603	2375	2375
q22	1057	992	982	982
Total cold run time: 116658 ms
Total hot run time: 33983 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5199	5274	5132	5132
q2	238	331	227	227
q3	2160	2663	2300	2300
q4	1340	1775	1410	1410
q5	4614	4473	4398	4398
q6	219	173	128	128
q7	1952	1930	1709	1709
q8	2607	2658	2601	2601
q9	7172	7163	7161	7161
q10	2980	3150	2754	2754
q11	573	497	517	497
q12	684	762	645	645
q13	3503	3825	3286	3286
q14	277	298	272	272
q15	517	477	476	476
q16	458	475	429	429
q17	1150	1478	1419	1419
q18	7792	7470	7399	7399
q19	868	886	1052	886
q20	1969	2010	1901	1901
q21	4730	4373	4299	4299
q22	1078	997	987	987
Total cold run time: 52080 ms
Total hot run time: 50316 ms

@wyxxxcat wyxxxcat force-pushed the total_column_data_columnMeta branch 3 times, most recently from c3a5141 to c7f7483 Compare May 21, 2025 10:56
@wyxxxcat wyxxxcat requested a review from zhannngchen May 21, 2025 10:57
@wyxxxcat
Copy link
Contributor Author

run buildall

@hello-stephen
Copy link
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 83.25% (1113/1337)
Line Coverage 66.19% (18682/28223)
Region Coverage 65.81% (9264/14077)
Branch Coverage 55.61% (4985/8964)

@doris-robot
Copy link

TPC-H: Total hot run time: 34076 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c7f748323d3e664e5add4a06d1b3c1f6882078ea, data reload: false

------ Round 1 ----------------------------------
q1	26336	5164	5086	5086
q2	2078	272	188	188
q3	10582	1267	711	711
q4	10232	983	515	515
q5	7772	2265	2435	2265
q6	189	164	135	135
q7	923	761	633	633
q8	9336	1303	1124	1124
q9	6802	5059	5087	5059
q10	6816	2316	1898	1898
q11	477	286	289	286
q12	348	352	223	223
q13	17791	3670	3123	3123
q14	235	240	224	224
q15	547	504	503	503
q16	439	433	379	379
q17	601	859	371	371
q18	7815	7227	7150	7150
q19	1813	946	583	583
q20	338	340	226	226
q21	3739	3267	2414	2414
q22	1056	997	980	980
Total cold run time: 116265 ms
Total hot run time: 34076 ms

----- Round 2, with runtime_filter_mode=off -----
version_comment	Doris version doris-0.0.0--b1b46766a7	Doris version doris-0.0.0--b1b46766a7	0
wait_timeout	28800	28800	0
workload_group			0
============================================
show table status;
Name	Engine	Version	Row_format	Rows	Avg_row_length	Data_length	Max_data_length	Index_length	Data_free	Auto_increment	Create_time	Update_time	Check_time	Collation	Checksum	Create_options	Comment
revenue0	View	NULL	NULL	-1	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:24	NULL	NULL	utf-8	NULL	NULL	
partsupp	Doris	NULL	NULL	80000000	56	4534120086	NULL	44625495	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:44:20	NULL	utf-8	NULL	NULL	
part	Doris	NULL	NULL	20000000	37	748811035	NULL	1935627	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:27:57	NULL	utf-8	NULL	NULL	
nation	Doris	NULL	NULL	25	138	3473	NULL	366	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:27:24	NULL	utf-8	NULL	NULL	
customer	Doris	NULL	NULL	15000000	92	1381653732	NULL	4374759	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:27:43	NULL	utf-8	NULL	NULL	
lineitem	Doris	NULL	NULL	600037902	33	19843441616	NULL	61784740	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:38:59	NULL	utf-8	NULL	NULL	
supplier	Doris	NULL	NULL	1000000	87	87519212	NULL	194931	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:27:25	NULL	utf-8	NULL	NULL	
region	Doris	NULL	NULL	5	240	1201	NULL	147	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:27:24	NULL	utf-8	NULL	NULL	
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	5287	5125	5120	5120
q2	235	329	238	238
q3	2162	2658	2259	2259
q4	1381	1815	1460	1460
q5	4451	4439	4415	4415
q6	216	172	131	131
q7	q8	q9	q10	q11	q12	q13	q14	q15	q16	q17	q18	q19	q20	q21	q22	Total cold run time: 13732 ms
Total hot run time: 13623 ms

@wyxxxcat wyxxxcat force-pushed the total_column_data_columnMeta branch from c7f7483 to d3ad704 Compare May 22, 2025 02:05
zhannngchen
zhannngchen reviewed May 23, 2025
View reviewed changes
zhannngchen
zhannngchen reviewed May 23, 2025
View reviewed changes
zhannngchen
zhannngchen reviewed May 23, 2025
View reviewed changes
zhannngchen
zhannngchen reviewed May 23, 2025
View reviewed changes
@wyxxxcat wyxxxcat force-pushed the total_column_data_columnMeta branch from d3ad704 to c072b09 Compare May 26, 2025 06:57
zhannngchen
zhannngchen previously approved these changes May 27, 2025
View reviewed changes
Copy link
Contributor

@zhannngchen zhannngchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zhannngchen
Copy link
Contributor

run buildall

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 27, 2025
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 83.25% (1113/1337)
Line Coverage 66.13% (18664/28224)
Region Coverage 65.80% (9264/14079)
Branch Coverage 55.53% (4979/8966)

@doris-robot
Copy link

TPC-H: Total hot run time: 33837 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c072b09b56f0fd649cdfaac0f1faa190b2fef4f3, data reload: false

------ Round 1 ----------------------------------
q1	26243	5053	4987	4987
q2	2081	279	181	181
q3	10542	1273	699	699
q4	10226	1004	521	521
q5	7749	2419	2338	2338
q6	181	160	130	130
q7	923	746	616	616
q8	9319	1280	1077	1077
q9	6844	5118	5087	5087
q10	6869	2334	1893	1893
q11	496	286	271	271
q12	347	351	220	220
q13	17800	3647	3075	3075
q14	242	221	216	216
q15	545	479	496	479
q16	430	438	375	375
q17	615	855	382	382
q18	7788	7259	7143	7143
q19	1707	984	581	581
q20	332	340	234	234
q21	3724	3233	2396	2396
q22	996	1001	936	936
Total cold run time: 115999 ms
Total hot run time: 33837 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
show table status;
Name	Engine	Version	Row_format	Rows	Avg_row_length	Data_length	Max_data_length	Index_length	Data_free	Auto_increment	Create_time	Update_time	Check_time	Collation	Checksum	Create_options	Comment
revenue0	View	NULL	NULL	-1	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:24	NULL	NULL	utf-8	NULL	NULL	
partsupp	Doris	NULL	NULL	80000000	56	4534120086	NULL	44625495	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:44:20	NULL	utf-8	NULL	NULL	
part	Doris	NULL	NULL	20000000	37	748811035	NULL	1935627	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:27:57	NULL	utf-8	NULL	NULL	
nation	Doris	NULL	NULL	25	138	3473	NULL	366	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:27:24	NULL	utf-8	NULL	NULL	
customer	Doris	NULL	NULL	15000000	92	1381653732	NULL	4374759	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:27:43	NULL	utf-8	NULL	NULL	
lineitem	Doris	NULL	NULL	600037902	33	19843441616	NULL	61784740	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:38:59	NULL	utf-8	NULL	NULL	
supplier	Doris	NULL	NULL	1000000	87	87519212	NULL	194931	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:27:25	NULL	utf-8	NULL	NULL	
region	Doris	NULL	NULL	5	240	1201	NULL	147	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:27:24	NULL	utf-8	NULL	NULL	
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	5171	5095	5095	5095
q2	243	331	229	229
q3	2134	2682	2304	2304
q4	1349	1765	1335	1335
q5	4596	4427	4387	4387
q6	214	166	127	127
q7	2025	1977	1740	1740
q8	2589	2526	2517	2517
q9	7202	7186	7067	7067
q10	3001	3230	q11	q12	q13	q14	q15	q16	q17	q18	q19	q20	q21	q22	Total cold run time: 25523 ms
Total hot run time: 24801 ms

@wyxxxcat wyxxxcat marked this pull request as draft May 28, 2025 06:39
@wyxxxcat wyxxxcat force-pushed the total_column_data_columnMeta branch from c072b09 to b061f31 Compare June 2, 2025 13:32
@wyxxxcat wyxxxcat force-pushed the total_column_data_columnMeta branch from b061f31 to dc50aec Compare June 2, 2025 13:35
@wyxxxcat wyxxxcat force-pushed the total_column_data_columnMeta branch from dc50aec to a439492 Compare June 2, 2025 13:36
@wyxxxcat
Copy link
Contributor Author

wyxxxcat commented Jun 2, 2025

run buildall

@doris-robot
Copy link

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 83.31% (1118/1342)
Line Coverage 66.75% (19132/28663)
Region Coverage 66.38% (9467/14261)
Branch Coverage 56.31% (5128/9106)

@doris-robot
Copy link

TPC-H: Total hot run time: 34498 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a439492bd3bfb6fafdab904ac61d3030eb79ddf9, data reload: false

------ Round 1 ----------------------------------
q1	26263	5149	5075	5075
q2	1968	289	185	185
q3	10380	1250	755	755
q4	10242	995	548	548
q5	7632	2544	2391	2391
q6	184	162	136	136
q7	924	767	627	627
q8	9302	1338	1148	1148
q9	6732	5078	5152	5078
q10	6855	2311	1942	1942
q11	492	317	283	283
q12	352	367	225	225
q13	17804	3706	3145	3145
q14	237	236	207	207
q15	557	485	494	485
q16	453	443	378	378
q17	627	877	385	385
q18	7690	7238	7234	7234
q19	1829	968	594	594
q20	342	342	230	230
q21	3912	3231	2485	2485
q22	1056	989	962	962
Total cold run time: 115833 ms
Total hot run time: 34498 ms

----- Round 2, with runtime_filter_mode=off -----
version_comment	Doris version doris-0.0.0--a439492bd3	Doris version doris-0.0.0--a439492bd3	0
wait_timeout	28800	28800	0
workload_group			0
============================================
show table status;
Name	Engine	Version	Row_format	Rows	Avg_row_length	Data_length	Max_data_length	Index_length	Data_free	Auto_increment	Create_time	Update_time	Check_time	Collation	Checksum	Create_options	Comment
revenue0	View	NULL	NULL	-1	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:24	NULL	NULL	utf-8	NULL	NULL	
partsupp	Doris	NULL	NULL	80000000	56	4534120086	NULL	44625495	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:44:20	NULL	utf-8	NULL	NULL	
part	Doris	NULL	NULL	20000000	37	748811035	NULL	1935627	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:27:57	NULL	utf-8	NULL	NULL	
nation	Doris	NULL	NULL	25	138	3473	NULL	366	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:27:24	NULL	utf-8	NULL	NULL	
customer	Doris	NULL	NULL	15000000	92	1381653732	NULL	4374759	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:27:43	NULL	utf-8	NULL	NULL	
lineitem	Doris	NULL	NULL	600037902	33	19843441616	NULL	61784740	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:38:59	NULL	utf-8	NULL	NULL	
supplier	Doris	NULL	NULL	1000000	87	87519212	NULL	194931	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:27:25	NULL	utf-8	NULL	NULL	
region	Doris	NULL	NULL	5	240	1201	NULL	147	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:27:24	NULL	utf-8	NULL	NULL	
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	5234	5133	5205	5133
q2	243	316	222	222
q3	2159	2692	2343	2343
q4	1349	1830	1482	1482
q5	4468	4409	4409	4409
q6	234	170	131	131
q7	2136	q8	q9	q10	q11	q12	q13	q14	q15	q16	q17	q18	q19	q20	q21	q22	Total cold run time: 13687 ms
Total hot run time: 13720 ms

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zhannngchen zhannngchen zhannngchen left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@wyxxxcat @Thearas @doris-robot @hello-stephen @zhannngchen