Skip to content

[fix](mow) remove delete bitmap when remove unused rowsets #50973

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Jun 10, 2025
Merged

[fix](mow) remove delete bitmap when remove unused rowsets #50973

merged 3 commits into from
Jun 10, 2025

Conversation

mymeiyi
Copy link
Contributor

@mymeiyi mymeiyi commented May 16, 2025 •
edited
Loading

What problem does this PR solve?

now, the delete bitmap is removed when move stale rowsets to unused rowsets, but if some queries is still read the stale rowsets, the query result may get duplicated keys.
this pr change the remove of delete bitmap when remove unused rowsets.
this pr is based on #49383

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@mymeiyi mymeiyi force-pushed the delete-dm-of-stale branch 2 times, most recently from 99861a4 to b61a0d4 Compare May 19, 2025 07:09
@mymeiyi
Copy link
Contributor Author

mymeiyi commented May 19, 2025

run buildall

@doris-robot
Copy link

Cloud UT Coverage Report

Increment line coverage 85.61% (238/278) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 83.35% (1116/1339)
Line Coverage 66.55% (18982/28522)
Region Coverage 66.02% (9383/14212)
Branch Coverage 55.88% (5069/9072)

@doris-robot
Copy link

TPC-H: Total hot run time: 33516 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b61a0d4cc454ce4dcdd433575c7d115f46265ff2, data reload: false

------ Round 1 ----------------------------------
q1	26429	5936	4983	4983
q2	2086	278	183	183
q3	10596	1254	689	689
q4	10221	1023	520	520
q5	7751	2358	2324	2324
q6	186	169	135	135
q7	917	744	611	611
q8	9320	1277	1071	1071
q9	6855	5087	5144	5087
q10	6836	2328	1865	1865
q11	497	311	262	262
q12	352	345	210	210
q13	17768	3642	3050	3050
q14	236	227	213	213
q15	527	476	486	476
q16	436	432	385	385
q17	605	864	365	365
q18	7525	7122	7040	7040
q19	1741	963	553	553
q20	338	335	226	226
q21	3773	3130	2317	2317
q22	999	998	951	951
Total cold run time: 115994 ms
Total hot run time: 33516 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5107	5034	5075	5034
q2	248	329	229	229
q3	2152	2687	2324	2324
q4	1356	1784	1343	1343
q5	4448	4411	4423	4411
q6	215	168	128	128
q7	1948	1946	1784	1784
q8	2548	2471	2526	2471
q9	7279	7164	7152	7152
q10	3001	3186	2779	2779
q11	579	510	510	510
q12	686	761	635	635
q13	3531	3924	3318	3318
q14	300	296	282	282
q15	551	503	491	491
q16	452	503	461	461
q17	1155	1528	1395	1395
q18	7794	7592	7365	7365
q19	783	862	1083	862
q20	1939	1972	1837	1837
q21	4786	4513	4286	4286
q22	1094	1068	1014	1014
Total cold run time: 51952 ms
Total hot run time: 50111 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192162 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b61a0d4cc454ce4dcdd433575c7d115f46265ff2, data reload: false

query1	1406	1098	1081	1081
query2	6365	1793	1784	1784
query3	10966	4459	4407	4407
query4	52074	25635	22977	22977
query5	5169	576	437	437
query6	340	202	210	202
query7	4915	493	295	295
query8	320	265	255	255
query9	5742	2653	2659	2653
query10	452	333	280	280
query11	15059	15091	14788	14788
query12	159	110	110	110
query13	1076	541	417	417
query14	10208	6270	6164	6164
query15	208	201	178	178
query16	7058	657	523	523
query17	1110	758	591	591
query18	1588	422	328	328
query19	216	207	180	180
query20	130	131	124	124
query21	217	131	105	105
query22	4334	4629	4416	4416
query23	34500	33401	33661	33401
query24	6573	2424	2428	2424
query25	470	469	406	406
query26	713	282	162	162
query27	2392	523	347	347
query28	3220	2168	2156	2156
query29	586	559	435	435
query30	268	225	186	186
query31	866	860	800	800
query32	71	63	64	63
query33	469	374	323	323
query34	787	870	541	541
query35	785	838	762	762
query36	924	1001	909	909
query37	118	107	72	72
query38	4261	4284	4254	4254
query39	1550	1448	1470	1448
query40	224	121	110	110
query41	57	53	56	53
query42	123	119	118	118
query43	520	513	483	483
query44	1384	869	854	854
query45	183	177	169	169
query46	862	1022	660	660
query47	1835	1910	1846	1846
query48	402	450	336	336
query49	688	513	418	418
query50	680	689	414	414
query51	4196	4284	4206	4206
query52	118	108	100	100
query53	236	256	184	184
query54	587	583	522	522
query55	87	90	82	82
query56	311	307	322	307
query57	1183	1200	1116	1116
query58	270	275	268	268
query59	2724	2704	2688	2688
query60	339	354	319	319
query61	146	125	122	122
query62	737	758	675	675
query63	228	187	191	187
query64	1833	1033	726	726
query65	4332	4209	4229	4209
query66	746	394	300	300
query67	15819	15509	15315	15315
query68	6821	877	528	528
query69	539	310	325	310
query70	1197	1102	1102	1102
query71	506	320	307	307
query72	5910	4734	4775	4734
query73	1419	625	353	353
query74	9286	9216	8868	8868
query75	3967	3220	2671	2671
query76	4337	1196	751	751
query77	689	369	287	287
query78	10063	9909	9312	9312
query79	5625	805	555	555
query80	681	535	455	455
query81	489	251	214	214
query82	633	127	94	94
query83	318	253	237	237
query84	301	109	89	89
query85	773	349	313	313
query86	371	293	281	281
query87	4468	4423	4334	4334
query88	3857	2324	2308	2308
query89	447	317	293	293
query90	1848	213	210	210
query91	148	142	113	113
query92	73	61	57	57
query93	3353	934	566	566
query94	661	408	304	304
query95	367	293	294	293
query96	510	570	290	290
query97	2766	2826	2646	2646
query98	243	206	204	204
query99	1420	1396	1312	1312
Total cold run time: 302601 ms
Total hot run time: 192162 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.84 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b61a0d4cc454ce4dcdd433575c7d115f46265ff2, data reload: false

query1	0.04	0.03	0.03
query2	0.15	0.10	0.11
query3	0.34	0.20	0.20
query4	1.59	0.19	0.08
query5	0.43	0.41	0.41
query6	1.16	0.67	0.66
query7	0.03	0.02	0.02
query8	0.06	0.05	0.05
query9	0.62	0.53	0.53
query10	0.59	0.59	0.58
query11	0.26	0.13	0.12
query12	0.26	0.14	0.15
query13	0.64	0.63	0.63
query14	0.80	0.83	0.84
query15	0.96	0.88	0.89
query16	0.38	0.38	0.37
query17	1.06	1.06	1.04
query18	0.19	0.18	0.18
query19	2.14	1.81	1.86
query20	0.01	0.02	0.01
query21	15.40	0.98	0.67
query22	0.94	1.07	0.81
query23	14.69	1.53	0.76
query24	5.57	0.55	0.28
query25	0.17	0.08	0.09
query26	0.56	0.22	0.19
query27	0.09	0.08	0.08
query28	10.99	1.20	0.59
query29	12.58	4.15	3.48
query30	0.28	0.09	0.07
query31	2.82	0.65	0.45
query32	3.24	0.60	0.52
query33	3.09	3.11	3.17
query34	16.54	5.11	4.50
query35	4.50	4.42	4.44
query36	0.65	0.51	0.52
query37	0.19	0.17	0.18
query38	0.17	0.15	0.16
query39	0.05	0.04	0.04
query40	0.21	0.17	0.15
query41	0.10	0.06	0.05
query42	0.06	0.06	0.05
query43	0.05	0.05	0.04
Total cold run time: 104.65 s
Total hot run time: 29.84 s

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 1.17% (7/597) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 55.85% (14906/26689)
Line Coverage 44.64% (132285/296321)
Region Coverage 43.74% (66561/152166)
Branch Coverage 38.32% (34102/88984)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 28.69% (171/596) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 79.26% (20825/26273)
Line Coverage 72.47% (214745/296318)
Region Coverage 70.72% (126388/178726)
Branch Coverage 64.49% (65546/101636)

@mymeiyi mymeiyi force-pushed the delete-dm-of-stale branch from b61a0d4 to 4b31c15 Compare May 22, 2025 06:43
@mymeiyi
Copy link
Contributor Author

mymeiyi commented May 22, 2025

run buildall

@hello-stephen
Copy link
Contributor

Cloud UT Coverage Report

Increment line coverage 85.61% (238/278) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 83.28% (1116/1340)
Line Coverage 66.43% (18971/28558)
Region Coverage 65.97% (9377/14213)
Branch Coverage 55.86% (5068/9072)

@doris-robot
Copy link

TPC-H: Total hot run time: 34029 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4b31c1578a5d2a8a11d1ad0cce8666fb9e23800f, data reload: false

------ Round 1 ----------------------------------
q1	27026	5067	5027	5027
q2	2089	298	191	191
q3	10499	1258	685	685
q4	10245	992	525	525
q5	7707	2429	2317	2317
q6	187	162	130	130
q7	918	752	613	613
q8	9328	1308	1140	1140
q9	6806	5115	5096	5096
q10	6819	2311	1922	1922
q11	496	297	296	296
q12	356	348	210	210
q13	17778	3630	3082	3082
q14	227	230	210	210
q15	531	475	495	475
q16	426	439	383	383
q17	624	862	359	359
q18	7633	7371	7247	7247
q19	2457	957	563	563
q20	325	329	212	212
q21	4107	3227	2361	2361
q22	1048	1001	985	985
Total cold run time: 117632 ms
Total hot run time: 34029 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5256	5121	5082	5082
q2	240	332	229	229
q3	2184	2650	2291	2291
q4	1331	1880	1520	1520
q5	4558	4432	4369	4369
q6	214	170	126	126
q7	1964	1926	1772	1772
q8	2695	2518	2666	2518
q9	7171	7193	7121	7121
q10	2983	3161	2725	2725
q11	566	505	489	489
q12	668	772	633	633
q13	3524	3869	3270	3270
q14	276	307	257	257
q15	516	470	470	470
q16	431	484	461	461
q17	1171	1564	1400	1400
q18	7865	7644	7563	7563
q19	830	834	1011	834
q20	2015	2021	1813	1813
q21	4826	4331	4290	4290
q22	1063	1018	980	980
Total cold run time: 52347 ms
Total hot run time: 50213 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 187204 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4b31c1578a5d2a8a11d1ad0cce8666fb9e23800f, data reload: false

query1	1016	484	508	484
query2	6577	1900	1843	1843
query3	6769	227	228	227
query4	26780	23811	23473	23473
query5	4371	676	458	458
query6	312	197	200	197
query7	4620	504	289	289
query8	287	237	237	237
query9	8617	2616	2636	2616
query10	444	331	273	273
query11	15819	15129	14937	14937
query12	157	107	107	107
query13	1654	541	400	400
query14	8778	6470	6352	6352
query15	215	191	169	169
query16	7139	625	489	489
query17	1209	745	608	608
query18	1980	410	313	313
query19	198	199	172	172
query20	128	122	121	121
query21	211	131	111	111
query22	4105	4172	4126	4126
query23	34182	33186	33145	33145
query24	8482	2470	2439	2439
query25	581	500	440	440
query26	1256	286	160	160
query27	2734	515	343	343
query28	4368	2137	2120	2120
query29	812	590	457	457
query30	282	279	187	187
query31	951	856	781	781
query32	74	67	69	67
query33	563	402	311	311
query34	813	870	539	539
query35	785	797	740	740
query36	971	981	880	880
query37	115	106	78	78
query38	4150	4131	4080	4080
query39	1461	1485	1411	1411
query40	232	124	112	112
query41	58	61	52	52
query42	132	116	119	116
query43	518	508	486	486
query44	1351	813	803	803
query45	182	171	166	166
query46	893	1075	664	664
query47	1758	1795	1711	1711
query48	387	433	322	322
query49	780	520	435	435
query50	685	716	423	423
query51	4178	4244	4066	4066
query52	114	112	103	103
query53	228	258	191	191
query54	595	586	505	505
query55	82	90	85	85
query56	317	306	303	303
query57	1148	1165	1060	1060
query58	268	261	268	261
query59	2714	2745	2633	2633
query60	352	340	318	318
query61	128	127	128	127
query62	798	727	668	668
query63	234	191	195	191
query64	4406	1023	685	685
query65	4328	4284	4200	4200
query66	1147	413	322	322
query67	15775	15715	15414	15414
query68	8894	922	519	519
query69	471	376	277	277
query70	1305	1155	1099	1099
query71	463	334	310	310
query72	5295	4798	4932	4798
query73	750	704	361	361
query74	8847	9110	8644	8644
query75	4181	3194	2722	2722
query76	3628	1226	810	810
query77	814	383	306	306
query78	10019	10260	9438	9438
query79	1435	897	591	591
query80	643	546	506	506
query81	481	254	222	222
query82	189	128	102	102
query83	295	265	238	238
query84	303	106	90	90
query85	763	376	325	325
query86	334	329	304	304
query87	4385	4505	4369	4369
query88	2934	2250	2272	2250
query89	389	326	296	296
query90	2057	218	215	215
query91	151	156	114	114
query92	81	61	61	61
query93	1088	961	583	583
query94	674	415	301	301
query95	379	303	282	282
query96	515	583	290	290
query97	2665	2738	2688	2688
query98	225	230	203	203
query99	1428	1394	1288	1288
Total cold run time: 273835 ms
Total hot run time: 187204 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.64 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4b31c1578a5d2a8a11d1ad0cce8666fb9e23800f, data reload: false

query1	0.04	0.04	0.04
query2	0.14	0.10	0.12
query3	0.26	0.20	0.19
query4	1.60	0.18	0.20
query5	0.45	0.45	0.45
query6	1.15	0.66	0.66
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.59	0.52	0.55
query10	0.57	0.58	0.57
query11	0.15	0.11	0.10
query12	0.15	0.11	0.12
query13	0.61	0.60	0.59
query14	0.79	0.80	0.81
query15	0.88	0.86	0.86
query16	0.38	0.36	0.38
query17	1.02	1.02	1.08
query18	0.22	0.20	0.21
query19	1.92	1.84	1.81
query20	0.01	0.01	0.01
query21	15.44	0.91	0.53
query22	0.77	1.23	0.67
query23	14.84	1.37	0.60
query24	7.87	1.03	0.32
query25	0.30	0.10	0.08
query26	0.60	0.17	0.13
query27	0.05	0.04	0.05
query28	8.35	0.90	0.44
query29	12.56	4.03	3.42
query30	0.24	0.10	0.07
query31	2.80	0.59	0.38
query32	3.24	0.55	0.48
query33	3.06	3.05	3.12
query34	15.72	5.06	4.51
query35	4.51	4.52	4.50
query36	0.66	0.48	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.03
query43	0.04	0.04	0.03
Total cold run time: 102.48 s
Total hot run time: 28.64 s

@bobhan1 bobhan1 mentioned this pull request May 26, 2025
Closed
16 tasks
@mymeiyi mymeiyi force-pushed the delete-dm-of-stale branch 2 times, most recently from 90cacd7 to 793d140 Compare May 30, 2025 08:07
@mymeiyi
Copy link
Contributor Author

mymeiyi commented May 30, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33799 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 793d140161fab159842ce071fc85cfb7a97aa04a, data reload: false

------ Round 1 ----------------------------------
q1	26066	5135	5071	5071
q2	1953	275	174	174
q3	10373	1258	705	705
q4	10232	1010	513	513
q5	7592	2432	2317	2317
q6	183	176	134	134
q7	909	735	615	615
q8	9296	1248	1163	1163
q9	6812	5032	5092	5032
q10	6882	2303	1885	1885
q11	481	301	283	283
q12	342	345	220	220
q13	17786	3673	3090	3090
q14	233	233	209	209
q15	556	488	502	488
q16	433	429	369	369
q17	602	856	382	382
q18	7608	7075	7077	7075
q19	3290	1593	583	583
q20	333	343	225	225
q21	3684	3177	2315	2315
q22	1049	1009	951	951
Total cold run time: 116695 ms
Total hot run time: 33799 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5348	5140	5110	5110
q2	236	323	222	222
q3	2178	2653	2298	2298
q4	1349	1760	1536	1536
q5	4495	4444	4308	4308
q6	208	168	127	127
q7	1969	1899	1699	1699
q8	2613	2560	2578	2560
q9	7148	7102	7167	7102
q10	3007	3151	2744	2744
q11	575	506	501	501
q12	713	745	602	602
q13	3411	3847	3243	3243
q14	273	286	266	266
q15	516	473	477	473
q16	467	482	424	424
q17	1190	1582	1403	1403
q18	7764	7541	7370	7370
q19	855	880	861	861
q20	1970	2022	1851	1851
q21	4673	4347	4412	4347
q22	1092	1015	957	957
Total cold run time: 52050 ms
Total hot run time: 50004 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185465 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 793d140161fab159842ce071fc85cfb7a97aa04a, data reload: false

query1	1005	469	499	469
query2	6559	1830	1843	1830
query3	6772	223	219	219
query4	25979	23619	23307	23307
query5	4422	631	485	485
query6	310	221	199	199
query7	4620	485	292	292
query8	263	222	208	208
query9	8637	2653	2684	2653
query10	459	332	289	289
query11	15539	15067	14748	14748
query12	163	111	111	111
query13	1674	541	418	418
query14	8942	6084	6098	6084
query15	211	201	163	163
query16	7202	677	490	490
query17	1198	713	588	588
query18	1963	388	284	284
query19	183	184	157	157
query20	117	116	117	116
query21	212	127	104	104
query22	3929	4039	4050	4039
query23	34041	33161	33124	33124
query24	8559	2381	2384	2381
query25	528	480	409	409
query26	1239	271	147	147
query27	2747	499	333	333
query28	4359	2133	2119	2119
query29	800	570	436	436
query30	282	205	187	187
query31	921	831	789	789
query32	71	65	73	65
query33	548	357	313	313
query34	801	858	532	532
query35	762	812	736	736
query36	936	973	888	888
query37	110	105	87	87
query38	4104	4019	3989	3989
query39	1488	1399	1411	1399
query40	206	119	110	110
query41	64	59	57	57
query42	131	110	116	110
query43	492	508	473	473
query44	1325	845	835	835
query45	176	178	170	170
query46	832	1018	646	646
query47	1753	1783	1705	1705
query48	390	440	327	327
query49	773	503	420	420
query50	640	670	415	415
query51	4068	4157	4117	4117
query52	116	113	101	101
query53	230	266	183	183
query54	588	590	514	514
query55	87	86	90	86
query56	316	317	341	317
query57	1110	1152	1066	1066
query58	273	269	262	262
query59	2559	2596	2455	2455
query60	337	318	302	302
query61	131	125	152	125
query62	781	716	660	660
query63	232	206	199	199
query64	4372	1080	670	670
query65	4242	4167	4144	4144
query66	1132	416	325	325
query67	15831	15792	15253	15253
query68	8402	879	523	523
query69	492	304	264	264
query70	1200	1125	1070	1070
query71	450	320	313	313
query72	5722	4751	4929	4751
query73	751	691	359	359
query74	8931	8937	8857	8857
query75	3836	3195	2690	2690
query76	3639	1180	753	753
query77	797	383	281	281
query78	10063	10067	9223	9223
query79	3061	816	581	581
query80	646	525	462	462
query81	490	257	222	222
query82	491	132	99	99
query83	290	252	231	231
query84	294	107	92	92
query85	836	341	301	301
query86	394	306	284	284
query87	4376	4328	4311	4311
query88	3571	2306	2309	2306
query89	427	312	283	283
query90	1834	210	216	210
query91	142	141	107	107
query92	72	63	60	60
query93	2819	928	580	580
query94	664	458	303	303
query95	375	294	283	283
query96	506	581	290	290
query97	2732	2775	2660	2660
query98	245	207	205	205
query99	1418	1414	1277	1277
Total cold run time: 275808 ms
Total hot run time: 185465 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.36 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 793d140161fab159842ce071fc85cfb7a97aa04a, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.10	0.12
query3	0.25	0.19	0.19
query4	1.60	0.19	0.19
query5	0.43	0.44	0.47
query6	1.16	0.65	0.68
query7	0.02	0.02	0.02
query8	0.04	0.04	0.04
query9	0.59	0.52	0.51
query10	0.55	0.57	0.56
query11	0.16	0.11	0.11
query12	0.14	0.12	0.12
query13	0.62	0.60	0.59
query14	0.78	0.81	0.81
query15	0.86	0.85	0.86
query16	0.40	0.38	0.39
query17	1.03	1.03	1.03
query18	0.22	0.21	0.20
query19	1.97	1.85	1.76
query20	0.01	0.01	0.01
query21	15.40	0.92	0.53
query22	0.76	1.25	1.01
query23	14.69	1.39	0.63
query24	6.71	1.81	0.65
query25	0.46	0.27	0.16
query26	0.55	0.16	0.15
query27	0.05	0.05	0.05
query28	9.18	0.91	0.45
query29	12.55	4.02	3.32
query30	0.25	0.08	0.06
query31	2.83	0.59	0.39
query32	3.24	0.57	0.48
query33	2.96	3.03	3.05
query34	15.69	5.17	4.51
query35	4.54	4.56	4.58
query36	0.65	0.50	0.48
query37	0.08	0.06	0.06
query38	0.04	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.14	0.13
query41	0.07	0.02	0.02
query42	0.03	0.03	0.02
query43	0.04	0.03	0.02
Total cold run time: 101.96 s
Total hot run time: 29.36 s

@mymeiyi mymeiyi force-pushed the delete-dm-of-stale branch from 793d140 to d7dda3e Compare June 4, 2025 02:19
@mymeiyi
Copy link
Contributor Author

mymeiyi commented Jun 4, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 35039 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d7dda3ee0e6f8d93e3d135675a38f90304cffaa3, data reload: false

------ Round 1 ----------------------------------
q1	26100	5226	5079	5079
q2	1956	275	180	180
q3	10428	1291	731	731
q4	10230	1077	564	564
q5	7798	2510	2388	2388
q6	191	170	135	135
q7	922	749	640	640
q8	9324	1315	1155	1155
q9	6816	5128	5163	5128
q10	6908	2321	1912	1912
q11	481	306	274	274
q12	363	361	229	229
q13	17780	3748	3153	3153
q14	235	235	222	222
q15	569	500	494	494
q16	427	437	378	378
q17	603	878	374	374
q18	7799	7166	7137	7137
q19	1561	995	618	618
q20	338	343	220	220
q21	3941	3254	3040	3040
q22	1063	1043	988	988
Total cold run time: 115833 ms
Total hot run time: 35039 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5246	5185	5217	5185
q2	239	323	225	225
q3	2169	2667	2311	2311
q4	1371	1928	1522	1522
q5	4614	4469	4355	4355
q6	219	180	133	133
q7	1953	1922	1809	1809
q8	2620	2613	2566	2566
q9	7172	7220	7171	7171
q10	2990	3172	2763	2763
q11	576	535	497	497
q12	721	789	643	643
q13	3565	3903	3296	3296
q14	289	304	273	273
q15	532	500	494	494
q16	429	482	428	428
q17	1148	1553	1427	1427
q18	7784	7637	7325	7325
q19	872	885	810	810
q20	2015	2126	1890	1890
q21	4856	4281	4305	4281
q22	1098	1044	966	966
Total cold run time: 52478 ms
Total hot run time: 50370 ms

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 0.00% (0/89) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 56.37% (15085/26759)
Line Coverage 45.12% (134635/298402)
Region Coverage 44.22% (67769/153248)
Branch Coverage 38.75% (34716/89594)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 39.33% (35/89) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 79.72% (20998/26339)
Line Coverage 72.67% (216772/298298)
Region Coverage 70.90% (127806/180258)
Branch Coverage 64.56% (66134/102438)

1 similar comment
@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 39.33% (35/89) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 79.72% (20998/26339)
Line Coverage 72.67% (216772/298298)
Region Coverage 70.90% (127806/180258)
Branch Coverage 64.56% (66134/102438)

SWJTU-ZhangLei
SWJTU-ZhangLei reviewed Jun 6, 2025
View reviewed changes
be/src/cloud/cloud_tablet.cpp
<< "tablet_id=" << tablet_id()
<< ", rowset_id=" << rs->rowset_id().to_string()
<< ", existed rowset=" << find_it->second->rowset_id().to_string();
_unused_rowsets.emplace(find_it->second->rowset_id(), find_it->second);
Copy link
Contributor

@SWJTU-ZhangLei SWJTU-ZhangLei Jun 6, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does here need std::lock_guard < std::mutex > lock(_gc_mutex);?

dataroaring
dataroaring approved these changes Jun 10, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 10, 2025
@dataroaring dataroaring merged commit 4100fc2 into apache:master Jun 10, 2025
28 of 30 checks passed
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

mymeiyi added a commit to mymeiyi/doris that referenced this pull request Jun 18, 2025
@mymeiyi
branch-3.0: [fix](mow) remove delete bitmap when remove unused rowsets (
182ac85 
apache#50973)
@mymeiyi mymeiyi mentioned this pull request Jun 18, 2025
Merged
mymeiyi added a commit to mymeiyi/doris that referenced this pull request Jun 18, 2025
@mymeiyi
branch-3.0: [fix](mow) remove delete bitmap when remove unused rowsets (
0d71edb 
apache#50973)
dataroaring pushed a commit that referenced this pull request Jun 19, 2025
@mymeiyi
branch-3.0: [fix](mow) remove delete bitmap when remove unused rowsets (
69912ec 
#50973) (#51871)

pick #50973
mymeiyi added a commit to mymeiyi/doris that referenced this pull request Jun 24, 2025
@mymeiyi
branch-3.0: [fix](mow) remove delete bitmap when remove unused rowsets (
a16cabf 
apache#50973) (apache#51871)

pick apache#50973
@mymeiyi mymeiyi mentioned this pull request Jun 24, 2025
Merged
morrySnow pushed a commit that referenced this pull request Jun 24, 2025
@mymeiyi
branch-3.1: [fix](mow) remove delete bitmap when remove unused rowsets …
ae53bac 
…#50973 (#52180)

Cherry-pick from #50973
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zhannngchen zhannngchen zhannngchen left review comments

@SWJTU-ZhangLei SWJTU-ZhangLei SWJTU-ZhangLei approved these changes

@dataroaring dataroaring dataroaring approved these changes

@gavinchou gavinchou Awaiting requested review from gavinchou gavinchou is a code owner

@w41ter w41ter Awaiting requested review from w41ter

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.7-merged dev/3.1.0-merged p0_b reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@mymeiyi @hello-stephen @doris-robot @SWJTU-ZhangLei @zhannngchen @dataroaring @morrySnow