Skip to content

[fix](cloud) Cloud tablet cache should not be pruned #51193

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
May 25, 2025
Merged

[fix](cloud) Cloud tablet cache should not be pruned #51193

merged 1 commit into from
May 25, 2025

Conversation

gavinchou
Copy link
Contributor

@gavinchou gavinchou commented May 23, 2025 •
edited
Loading

Otherwise there may be severe performance issue for query associated to many tablets

Related PR: #28940 #42281

@gavinchou gavinchou requested a review from dataroaring as a code owner May 23, 2025 09:34
@Thearas
Copy link
Contributor

Thearas commented May 23, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@gavinchou
Copy link
Contributor Author

run buildall

@gavinchou
[fix](cloud) Cloud tablet cache should not be pruned
6cd2b50 
Otherwise there may be severe performance issue for query associated to many tablets
@gavinchou gavinchou force-pushed the gavin-fix-table-cache branch from 06f2232 to 6cd2b50 Compare May 23, 2025 09:53
@gavinchou
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 56.00% (14920/26643)
Line Coverage 44.82% (132839/296381)
Region Coverage 43.90% (66771/152112)
Branch Coverage 38.52% (34263/88942)

@doris-robot
Copy link

TPC-H: Total hot run time: 33918 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6cd2b50570a936302831f4687f6d0dbe13cae3f1, data reload: false

------ Round 1 ----------------------------------
q1	26430	5106	5061	5061
q2	2078	286	194	194
q3	10497	1277	717	717
q4	10231	999	540	540
q5	7660	2349	2310	2310
q6	189	164	132	132
q7	921	728	602	602
q8	9298	1303	1100	1100
q9	6796	5033	5117	5033
q10	6876	2308	1893	1893
q11	503	298	280	280
q12	344	350	209	209
q13	17810	3727	3096	3096
q14	232	228	222	222
q15	526	490	474	474
q16	432	435	381	381
q17	640	866	387	387
q18	7629	7211	7179	7179
q19	1977	984	558	558
q20	333	346	224	224
q21	3959	3267	2365	2365
q22	1028	987	961	961
Total cold run time: 116389 ms
Total hot run time: 33918 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5238	5124	5164	5124
q2	242	329	237	237
q3	2214	2642	2290	2290
q4	1419	1779	1417	1417
q5	4450	4399	4418	4399
q6	226	170	126	126
q7	2056	1964	1744	1744
q8	2654	2658	2565	2565
q9	7209	7225	6911	6911
q10	3056	3195	2813	2813
q11	621	519	490	490
q12	721	779	604	604
q13	3526	3882	3282	3282
q14	277	311	272	272
q15	523	486	493	486
q16	449	482	459	459
q17	1161	1591	1376	1376
q18	7707	7598	7429	7429
q19	837	863	883	863
q20	1969	2107	1871	1871
q21	4900	4504	4493	4493
q22	1083	1064	1016	1016
Total cold run time: 52538 ms
Total hot run time: 50267 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192994 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6cd2b50570a936302831f4687f6d0dbe13cae3f1, data reload: false

query1	1405	1103	1058	1058
query2	6219	1841	1822	1822
query3	11032	4441	4420	4420
query4	54152	24178	23114	23114
query5	5206	622	455	455
query6	356	206	198	198
query7	4882	511	296	296
query8	322	260	246	246
query9	5492	2665	2663	2663
query10	442	344	289	289
query11	15126	15006	14814	14814
query12	175	108	114	108
query13	1046	545	419	419
query14	10201	6521	6568	6521
query15	218	202	192	192
query16	7049	665	490	490
query17	1083	733	619	619
query18	1602	403	308	308
query19	196	203	170	170
query20	135	133	122	122
query21	206	129	109	109
query22	4411	4504	4389	4389
query23	34328	33570	33479	33479
query24	6592	2402	2439	2402
query25	466	467	423	423
query26	698	273	159	159
query27	2333	523	355	355
query28	3071	2204	2173	2173
query29	588	566	458	458
query30	297	216	189	189
query31	861	841	779	779
query32	74	65	64	64
query33	463	365	316	316
query34	780	872	551	551
query35	815	845	766	766
query36	960	1004	892	892
query37	134	96	83	83
query38	4208	4330	4358	4330
query39	1526	1428	1486	1428
query40	218	119	106	106
query41	60	56	52	52
query42	128	112	117	112
query43	511	532	488	488
query44	1373	873	864	864
query45	194	176	172	172
query46	862	1043	650	650
query47	1826	1851	1821	1821
query48	398	443	336	336
query49	705	538	446	446
query50	687	690	427	427
query51	4280	4253	4266	4253
query52	109	108	101	101
query53	230	257	191	191
query54	609	579	533	533
query55	91	92	81	81
query56	322	313	298	298
query57	1173	1220	1134	1134
query58	278	265	274	265
query59	2646	2904	2657	2657
query60	329	326	324	324
query61	131	148	182	148
query62	740	792	664	664
query63	227	200	193	193
query64	1497	1060	693	693
query65	4320	4290	4228	4228
query66	731	398	300	300
query67	16463	15930	15486	15486
query68	6818	897	529	529
query69	557	313	275	275
query70	1199	1095	1113	1095
query71	502	332	300	300
query72	6054	4789	4604	4604
query73	1483	640	359	359
query74	8993	9223	8914	8914
query75	4071	3217	2704	2704
query76	4217	1187	767	767
query77	754	366	281	281
query78	10041	10033	9330	9330
query79	2407	826	577	577
query80	642	519	435	435
query81	472	254	220	220
query82	416	126	95	95
query83	307	245	242	242
query84	293	107	85	85
query85	802	358	315	315
query86	353	308	283	283
query87	4367	4544	4316	4316
query88	3404	2295	2288	2288
query89	403	319	292	292
query90	1959	204	204	204
query91	144	148	115	115
query92	77	61	55	55
query93	1157	931	587	587
query94	676	390	311	311
query95	362	288	288	288
query96	506	574	278	278
query97	2715	2768	2688	2688
query98	238	216	201	201
query99	1424	1383	1269	1269
Total cold run time: 298101 ms
Total hot run time: 192994 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.11 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6cd2b50570a936302831f4687f6d0dbe13cae3f1, data reload: false

query1	0.04	0.04	0.03
query2	0.13	0.11	0.12
query3	0.27	0.19	0.20
query4	1.59	0.20	0.21
query5	0.45	0.44	0.44
query6	1.16	0.68	0.67
query7	0.03	0.02	0.01
query8	0.05	0.04	0.04
query9	0.63	0.54	0.53
query10	0.59	0.60	0.59
query11	0.17	0.11	0.12
query12	0.15	0.12	0.12
query13	0.61	0.59	0.60
query14	0.78	0.81	0.80
query15	0.89	0.85	0.88
query16	0.39	0.39	0.38
query17	1.04	1.04	1.03
query18	0.22	0.21	0.21
query19	1.96	1.80	1.79
query20	0.01	0.01	0.01
query21	15.39	0.88	0.53
query22	0.75	1.15	0.72
query23	14.90	1.37	0.63
query24	7.56	1.40	0.70
query25	0.51	0.07	0.09
query26	0.68	0.16	0.14
query27	0.06	0.05	0.05
query28	9.43	0.89	0.44
query29	12.55	4.09	3.37
query30	0.25	0.09	0.06
query31	2.82	0.61	0.40
query32	3.23	0.55	0.48
query33	3.10	3.09	3.13
query34	15.75	5.06	4.48
query35	4.52	4.53	4.44
query36	0.68	0.50	0.49
query37	0.09	0.06	0.06
query38	0.05	0.05	0.04
query39	0.03	0.02	0.03
query40	0.16	0.13	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 103.82 s
Total hot run time: 29.11 s

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 79.43% (20832/26226)
Line Coverage 72.62% (215234/296368)
Region Coverage 70.82% (126497/178618)
Branch Coverage 64.56% (65567/101566)

dataroaring
dataroaring approved these changes May 24, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels May 24, 2025
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@dataroaring dataroaring merged commit 2c257f2 into apache:master May 25, 2025
25 of 27 checks passed
github-actions bot pushed a commit that referenced this pull request May 25, 2025
@gavinchou
[fix](cloud) Cloud tablet cache should not be pruned (#51193)
421e0f8 
Otherwise there may be severe performance issue for query associated to
many tablets

Related PR:  #28940 #42281
@github-actions github-actions bot mentioned this pull request May 25, 2025
Merged
dataroaring pushed a commit that referenced this pull request May 26, 2025
@github-actions @gavinchou
branch-3.0: [fix](cloud) Cloud tablet cache should not be pruned #51193…
7b75be8 
… (#51215)

Cherry-picked from #51193

Co-authored-by: Gavin Chou <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Yukang-Lian Yukang-Lian Yukang-Lian approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.6-merged p0_b reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@gavinchou @Thearas @doris-robot @hello-stephen @Yukang-Lian @dataroaring