Skip to content

[Fix](multicatelog) Fix insert iceberg/hive table when use broker #51187

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
May 26, 2025
Merged

[Fix](multicatelog) Fix insert iceberg/hive table when use broker #51187

merged 1 commit into from
May 26, 2025

Conversation

liutang123
Copy link
Contributor

What problem does this PR solve?

Issue Number: close #xxx

0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) in /usr/local/service/doris/lib/be/doris_be
 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.10] in /usr/local/jdk17/lib/server/libjvm.so
 2# JVM_handle_linux_signal in /usr/local/jdk17/lib/server/libjvm.so
 3# 0x00007F1AD2254AD0 in /lib64/libc.so.6
 4# doris::FileFactory::create_fs(doris::io::FSPropertiesRef const&, doris::io::FileDescription const&) in /usr/local/service/doris/lib/be/doris_be
 5# doris::vectorized::VIcebergPartitionWriter::open(doris::RuntimeState*, doris::RuntimeProfile*) in /usr/local/service/doris/lib/be/doris_be
 6# doris::vectorized::VIcebergTableWriter::write(doris::RuntimeState*, doris::vectorized::Block&) in /usr/local/service/doris/lib/be/doris_be
 7# doris::vectorized::AsyncResultWriter::process_block(doris::RuntimeState*, doris::RuntimeProfile*) in /usr/local/service/doris/lib/be/doris_be
 8# std::_Function_handler<void (), doris::vectorized::AsyncResultWriter::start_writer(doris::RuntimeState*, doris::RuntimeProfile*)::$_0>::_M_invoke(std::_Any_data const&) in /usr/local/service/doris/lib/be/doris_be

Related PR: #xxx

Problem Summary:
When write Iceberg/Hive table and use broker, FE should pass broker list to BE.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented May 23, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@liutang123 liutang123 force-pushed the fix-broker-write-master branch from da6ce3d to 6674f99 Compare May 23, 2025 08:58
@liutang123
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 83.25% (1113/1337)
Line Coverage 66.18% (18680/28224)
Region Coverage 65.85% (9271/14079)
Branch Coverage 55.60% (4985/8966)

@doris-robot
Copy link

TPC-H: Total hot run time: 33728 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6674f9922dfd07c53e4f3189d5ae80de0a3a5e13, data reload: false

------ Round 1 ----------------------------------
q1	26417	5006	4955	4955
q2	2083	284	178	178
q3	10502	1228	704	704
q4	10236	993	517	517
q5	7655	2370	2321	2321
q6	185	162	131	131
q7	922	710	609	609
q8	9319	1492	1078	1078
q9	6846	5155	5105	5105
q10	6868	2318	1918	1918
q11	504	287	288	287
q12	348	352	210	210
q13	17795	3629	3083	3083
q14	225	243	219	219
q15	545	495	488	488
q16	436	434	377	377
q17	591	856	376	376
q18	7711	7193	7095	7095
q19	2064	1039	559	559
q20	331	329	222	222
q21	3677	3161	2355	2355
q22	1066	1021	941	941
Total cold run time: 116326 ms
Total hot run time: 33728 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5200	5060	5039	5039
q2	230	323	239	239
q3	2168	2676	2296	2296
q4	1355	1774	1412	1412
q5	4468	4393	4408	4393
q6	219	178	131	131
q7	2001	1939	1786	1786
q8	2570	2491	2605	2491
q9	7275	7238	7019	7019
q10	3042	3185	2754	2754
q11	568	512	482	482
q12	683	745	609	609
q13	3526	3876	3312	3312
q14	278	295	275	275
q15	527	476	472	472
q16	459	466	443	443
q17	1140	1567	1371	1371
q18	7947	7567	7488	7488
q19	790	874	1086	874
q20	2001	2029	1853	1853
q21	4711	4415	4552	4415
q22	1107	1068	1038	1038
Total cold run time: 52265 ms
Total hot run time: 50192 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192268 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6674f9922dfd07c53e4f3189d5ae80de0a3a5e13, data reload: false

query1	1430	1069	1062	1062
query2	6295	1823	1832	1823
query3	11005	4475	4390	4390
query4	55297	25226	23235	23235
query5	5149	491	467	467
query6	374	212	186	186
query7	4904	508	292	292
query8	313	242	230	230
query9	5904	2624	2625	2624
query10	450	323	273	273
query11	15092	14991	14814	14814
query12	168	115	106	106
query13	1052	540	417	417
query14	10204	6251	6259	6251
query15	202	197	185	185
query16	7131	655	526	526
query17	1059	730	590	590
query18	1514	433	314	314
query19	200	222	164	164
query20	118	126	127	126
query21	214	124	107	107
query22	4492	4411	4319	4319
query23	34026	33326	33616	33326
query24	6674	2418	2423	2418
query25	484	504	442	442
query26	699	278	165	165
query27	2199	517	358	358
query28	3009	2178	2144	2144
query29	600	596	476	476
query30	274	227	194	194
query31	858	875	777	777
query32	76	66	67	66
query33	460	379	336	336
query34	785	874	530	530
query35	787	885	737	737
query36	938	1045	919	919
query37	122	101	85	85
query38	4213	4292	4155	4155
query39	1561	1496	1506	1496
query40	212	120	109	109
query41	54	55	53	53
query42	133	117	113	113
query43	512	536	491	491
query44	1343	851	841	841
query45	184	174	166	166
query46	856	1028	656	656
query47	1884	1885	1804	1804
query48	407	455	344	344
query49	697	515	431	431
query50	678	705	409	409
query51	4216	4251	4271	4251
query52	114	111	103	103
query53	227	262	199	199
query54	614	586	529	529
query55	92	90	87	87
query56	326	319	301	301
query57	1151	1202	1163	1163
query58	271	265	275	265
query59	2731	2821	2701	2701
query60	337	331	326	326
query61	129	126	125	125
query62	705	760	679	679
query63	227	189	198	189
query64	1485	1048	666	666
query65	4338	4216	4184	4184
query66	728	407	303	303
query67	16212	15692	15516	15516
query68	7237	897	541	541
query69	560	311	287	287
query70	1196	1130	1062	1062
query71	489	329	306	306
query72	5527	4727	4854	4727
query73	1381	647	361	361
query74	8928	9180	8652	8652
query75	3698	3212	2690	2690
query76	4264	1205	744	744
query77	632	371	294	294
query78	10019	10068	9230	9230
query79	2532	804	588	588
query80	651	513	443	443
query81	485	253	221	221
query82	470	126	93	93
query83	377	252	237	237
query84	292	121	85	85
query85	797	394	312	312
query86	404	286	283	283
query87	4500	4413	4384	4384
query88	3612	2313	2304	2304
query89	410	317	279	279
query90	1907	210	216	210
query91	140	137	110	110
query92	70	58	58	58
query93	1409	959	593	593
query94	668	409	311	311
query95	372	288	289	288
query96	497	557	277	277
query97	2709	2772	2669	2669
query98	231	221	211	211
query99	1453	1396	1289	1289
Total cold run time: 299112 ms
Total hot run time: 192268 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.62 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6674f9922dfd07c53e4f3189d5ae80de0a3a5e13, data reload: false

query1	0.04	0.04	0.03
query2	0.15	0.11	0.10
query3	0.34	0.20	0.20
query4	1.59	0.21	0.20
query5	0.48	0.45	0.46
query6	1.21	0.65	0.66
query7	0.02	0.02	0.02
query8	0.06	0.05	0.05
query9	0.62	0.53	0.52
query10	0.58	0.60	0.57
query11	0.25	0.15	0.12
query12	0.24	0.14	0.13
query13	0.63	0.61	0.62
query14	0.78	0.82	0.82
query15	0.96	0.87	0.87
query16	0.36	0.38	0.37
query17	1.05	1.05	1.05
query18	0.17	0.18	0.20
query19	1.98	1.89	1.84
query20	0.02	0.01	0.02
query21	15.40	0.96	0.66
query22	0.93	1.07	0.79
query23	14.72	1.54	0.74
query24	5.44	0.57	0.29
query25	0.17	0.10	0.08
query26	0.55	0.21	0.18
query27	0.09	0.09	0.09
query28	11.04	1.19	0.58
query29	12.54	4.10	3.39
query30	0.28	0.08	0.07
query31	2.84	0.66	0.43
query32	3.23	0.60	0.51
query33	3.05	3.05	3.16
query34	16.67	5.13	4.40
query35	4.49	4.43	4.42
query36	0.63	0.51	0.49
query37	0.20	0.18	0.17
query38	0.16	0.15	0.16
query39	0.05	0.04	0.04
query40	0.18	0.17	0.16
query41	0.10	0.06	0.05
query42	0.06	0.05	0.05
query43	0.06	0.05	0.04
Total cold run time: 104.41 s
Total hot run time: 29.62 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 24, 2025
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

lide-reed
lide-reed approved these changes May 26, 2025
View reviewed changes
Copy link
Contributor

@lide-reed lide-reed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@liutang123 liutang123 merged commit c2e867e into apache:master May 26, 2025
26 of 28 checks passed
github-actions bot pushed a commit that referenced this pull request May 26, 2025
@liutang123
[Fix](multicatelog) Fix insert iceberg/hive table when use broker (#5…
3446e39 
…1187)

When write Iceberg/Hive table and use broker, FE should pass broker list to BE.

Co-authored-by: liutang123 <[email protected]>
@github-actions github-actions bot mentioned this pull request May 26, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lide-reed lide-reed lide-reed approved these changes

@cambyzju cambyzju cambyzju approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.x dev/3.1.x reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@liutang123 @Thearas @doris-robot @lide-reed @cambyzju