[Proposal] Hugegraph 1.5 support bulkload for HStore #2669
haohao0103
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
HStore Bulkload 整体流程设计:
整体流程已跑通,点数据抽样验证也是符合预期,但是遇到一些问题需要跟大家讨论确认下:
待讨论项
1、partitionEngine的创建在store侧是数据的到来然后创建的,bulkload 需要提前创建好pe(rocksdb),由PD驱动实现所有pe的初始化?
2、pd下发任务指令到store,原来都是到leader分区,bulkload是所有分区引擎, 所以之前指令状态管理机制不是特别合适,现在是粗糙的实现,在pd 侧对来自store的bulkload 任务状态汇报进行数量统计来确定一个终态?
3、loader 提交bulkload任务到pd后同步阻塞直到获取到任务终态,是否有更好的方式?
4、公司yarn使用的jdk8环境,对1.5版本的代码无法运行,loader侧不好引入pd-client依赖,有无更好的建议?
Beta Was this translation helpful? Give feedback.
All reactions