Skip to content

feat(hbase): support hash rowkey struct & pre-init tables#1696

Merged
javeme merged 3 commits intoapache:masterfrom
JackyYangPassion:HBase-presplit-table
Apr 6, 2022
Merged

feat(hbase): support hash rowkey struct & pre-init tables#1696
javeme merged 3 commits intoapache:masterfrom
JackyYangPassion:HBase-presplit-table

Conversation

@JackyYangPassion
Copy link
Copy Markdown
Contributor

@JackyYangPassion JackyYangPassion commented Dec 14, 2021 •
edited by imbajin
Loading

implement feature #822

Imp HBase Vertex/Edge Table pre-spilit & configurable:

Core design logic：
取2字节作为rowkey前缀：(short) (startNode.hashCode() % totalPartitions)
具体使用：
hugegraph.properties 配置文件 增加如下配置

是否打开预分区
hbase.enable_partition=true「默认值」

点表预分区
hbase.vertex_partition=10「默认值」

边表预分区
hbase.edge_partition=30「默认值」

Note: The following structural diagram is a simple speculation. If there are any differences in the actual situation, please update it ↓

graph LR

full((Hbase Rowkey)) --字符串 vid--> a0(2 Byte hash) --append--> a(1 Byte vid长度)--append-->a1(1-128 Byte 不定长字符串 vid)--final-->a2(完整rowkey: 4-131 字节)

full --数值 vid--> b0(2 Byte hash) --append--> b(1 Byte 标识长度)--append-->b1(1-8 Byte-变长)--final-->b2(完整rowkey: 4-8 字节)

full --UUID 类 vid--> c0(2 Byte hash) --append--> c(1 Byte 标识UUID)--append-->c1(写入固定16 Byte)--final-->c2(完整rowkey: 19 字节)
Loading

@codecov
Copy link
Copy Markdown

codecov bot commented Dec 14, 2021 •
edited
Loading

Codecov Report

Merging #1696 (6d5878a) into master (0cc02e4) will increase coverage by 3.97%.
The diff coverage is 94.30%.

@@             Coverage Diff              @@
##             master    #1696      +/-   ##
============================================
+ Coverage     66.93%   70.91%   +3.97%     
+ Complexity      980      695     -285     
============================================
  Files           446      446              
  Lines         37781    37883     +102     
  Branches       5380     5397      +17     
============================================
+ Hits          25289    26863    +1574     
+ Misses         9768     8290    -1478     
- Partials       2724     2730       +6
Impacted Files Coverage Δ
...ackend/store/cassandra/CassandraStoreProvider.java 100.00% <ø> (ø)
.../backend/store/memory/InMemoryDBStoreProvider.java 100.00% <ø> (ø)
...egraph/backend/store/mysql/MysqlStoreProvider.java 100.00% <ø> (ø)
...kend/store/postgresql/PostgresqlStoreProvider.java 92.50% <ø> (ø)
...ph/backend/store/rocksdb/RocksDBStoreProvider.java 31.57% <ø> (ø)
...kend/store/rocksdbsst/RocksDBSstStoreProvider.java 0.00% <ø> (ø)
.../backend/store/scylladb/ScyllaDBStoreProvider.java 97.29% <ø> (ø)
...aidu/hugegraph/backend/serializer/BytesBuffer.java 89.11% <50.00%> (+0.50%) ⬆️
...om/baidu/hugegraph/backend/store/ram/RamTable.java 66.44% <50.00%> (-0.11%) ⬇️
...u/hugegraph/backend/store/hbase/HbaseFeatures.java 83.33% <60.00%> (-7.58%) ⬇️
... and 144 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0cc02e4...6d5878a. Read the comment docs.

@javeme
Copy link
Copy Markdown
Contributor

javeme commented Dec 15, 2021 •
edited by imbajin
Loading

@JackyYangPassion 非常感谢贡献feature代码^-^

大体看了下代码，这个pr应该是支持了hbase顶点和边的预分区了，整体方案没有问题👍。稍晚点进行详细的review，有几点小建议先大体列下：

  • BackendStorede的加载传递config，可以统一为一个版本，比如所有后端都构造的时候均传递config（没关系的，按照你认为最优雅的方式来修改代码），类似的Serializer加载也可以统一为一个版本。
  • HBaseSerializer的主体逻辑是否可以上移到BinarySerializer，然后由BinarySerializer提供一个hook给HBaseSerializer做自定义相关的处理？这样预分区特性容易被其它后端存储复用。具体操作方式（仅供参考）：比如给BinarySerializer增加enablePartition成员变量、protected short getPartition(Id)抽象方法，HBaseSerializer构造时可以设置enablePartition=true，重载getPartition方法以返回hash(id)方式的分区。
  • 代码风格有些差异，需要整体改下。代码风格规范可以参考此篇
  • 简单起见, 如果你用的 IDEA, 代码风格可以直接使用 hugeraph 专属的配置文件(根目录下), 直接导入后大部分 code style 就和社区规范一致了 (少部分手调一下即可)

索引部分，考虑到后续secondary-index索引也统一为类似range-index的表格式，对于索引部分的预分区有什么计划吗？

@JackyYangPassion
Copy link
Copy Markdown
Contributor Author

JackyYangPassion commented Dec 16, 2021

@JackyYangPassion 非常感谢贡献feature代码^-^

大体看了下代码，这个pr应该是支持了hbase顶点和边的预分区了，整体方案没有问题👍。稍晚点进行详细的review，有几点小建议先大体列下：

  • BackendStorede的加载传递config，可以统一为一个版本，比如所有后端都构造的时候均传递config（没关系的，按照你认为最优雅的方式来修改代码），类似的Serializer加载也可以统一为一个版本。
  • HBaseSerializer的主体逻辑是否可以上移到BinarySerializer，然后由BinarySerializer提供一个hook给HBaseSerializer做自定义相关的处理？这样预分区特性容易被其它后端存储复用。具体操作方式（仅供参考）：比如给BinarySerializer增加enablePartition成员变量、protected short getPartition(Id)抽象方法，HBaseSerializer构造时可以设置enablePartition=true，重载getPartition方法以返回hash(id)方式的分区。
  • 代码风格有些差异，需要整体改下。代码风格规范可以参考此篇
  • 简单起见, 如果你用的 IDEA, 代码风格可以直接使用 hugeraph 专属的配置文件(根目录下), 直接导入后大部分 code style 就和社区规范一致了 (少部分手调一下即可)

索引部分，考虑到后续secondary-index索引也统一为类似range-index的表格式，对于索引部分的预分区有什么计划吗？

感谢提供建议：

  1. 关于config 传递 这一块我再想想优雅的方式 这一块加上后当时第一感觉对原框架侵入性太大
  2. HBaseSerializer的主体逻辑是否可以上移到BinarySerializer 这一块我按照提示进行修改
  3. 代码风格本周规范下，还有部分测试未过的问题一并修复
  4. 索引部分正在开发验证

@JackyYangPassion JackyYangPassion force-pushed the HBase-presplit-table branch 2 times, most recently from 9f1dbdf to 4f14d70 Compare December 16, 2021 13:40
javeme
javeme reviewed Dec 17, 2021
View reviewed changes
@imbajin
Copy link
Copy Markdown
Member

imbajin commented Dec 19, 2021

seems u format the whole file. and it'll influence a lot of lines.

if u want format your code by IDEA, u could select the code block first, then press format key (or click it), it will only format the code u select

@JackyYangPassion JackyYangPassion force-pushed the HBase-presplit-table branch 2 times, most recently from c1b96f4 to a13c2a6 Compare December 20, 2021 15:13
javeme
javeme reviewed Dec 21, 2021
View reviewed changes
@JackyYangPassion JackyYangPassion force-pushed the HBase-presplit-table branch 10 times, most recently from 9ccf0e7 to 894f004 Compare December 22, 2021 15:50
javeme
javeme reviewed Dec 23, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor

@javeme javeme left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some minor comments. and you can just append a new commit instead of a forced push(that's avoiding git commit --amend as possible), so that we can review the diff.

@javeme
Copy link
Copy Markdown
Contributor

javeme commented Jan 12, 2022

please rebase on the latest code: git rebase -i master

@JackyYangPassion
Copy link
Copy Markdown
Contributor Author

JackyYangPassion commented Feb 9, 2022 •
edited by javeme
Loading

some minor comments. and you can just append a new commit instead of a forced push(that's avoiding git commit --amend as possible), so that we can review the diff.

append a new commit with same or different commit comment ？
like this feature commit 「add HBase pre-split table feature」

@javeme
Copy link
Copy Markdown
Contributor

javeme commented Feb 9, 2022

You can set any commit message for a new commit, since we finally squash them into a commit and then merge to master branch.

@github-actions
Copy link
Copy Markdown

github-actions bot commented Mar 11, 2022

Due to the lack of activity, the current pr is marked as stale and will be closed after 180 days, any update will remove the stale label

@javeme javeme removed the inactive label Mar 14, 2022
@github-actions
Copy link
Copy Markdown

github-actions bot commented Mar 18, 2022 •
edited
Loading

CLA Assistant Lite bot Good! All Contributors have signed the CLA.

imbajin
imbajin reviewed Mar 21, 2022
View reviewed changes
javeme
javeme reviewed Mar 21, 2022
View reviewed changes
javeme
javeme reviewed Mar 21, 2022
View reviewed changes
javeme
javeme reviewed Mar 22, 2022
View reviewed changes
javeme
javeme previously approved these changes Mar 23, 2022
View reviewed changes
@javeme
Copy link
Copy Markdown
Contributor

javeme commented Mar 23, 2022

unit test error:

Error:  Tests run: 395, Failures: 0, Errors: 1, Skipped: 1, Time elapsed: 624.87 s <<< FAILURE! - in com.baidu.hugegraph.unit.UnitTestSuite
Error:  testEdgeForPartition(com.baidu.hugegraph.unit.serializer.BinarySerializerTest)  Time elapsed: 0.012 s  <<< ERROR!
java.lang.NullPointerException
	at com.baidu.hugegraph.unit.serializer.BinarySerializerTest.testEdgeForPartition(BinarySerializerTest.java:120)

hbase ci errors:

Error:  Tests run: 722, Failures: 5, Errors: 5, Skipped: 22, Time elapsed: 945.282 s <<< FAILURE! - in com.baidu.hugegraph.core.CoreTestSuite
Error:  testQueryOutEdgesOfVertexBySortkeyWithMoreFields(com.baidu.hugegraph.core.EdgeCoreTest)  Time elapsed: 0.602 s  <<< FAILURE!
java.lang.AssertionError: expected:<5> but was:<0>
	at com.baidu.hugegraph.core.EdgeCoreTest.testQueryOutEdgesOfVertexBySortkeyWithMoreFields(EdgeCoreTest.java:3865)

Error:  testQueryOutEdgesOfVertexBySortkeyWithPrefix(com.baidu.hugegraph.core.EdgeCoreTest)  Time elapsed: 0.601 s  <<< FAILURE!
java.lang.AssertionError: expected:<5> but was:<0>
	at com.baidu.hugegraph.core.EdgeCoreTest.testQueryOutEdgesOfVertexBySortkeyWithPrefix(EdgeCoreTest.java:3657)

Error:  testQueryOutEdgesOfVertexBySortkeyWithMoreFieldsInPage(com.baidu.hugegraph.core.EdgeCoreTest)  Time elapsed: 0.944 s  <<< FAILURE!
java.lang.AssertionError: expected:<5> but was:<0>
	at com.baidu.hugegraph.core.EdgeCoreTest.testQueryOutEdgesOfVertexBySortkeyWithMoreFieldsInPage(EdgeCoreTest.java:4084)

Error:  testQueryOutEdgesOfVertexBySortkeyWithRange(com.baidu.hugegraph.core.EdgeCoreTest)  Time elapsed: 0.936 s  <<< FAILURE!
java.lang.AssertionError: expected:<1> but was:<0>
	at com.baidu.hugegraph.core.EdgeCoreTest.testQueryOutEdgesOfVertexBySortkeyWithRange(EdgeCoreTest.java:3560)

Error:  testQueryOutEdgesOfVertexBySortkeyWithPrefixInPage(com.baidu.hugegraph.core.EdgeCoreTest)  Time elapsed: 0.961 s  <<< FAILURE!
java.lang.AssertionError: expected:<5> but was:<0>
	at com.baidu.hugegraph.core.EdgeCoreTest.testQueryOutEdgesOfVertexBySortkeyWithPrefixInPage(EdgeCoreTest.java:3753)

Error:  testReloadAndQueryWithProperty(com.baidu.hugegraph.core.RamTableTest)  Time elapsed: 0.265 s  <<< ERROR!
com.baidu.hugegraph.HugeException: Failed to load ramtable
	at com.baidu.hugegraph.core.RamTableTest.testReloadAndQueryWithProperty(RamTableTest.java:447)
Caused by: com.baidu.hugegraph.HugeException: The ramtable feature is not supported by hbase backend
	at com.baidu.hugegraph.core.RamTableTest.testReloadAndQueryWithProperty(RamTableTest.java:447)

Error:  testReloadAndQueryWithBigVertex(com.baidu.hugegraph.core.RamTableTest)  Time elapsed: 0.405 s  <<< ERROR!
com.baidu.hugegraph.HugeException: Failed to load ramtable
	at com.baidu.hugegraph.core.RamTableTest.testReloadAndQueryWithBigVertex(RamTableTest.java:372)
Caused by: com.baidu.hugegraph.HugeException: The ramtable feature is not supported by hbase backend
	at com.baidu.hugegraph.core.RamTableTest.testReloadAndQueryWithBigVertex(RamTableTest.java:372)

Error:  testReloadAndQueryWithMultiEdges(com.baidu.hugegraph.core.RamTableTest)  Time elapsed: 1.013 s  <<< ERROR!
com.baidu.hugegraph.HugeException: Failed to load ramtable
	at com.baidu.hugegraph.core.RamTableTest.testReloadAndQueryWithMultiEdges(RamTableTest.java:242)
Caused by: com.baidu.hugegraph.HugeException: The ramtable feature is not supported by hbase backend
	at com.baidu.hugegraph.core.RamTableTest.testReloadAndQueryWithMultiEdges(RamTableTest.java:242)

Error:  testReloadFromFileAndQuery(com.baidu.hugegraph.core.RamTableTest)  Time elapsed: 1.709 s  <<< ERROR!
com.baidu.hugegraph.HugeException: Failed to load ramtable
	at com.baidu.hugegraph.core.RamTableTest.testReloadFromFileAndQuery(RamTableTest.java:162)
Caused by: com.baidu.hugegraph.HugeException: The ramtable feature is not supported by hbase backend
	at com.baidu.hugegraph.core.RamTableTest.testReloadFromFileAndQuery(RamTableTest.java:162)

Error:  testReloadAndQuery(com.baidu.hugegraph.core.RamTableTest)  Time elapsed: 1.055 s  <<< ERROR!
com.baidu.hugegraph.HugeException: Failed to load ramtable
	at com.baidu.hugegraph.core.RamTableTest.testReloadAndQuery(RamTableTest.java:115)
Caused by: com.baidu.hugegraph.HugeException: The ramtable feature is not supported by hbase backend
	at com.baidu.hugegraph.core.RamTableTest.testReloadAndQuery(RamTableTest.java:115)

@imbajin imbajin changed the title add HBase pre-split table feature feat(hbase): refactor to hashed rowkey struct & support pre-init tables Mar 23, 2022
@JackyYangPassion
Copy link
Copy Markdown
Contributor Author

JackyYangPassion commented Mar 28, 2022

I have read the CLA Document and I hereby sign the CLA

github-actions bot added a commit that referenced this pull request Mar 28, 2022
@github-actions
@JackyYangPassion has signed the CLA from Pull Request #1696
9a78959
@JackyYangPassion
Copy link
Copy Markdown
Contributor Author

JackyYangPassion commented Mar 29, 2022

unit test error:

Error:  Tests run: 395, Failures: 0, Errors: 1, Skipped: 1, Time elapsed: 624.87 s <<< FAILURE! - in com.baidu.hugegraph.unit.UnitTestSuite
Error:  testEdgeForPartition(com.baidu.hugegraph.unit.serializer.BinarySerializerTest)  Time elapsed: 0.012 s  <<< ERROR!
java.lang.NullPointerException
	at com.baidu.hugegraph.unit.serializer.BinarySerializerTest.testEdgeForPartition(BinarySerializerTest.java:120)

hbase ci errors:

Error:  Tests run: 722, Failures: 5, Errors: 5, Skipped: 22, Time elapsed: 945.282 s <<< FAILURE! - in com.baidu.hugegraph.core.CoreTestSuite
Error:  testQueryOutEdgesOfVertexBySortkeyWithMoreFields(com.baidu.hugegraph.core.EdgeCoreTest)  Time elapsed: 0.602 s  <<< FAILURE!
java.lang.AssertionError: expected:<5> but was:<0>
	at com.baidu.hugegraph.core.EdgeCoreTest.testQueryOutEdgesOfVertexBySortkeyWithMoreFields(EdgeCoreTest.java:3865)

Error:  testQueryOutEdgesOfVertexBySortkeyWithPrefix(com.baidu.hugegraph.core.EdgeCoreTest)  Time elapsed: 0.601 s  <<< FAILURE!
java.lang.AssertionError: expected:<5> but was:<0>
	at com.baidu.hugegraph.core.EdgeCoreTest.testQueryOutEdgesOfVertexBySortkeyWithPrefix(EdgeCoreTest.java:3657)

Error:  testQueryOutEdgesOfVertexBySortkeyWithMoreFieldsInPage(com.baidu.hugegraph.core.EdgeCoreTest)  Time elapsed: 0.944 s  <<< FAILURE!
java.lang.AssertionError: expected:<5> but was:<0>
	at com.baidu.hugegraph.core.EdgeCoreTest.testQueryOutEdgesOfVertexBySortkeyWithMoreFieldsInPage(EdgeCoreTest.java:4084)

Error:  testQueryOutEdgesOfVertexBySortkeyWithRange(com.baidu.hugegraph.core.EdgeCoreTest)  Time elapsed: 0.936 s  <<< FAILURE!
java.lang.AssertionError: expected:<1> but was:<0>
	at com.baidu.hugegraph.core.EdgeCoreTest.testQueryOutEdgesOfVertexBySortkeyWithRange(EdgeCoreTest.java:3560)

Error:  testQueryOutEdgesOfVertexBySortkeyWithPrefixInPage(com.baidu.hugegraph.core.EdgeCoreTest)  Time elapsed: 0.961 s  <<< FAILURE!
java.lang.AssertionError: expected:<5> but was:<0>
	at com.baidu.hugegraph.core.EdgeCoreTest.testQueryOutEdgesOfVertexBySortkeyWithPrefixInPage(EdgeCoreTest.java:3753)

Error:  testReloadAndQueryWithProperty(com.baidu.hugegraph.core.RamTableTest)  Time elapsed: 0.265 s  <<< ERROR!
com.baidu.hugegraph.HugeException: Failed to load ramtable
	at com.baidu.hugegraph.core.RamTableTest.testReloadAndQueryWithProperty(RamTableTest.java:447)
Caused by: com.baidu.hugegraph.HugeException: The ramtable feature is not supported by hbase backend
	at com.baidu.hugegraph.core.RamTableTest.testReloadAndQueryWithProperty(RamTableTest.java:447)

Error:  testReloadAndQueryWithBigVertex(com.baidu.hugegraph.core.RamTableTest)  Time elapsed: 0.405 s  <<< ERROR!
com.baidu.hugegraph.HugeException: Failed to load ramtable
	at com.baidu.hugegraph.core.RamTableTest.testReloadAndQueryWithBigVertex(RamTableTest.java:372)
Caused by: com.baidu.hugegraph.HugeException: The ramtable feature is not supported by hbase backend
	at com.baidu.hugegraph.core.RamTableTest.testReloadAndQueryWithBigVertex(RamTableTest.java:372)

Error:  testReloadAndQueryWithMultiEdges(com.baidu.hugegraph.core.RamTableTest)  Time elapsed: 1.013 s  <<< ERROR!
com.baidu.hugegraph.HugeException: Failed to load ramtable
	at com.baidu.hugegraph.core.RamTableTest.testReloadAndQueryWithMultiEdges(RamTableTest.java:242)
Caused by: com.baidu.hugegraph.HugeException: The ramtable feature is not supported by hbase backend
	at com.baidu.hugegraph.core.RamTableTest.testReloadAndQueryWithMultiEdges(RamTableTest.java:242)

Error:  testReloadFromFileAndQuery(com.baidu.hugegraph.core.RamTableTest)  Time elapsed: 1.709 s  <<< ERROR!
com.baidu.hugegraph.HugeException: Failed to load ramtable
	at com.baidu.hugegraph.core.RamTableTest.testReloadFromFileAndQuery(RamTableTest.java:162)
Caused by: com.baidu.hugegraph.HugeException: The ramtable feature is not supported by hbase backend
	at com.baidu.hugegraph.core.RamTableTest.testReloadFromFileAndQuery(RamTableTest.java:162)

Error:  testReloadAndQuery(com.baidu.hugegraph.core.RamTableTest)  Time elapsed: 1.055 s  <<< ERROR!
com.baidu.hugegraph.HugeException: Failed to load ramtable
	at com.baidu.hugegraph.core.RamTableTest.testReloadAndQuery(RamTableTest.java:115)
Caused by: com.baidu.hugegraph.HugeException: The ramtable feature is not supported by hbase backend
	at com.baidu.hugegraph.core.RamTableTest.testReloadAndQuery(RamTableTest.java:115)

  • The failure tests BinarySerializerTest.testEdgeForPartition and testQueryOutEdgesOfVertexBySortkey has been fix ,now it can test success

  • There are five error tests in com.baidu.hugegraph.core.RamTableTest Caused by: com.baidu.hugegraph.HugeException: The ramtable feature is not supported by hbase backend

@javeme
Copy link
Copy Markdown
Contributor

javeme commented Apr 1, 2022

should update HbaseFeatures:

    @Override
    public boolean supportsScanKeyPrefix() {
        return !enableHashPartition;
    }

    @Override
    public boolean supportsScanKeyRange() {
        return !enableHashPartition;
    }

@JackyYangPassion
Copy link
Copy Markdown
Contributor Author

JackyYangPassion commented Apr 1, 2022 •
edited
Loading

For com.baidu.hugegraph.core.RamTableTest.testReloadAndQueryWithProperty() test case

  • There are five error tests in com.baidu.hugegraph.core.RamTableTest Caused by: com.baidu.hugegraph.HugeException: The ramtable feature is not supported by hbase backend

in the RamTable.java

private void loadFromDB() throws Exception {
        Query query = new Query(HugeType.VERTEX);
        query.capacity(this.verticesCapacityHalf * 2L);
        query.limit(Query.NO_LIMIT);
        Iterator<Vertex> vertices = this.graph.vertices(query);

        // switch concurrent loading here
        boolean concurrent = true;
        if (concurrent) {
            try (LoadTraverser traverser = new LoadTraverser()) {
                traverser.load(vertices);
            }
            return;
        }
}
                     |
                     |
                     V
protected long load(Iterator<Vertex> vertices) {
 while (vertices.hasNext()) {
     this.addVertex(vertex);
  }
}
                     |
                     |
                     V
  private void addVertex(Id vertex) {
            Id lastId = IdGenerator.ZERO;
            if (this.vertices.size() > 0) {
                lastId = this.vertices.get(this.vertices.size() - 1);
            }
            LOG.info("scan from hbase source {} lastId value: {} compare {} size {}", vertex, lastId, vertex.compareTo(lastId), this.vertices.size());
            if (vertex.compareTo(lastId) < 0) {
                throw new HugeException("The ramtable feature is not " +
                                        "supported by %s backend",
                                        this.graph.backend());
            }
}

the log

2022-04-01 04:21:45 [main] [INFO] c.b.h.u.Consumers - Starting 4 workers[ramtable-loading] with queue size 4000...
2022-04-01 04:21:45 [main] [INFO] c.b.h.b.s.r.RamTable - scan from hbase source 3 lastId value: 0 compare 1 size 0
2022-04-01 04:21:45 [main] [INFO] c.b.h.b.s.r.RamTable - scan from hbase source 2 lastId value: 3 compare -1 size 1
2022-04-01 04:21:46 [main] [INFO] c.b.h.u.Consumers - Starting 4 workers[ramtable-loading] with queue size 4000...
2022-04-01 04:21:46 [main] [INFO] c.b.h.b.s.r.RamTable - scan from hbase source 103 lastId value: 0 compare 1 size 0
2022-04-01 04:21:46 [main] [INFO] c.b.h.b.s.r.RamTable - scan from hbase source 113 lastId value: 103 compare 1 size 1
2022-04-01 04:21:46 [main] [INFO] c.b.h.b.s.r.RamTable - scan from hbase source 123 lastId value: 113 compare 1 size 2
2022-04-01 04:21:46 [main] [INFO] c.b.h.b.s.r.RamTable - scan from hbase source 129 lastId value: 123 compare 1 size 3
2022-04-01 04:21:46 [main] [INFO] c.b.h.b.s.r.RamTable - scan from hbase source 139 lastId value: 129 compare 1 size 4
2022-04-01 04:21:46 [main] [INFO] c.b.h.b.s.r.RamTable - scan from hbase source 149 lastId value: 139 compare 1 size 5
2022-04-01 04:21:46 [main] [INFO] c.b.h.b.s.r.RamTable - scan from hbase source 159 lastId value: 149 compare 1 size 6
2022-04-01 04:21:46 [main] [INFO] c.b.h.b.s.r.RamTable - scan from hbase source 169 lastId value: 159 compare 1 size 7
2022-04-01 04:21:46 [main] [INFO] c.b.h.b.s.r.RamTable - scan from hbase source 179 lastId value: 169 compare 1 size 8
2022-04-01 04:21:46 [main] [INFO] c.b.h.b.s.r.RamTable - scan from hbase source 189 lastId value: 179 compare 1 size 9
2022-04-01 04:21:46 [main] [INFO] c.b.h.b.s.r.RamTable - scan from hbase source 199 lastId value: 189 compare 1 size 10
2022-04-01 04:21:46 [main] [INFO] c.b.h.b.s.r.RamTable - scan from hbase source 209 lastId value: 199 compare 1 size 11
2022-04-01 04:21:46 [main] [INFO] c.b.h.b.s.r.RamTable - scan from hbase source 219 lastId value: 209 compare 1 size 12
2022-04-01 04:21:46 [main] [INFO] c.b.h.b.s.r.RamTable - scan from hbase source 229 lastId value: 219 compare 1 size 13
2022-04-01 04:21:46 [main] [INFO] c.b.h.b.s.r.RamTable - scan from hbase source 239 lastId value: 229 compare 1 size 14
2022-04-01 04:21:46 [main] [INFO] c.b.h.b.s.r.RamTable - scan from hbase source 249 lastId value: 239 compare 1 size 15
2022-04-01 04:21:46 [main] [INFO] c.b.h.b.s.r.RamTable - scan from hbase source 258 lastId value: 249 compare 1 size 16
2022-04-01 04:21:46 [main] [INFO] c.b.h.b.s.r.RamTable - scan from hbase source 268 lastId value: 258 compare 1 size 17
2022-04-01 04:21:46 [main] [INFO] c.b.h.b.s.r.RamTable - scan from hbase source 278 lastId value: 268 compare 1 size 18
2022-04-01 04:21:46 [main] [INFO] c.b.h.b.s.r.RamTable - scan from hbase source 288 lastId value: 278 compare 1 size 19
2022-04-01 04:21:46 [main] [INFO] c.b.h.b.s.r.RamTable - scan from hbase source 298 lastId value: 288 compare 1 size 20
2022-04-01 04:21:46 [main] [INFO] c.b.h.b.s.r.RamTable - scan from hbase source 1004 lastId value: 298 compare 1 size 21
2022-04-01 04:21:46 [main] [INFO] c.b.h.b.s.r.RamTable - scan from hbase source 1014 lastId value: 1004 compare 1 size 22
2022-04-01 04:21:46 [main] [INFO] c.b.h.b.s.r.RamTable - scan from hbase source 1033 lastId value: 1014 compare 1 size 23
2022-04-01 04:21:46 [main] [INFO] c.b.h.b.s.r.RamTable - scan from hbase source 1043 lastId value: 1033 compare 1 size 24
2022-04-01 04:21:46 [main] [INFO] c.b.h.b.s.r.RamTable - scan from hbase source 1053 lastId value: 1043 compare 1 size 25
2022-04-01 04:21:46 [main] [INFO] c.b.h.b.s.r.RamTable - scan from hbase source 1063 lastId value: 1053 compare 1 size 26
2022-04-01 04:21:46 [main] [INFO] c.b.h.b.s.r.RamTable - scan from hbase source 1073 lastId value: 1063 compare 1 size 27
2022-04-01 04:21:46 [main] [INFO] c.b.h.b.s.r.RamTable - scan from hbase source 1083 lastId value: 1073 compare 1 size 28
2022-04-01 04:21:46 [main] [INFO] c.b.h.b.s.r.RamTable - scan from hbase source 1093 lastId value: 1083 compare 1 size 29
2022-04-01 04:21:46 [main] [INFO] c.b.h.b.s.r.RamTable - scan from hbase source 102 lastId value: 1093 compare -1 size 30

JackyYangPassion
JackyYangPassion commented Apr 1, 2022
View reviewed changes
hugegraph-core/src/main/java/com/baidu/hugegraph/backend/serializer/BinarySerializer.java
return 0;
}

public BackendEntry parse(BackendEntry originEntry) {
Copy link
Copy Markdown
Contributor Author

@JackyYangPassion JackyYangPassion Apr 1, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for serializer.BinarySerializerTest.testEdgeForPartition test error
add parse(BackendEntry originEntry) for deserialize edge BinanryBackendEntry

JackyYangPassion
JackyYangPassion commented Apr 1, 2022
View reviewed changes
hugegraph-core/src/main/java/com/baidu/hugegraph/backend/serializer/BinarySerializer.java
BytesBuffer start = BytesBuffer.allocate(size);
start.writeId(vertex);
BytesBuffer start = BytesBuffer.allocate(BytesBuffer.BUF_EDGE_ID);
writePartitionedId(HugeType.EDGE, vertex, start);
Copy link
Copy Markdown
Contributor Author

@JackyYangPassion JackyYangPassion Apr 1, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for com.baidu.hugegraph.core.EdgeCoreTest test error

writeQueryEdgeRangeCondition() add writePartitionedId()

javeme
javeme reviewed Apr 1, 2022
View reviewed changes
hugegraph-test/src/main/java/com/baidu/hugegraph/unit/serializer/BinarySerializerTest.java
HugeEdge edge2 = objects.newEdge("147", "789");

BackendEntry entry1 = ser.writeEdge(edge1);
HugeVertex vertex1 = ser.readVertex(edge1.graph(), ser.parse(entry1));
Copy link
Copy Markdown
Contributor

@javeme javeme Apr 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure why not call ser.readVertex(edge1.graph(), entry1)

Copy link
Copy Markdown
Contributor Author

@JackyYangPassion JackyYangPassion Apr 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO move the logic of HBaseTable to BinarySerializer

Copy link
Copy Markdown
Contributor

@javeme javeme Apr 5, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can merge this PR first, and then submit a new PR to implement the TODO

javeme
javeme reviewed Apr 1, 2022
View reviewed changes
@javeme
Copy link
Copy Markdown
Contributor

javeme commented Apr 6, 2022 •
edited
Loading

rocksdb test error:
https://github.com/hugegraph/hugegraph/runs/5835421558?check_suite_focus=true#step:10:3389

Error:  Tests run: 722, Failures: 0, Errors: 1, Skipped: 13, Time elapsed: 272.879 s <<< FAILURE! - in com.baidu.hugegraph.core.CoreTestSuite
Error:  testRemoveLeftRangeIndex(com.baidu.hugegraph.core.VertexCoreTest)  Time elapsed: 2.434 s  <<< ERROR!
java.lang.NullPointerException
	at com.baidu.hugegraph.core.VertexCoreTest.testRemoveLeftRangeIndex(VertexCoreTest.java:485)

VertexCoreTest.java#L485:
https://github.com/hugegraph/hugegraph/blob/master/hugegraph-test/src/main/java/com/baidu/hugegraph/core/VertexCoreTest.java#L485

@javeme javeme changed the title feat(hbase): refactor to hashed rowkey struct & support pre-init tables feat(hbase): support hash rowkey struct & pre-init tables Apr 6, 2022
@javeme javeme merged commit 035a03e into apache:master Apr 6, 2022
@imbajin imbajin mentioned this pull request May 18, 2022
Open
1 task
@dosubot dosubot bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Apr 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@javeme javeme javeme approved these changes

@imbajin imbajin imbajin approved these changes

Assignees

No one assigned

Labels

size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@JackyYangPassion @javeme @imbajin