Skip to content

feat(core): support batch+parallel edges traverse#2312

Merged
imbajin merged 11 commits intoapache:masterfrom
DanGuge:thread_parallel
Oct 24, 2023
Merged

feat(core): support batch+parallel edges traverse#2312
imbajin merged 11 commits intoapache:masterfrom
DanGuge:thread_parallel

Conversation

@DanGuge
Copy link
Copy Markdown
Contributor

@DanGuge DanGuge commented Sep 15, 2023

Purpose of the PR

Main Changes

  • Enhance Consumers.java, supporting ExceptionHandle and Future to handle InterruptedException when awaiting
  • Add Nested Iterator Edge and support batch execution
  • Support batch execution & thread parallel in KoutTraverser and Kneighbor
  • design document

Verifying these changes

  • Trivial rework / code cleanup without any test coverage. (No Need)
  • Already covered by existing tests, such as (please modify tests here).
  • Need tests and can be verified as follows:
    • Postman Test & Runtime Log: the tests of Kout Post are shown below

Initialize the Graph

  • Initialze the graph according to the link

Add LOG in consumer

image

Postman Test & Check Log

image image
  • About correctness, we conducted multiple sets of serial and parallel comparison tests and found no differences
  • About parallel acceleration, there is no significant acceleration due to the lack of larger levels of test data

Does this PR potentially affect the following parts?

  • Nope
  • Dependencies (add/update license info)
  • Modify configurations
  • The public API
  • Other affects (typed here)
    • Traverser execution logic

Documentation Status

  • Doc - TODO
  • Doc - Done
  • Doc - No Need

@codecov
Copy link
Copy Markdown

codecov bot commented Sep 15, 2023 •
edited
Loading

Codecov Report

Merging #2312 (23f48ca) into master (4ceef1a) will decrease coverage by 0.23%.
The diff coverage is 70.61%.

@@             Coverage Diff              @@
##             master    #2312      +/-   ##
============================================
- Coverage     68.26%   68.04%   -0.23%     
+ Complexity      989      981       -8     
============================================
  Files           500      501       +1     
  Lines         41516    41662     +146     
  Branches       5786     5796      +10     
============================================
+ Hits          28342    28350       +8     
- Misses        10415    10556     +141     
+ Partials       2759     2756       -3
Files Changed Coverage Δ
...che/hugegraph/traversal/algorithm/steps/Steps.java 49.12% <0.00%> (-2.67%) ⬇️
...main/java/org/apache/hugegraph/util/Consumers.java 51.95% <42.85%> (-4.90%) ⬇️
...e/hugegraph/traversal/algorithm/OltpTraverser.java 68.00% <74.71%> (+3.82%) ⬆️
...egraph/traversal/algorithm/KneighborTraverser.java 79.54% <76.19%> (-3.79%) ⬇️
...he/hugegraph/backend/query/EdgesQueryIterator.java 78.57% <78.57%> (ø)
...e/hugegraph/traversal/algorithm/HugeTraverser.java 51.91% <81.25%> (-7.57%) ⬇️
...e/hugegraph/traversal/algorithm/KoutTraverser.java 58.42% <96.15%> (+1.28%) ⬆️
...in/java/org/apache/hugegraph/task/TaskManager.java 74.69% <100.00%> (-1.86%) ⬇️
.../traversal/algorithm/records/KneighborRecords.java 91.66% <100.00%> (+2.19%) ⬆️
.../hugegraph/backend/store/rocksdb/RocksDBStore.java 73.46% <100.00%> (ø)

... and 17 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

javeme
javeme reviewed Sep 15, 2023
View reviewed changes
Copy link
Copy Markdown
Contributor

@javeme javeme left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice enhance

@DanGuge DanGuge changed the title feat(core): support batch processing and thread parallel feat(core): support batch+parallel edges traverse Sep 16, 2023
javeme
javeme reviewed Sep 16, 2023
View reviewed changes
hugegraph-core/src/main/java/org/apache/hugegraph/util/Consumers.java Outdated
public static final int THREADS = 4 + CoreOptions.CPUS / 4;
public static final int QUEUE_WORKER_SIZE = 1000;
public static final long CONSUMER_WAKE_PERIOD = 1;
private static final Object QUEUE_END = new VWrapper(null);
Copy link
Copy Markdown
Contributor

@javeme javeme Sep 16, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we remove VWrapper class and just use new Object() instead

Copy link
Copy Markdown
Contributor Author

@DanGuge DanGuge Sep 17, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

QUEUE_END will be put at the end of BlockingQueue<VWrapper<V>> queue, so it will be cast to VWrapper<V>.

If we just use new Object() instead, we will get the exception below.

image

Copy link
Copy Markdown
Contributor

@javeme javeme Sep 17, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

get it, can we remove VWrapper class and keep BlockingQueue<V> queue, then try to define V QUEUE_END = (V) new Object();

@imbajin
Copy link
Copy Markdown
Member

imbajin commented Sep 17, 2023

could use ldbc data to perf it (u could the instruction in ldbc-feishu-doc)

javeme
javeme reviewed Sep 17, 2023
View reviewed changes
hugegraph-core/src/main/java/org/apache/hugegraph/util/Consumers.java Outdated
public static final int THREADS = 4 + CoreOptions.CPUS / 4;
public static final int QUEUE_WORKER_SIZE = 1000;
public static final long CONSUMER_WAKE_PERIOD = 1;
private static final Object QUEUE_END = new VWrapper(null);
Copy link
Copy Markdown
Contributor

@javeme javeme Sep 17, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

get it, can we remove VWrapper class and keep BlockingQueue<V> queue, then try to define V QUEUE_END = (V) new Object();

@imbajin imbajin added improvement General improvement api Changes of API perf labels Sep 19, 2023
javeme
javeme reviewed Sep 20, 2023
View reviewed changes
javeme
javeme approved these changes Sep 22, 2023
View reviewed changes
Copy link
Copy Markdown
Contributor

@javeme javeme left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@imbajin imbajin requested a review from zyxxoo September 25, 2023 01:25
@imbajin imbajin merged commit 8db0a9b into apache:master Oct 24, 2023
VGalaxies pushed a commit to VGalaxies/incubator-hugegraph that referenced this pull request Nov 10, 2023
@DanGuge @VGalaxies
feat(core): support batch+parallel edges traverse (apache#2312)
f275477 
- Enhance Consumers.java, supporting ExceptionHandle and `Future` to handle InterruptedException when awaiting
- Add Nested Iterator Edge and support batch execution
- Support batch execution & thread parallel in KoutTraverser and Kneighbor
imbajin pushed a commit that referenced this pull request Nov 10, 2023
@DanGuge @imbajin
feat(core): support batch+parallel edges traverse (#2312)
94edfcd 
- Enhance Consumers.java, supporting ExceptionHandle and `Future` to handle InterruptedException when awaiting
- Add Nested Iterator Edge and support batch execution
- Support batch execution & thread parallel in KoutTraverser and Kneighbor
@imbajin imbajin mentioned this pull request Dec 4, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@javeme javeme javeme approved these changes

@imbajin imbajin imbajin approved these changes

@zyxxoo zyxxoo Awaiting requested review from zyxxoo

Assignees

No one assigned

Labels

api Changes of API improvement General improvement perf

Projects

HugeGraph Tasks
Status: ✅ Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@DanGuge @imbajin @javeme