feat(api-core): support label & property filtering for both edge and vertex & support kout dfs mode by DanGuge · Pull Request #2295 · apache/hugegraph

DanGuge · 2023-08-25T06:35:11Z

Purpose of the PR

Support label & property filtering for both edge and vertex and the filtering is implemented in Kout Post and Kneighbor Post Apis, reducing unnecessary graph searches through pruning
Support Kout dfs mode in Kout Post Api
related issue [Feature Summary] Enhancement and optimization OLTP algorithm & api #2276

Main Changes

Label & property filtering for edge and vertex

The filtering is supported in Kout Post and Kneighbor Post Apis, and is there a need for other apis?

Originally only edge label filtering was supported, now label and property filtering for edge and vertex is supported.

add classes VEStepEntity and VEStep to support serialization in request
add class Steps to support filtering of edge and vertex in runtime(core)
add new method edgesOfVertex(Id source, Steps steps) to support label and property filtering for both edge and vertex in HugeTraverser.java
design document

Kout dfs mode

Add DFS-based Kout mode in Kout Post Api and users can select the algorithm throught request parameters.

add dfs kout in KoutTraverser.java
add class NestedIterator to support dfs traversal which can remember the vertices and layers you traversed in the past
design document

Verifying these changes

Trivial rework / code cleanup without any test coverage. (No Need)
Already covered by existing tests, such as (please modify tests here).
Need tests and can be verified as follows:
- Postman Test: the tests of Kout Post are shown below

Initialize the Graph

Initialze the graph according to the link

Label & property filtering for edge and vertex in Kout Post

Method & Url

POST http://localhost:8080/graphs/{graph}/traversers/kout

Request Body

{
  "source": "1:marko",
    "steps": {
    "direction": "BOTH",
    "edge_steps": [
        {
            "label": "knows",
            "properties": {
                "weight": "P.gt(1.2)"
            }
        },
        {
            "label": "created",
            "properties": {
                "weight": "P.gt(0.39)"
            }
        }
    ],
    "vertex_steps": [
        {
            "label": "person",
            "properties": {
                "age": "P.lt(33)"
            }
        },
        {
            "label": "software",
            "properties": {}
        }
    ],
    "max_degree": 10000,
    "skip_degree": 100000
    },
  "max_depth": 3,
  "nearest": false,
  "limit": 10000,
  "with_vertex": false,
  "with_path": true,
  "with_edge": false
}

Response Body

{
    "kout": [
        "2:ripple",
        "2:lop"
    ],
    "size": 2,
    "paths": [
        {
            "objects": [
                "1:marko",
                "2:lop",
                "1:josh",
                "2:lop"
            ]
        },
        {
            "objects": [
                "1:marko",
                "2:lop",
                "1:josh",
                "2:ripple"
            ]
        }
    ],
    "vertices": [
        "2:ripple",
        "1:marko",
        "1:josh",
        "2:lop"
    ],
    "edges": [
        "S1:josh>2>>S2:ripple",
        "S1:josh>2>>S2:lop",
        "S1:marko>2>>S2:lop"
    ],
    "measure": {
        "edge_iterations": 5,
        "vertice_iterations": 3,
        "cost(ns)": 175009500
    }
}

Kout Post dfs mode

Method & Url

POST http://localhost:8080/graphs/{graph}/traversers/kout

Request Body

{
  "source": "1:marko",
    "steps": {
    "direction": "BOTH",
    "edge_steps": [
        {
            "label": "knows",
            "properties": {
                "weight": "P.gt(1.2)"
            }
        },
        {
            "label": "created",
            "properties": {
                "weight": "P.gt(0.39)"
            }
        }
    ],
    "vertex_steps": [
        {
            "label": "person",
            "properties": {
                "age": "P.lt(33)"
            }
        },
        {
            "label": "software",
            "properties": {}
        }
    ],
    "max_degree": 10000,
    "skip_degree": 100000
    },
  "max_depth": 3,
  "nearest": false,
  "limit": 10000,
  "with_vertex": false,
  "with_path": true,
  "with_edge": false,
  "algorithm": "depth_first_search"
}

Response Body

{
    "kout": [
        "2:ripple",
        "2:lop"
    ],
    "size": 2,
    "paths": [
        {
            "objects": [
                "1:marko",
                "2:lop",
                "1:josh",
                "2:lop"
            ]
        },
        {
            "objects": [
                "1:marko",
                "2:lop",
                "1:josh",
                "2:ripple"
            ]
        }
    ],
    "vertices": [
        "2:ripple",
        "1:marko",
        "1:josh",
        "2:lop"
    ],
    "edges": [
        "S1:josh>2>>S2:ripple",
        "S1:josh>2>>S2:lop",
        "S1:marko>2>>S2:lop"
    ],
    "measure": {
        "edge_iterations": 6,
        "vertice_iterations": 3,
        "cost(ns)": 60305791
    }
}

Does this PR potentially affect the following parts?

Documentation Status

Doc - TODO
Doc - Done
Doc - No Need

codecov · 2023-08-25T06:47:42Z

Codecov Report

Merging #2295 (bcae6b8) into master (aaf67cf) will increase coverage by 3.27%.
Report is 1 commits behind head on master.
The diff coverage is 30.69%.

@@             Coverage Diff              @@
##             master    #2295      +/-   ##
============================================
+ Coverage     65.09%   68.36%   +3.27%     
- Complexity      981      987       +6     
============================================
  Files           498      500       +2     
  Lines         41241    41516     +275     
  Branches       5738     5786      +48     
============================================
+ Hits          26844    28384    +1540     
+ Misses        11737    10375    -1362     
- Partials       2660     2757      +97

Files Changed	Coverage Δ
...java/org/apache/hugegraph/backend/query/Query.java	`86.56% <ø> (ø)`
...h/traversal/algorithm/iterator/NestedIterator.java	`0.00% <0.00%> (ø)`
...e/hugegraph/traversal/algorithm/KoutTraverser.java	`57.14% <9.09%> (-16.10%)`	⬇️
...graph/traversal/algorithm/records/KoutRecords.java	`42.85% <10.00%> (-42.86%)`	⬇️
...he/hugegraph/traversal/optimize/TraversalUtil.java	`59.41% <33.33%> (ø)`
...e/hugegraph/traversal/algorithm/HugeTraverser.java	`59.48% <37.87%> (-3.96%)`	⬇️
...a/org/apache/hugegraph/api/traversers/KoutAPI.java	`67.46% <38.46%> (-0.89%)`	⬇️
...che/hugegraph/traversal/algorithm/steps/Steps.java	`51.78% <51.78%> (ø)`
.../apache/hugegraph/api/traversers/TraverserAPI.java	`60.71% <55.00%> (-14.29%)`	⬇️
.../apache/hugegraph/backend/tx/GraphTransaction.java	`79.46% <57.89%> (-0.22%)`	⬇️
... and 3 more

... and 70 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

hugegraph-core/src/main/java/org/apache/hugegraph/backend/query/Query.java

hugegraph-core/src/main/java/org/apache/hugegraph/backend/tx/GraphTransaction.java

hugegraph-core/src/main/java/org/apache/hugegraph/traversal/algorithm/HugeTraverser.java

hugegraph-core/src/main/java/org/apache/hugegraph/traversal/algorithm/KoutTraverser.java

hugegraph-core/src/main/java/org/apache/hugegraph/traversal/optimize/TraversalUtil.java

hugegraph-core/src/main/java/org/apache/hugegraph/backend/query/Query.java

Add Steps parser to parse VESteps

…nfo) in Kneighbor

… info) in Kout * add Kout dfs algorithm

Main Changes: 1. move class NestedIterator from HugeTraverser.java to separate file 2. rollback to origin code

hugegraph-core/src/main/java/org/apache/hugegraph/backend/tx/GraphTransaction.java

hugegraph-core/src/main/java/org/apache/hugegraph/traversal/algorithm/HugeTraverser.java

hugegraph-core/src/main/java/org/apache/hugegraph/traversal/algorithm/KoutTraverser.java

javeme · 2023-08-31T13:07:09Z

hugegraph-core/src/main/java/org/apache/hugegraph/traversal/optimize/TraversalUtil.java

-                                 HugeGraph graph) {
+            ConditionQuery query,
+            List<HasContainer> hasContainers,
+            HugeGraph graph) {


can we revert the style changes in all files.

ok, I reverted the style changes in all files

ok, I reverted the style changes in all files

after applying the hugegraph-style.xml we still get a string of differ format?

ok, I reverted the style changes in all files

after apply the hugegraph-style.xml we still get a string of differ format?

The code after applying the hugegraph-style.xml is different from the origin format and these updates are automatically made by IDEA save on action.

ok, I reverted the style changes in all files

after apply the hugegraph-style.xml we still get a string of differ format?

The code after applying the hugegraph-style.xml is different from the origin format and these updates are automatically made by IDEA save on action.

get it，“save on action” need pay more attention，and I'll check it soon

hugegraph-core/src/main/java/org/apache/hugegraph/traversal/algorithm/HugeTraverser.java

hugegraph-core/src/main/java/org/apache/hugegraph/backend/tx/GraphTransaction.java

hugegraph-core/src/main/java/org/apache/hugegraph/traversal/algorithm/HugeTraverser.java

hugegraph-core/src/main/java/org/apache/hugegraph/traversal/algorithm/KoutTraverser.java

hugegraph-core/src/main/java/org/apache/hugegraph/traversal/algorithm/steps/Steps.java

javeme

LGTM

imbajin

The filtering is supported in Kout Post and Kneighbor Post Apis, and is there a need for other apis?

yep if we already supported them & we need update the doc for the changes

hugegraph-api/src/main/java/org/apache/hugegraph/api/traversers/KoutAPI.java

hugegraph-core/src/main/java/org/apache/hugegraph/traversal/algorithm/HugeTraverser.java

...aph-core/src/main/java/org/apache/hugegraph/traversal/algorithm/iterator/NestedIterator.java

imbajin · 2023-09-09T11:54:31Z

hugegraph-core/src/main/java/org/apache/hugegraph/traversal/algorithm/HugeTraverser.java

+    public static final String TRAVERSE_MODE_BFS = "breadth_first_search";
+    public static final String TRAVERSE_MODE_DFS = "depth_first_search";


shall we use bfs & dfs for users to input? (just desc the full name in doc)

Co-authored-by: imbajin <[email protected]>

DanGuge · 2023-09-09T14:42:34Z

The filtering is supported in Kout Post and Kneighbor Post Apis, and is there a need for other apis?

yep if we already supported them & we need update the doc for the changes

Can I supported this feature of the rest apis in the next PR? As the code submitted this time has already been reviewed.

And I will update the doc for all the changes in hugegraph-doc.

hugegraph-api/src/main/java/org/apache/hugegraph/api/traversers/KoutAPI.java

hugegraph-core/src/main/java/org/apache/hugegraph/traversal/algorithm/HugeTraverser.java

hugegraph-api/src/main/java/org/apache/hugegraph/api/traversers/KoutAPI.java

hugegraph-core/src/main/java/org/apache/hugegraph/traversal/algorithm/HugeTraverser.java

imbajin

Can I supported this feature of the rest apis in the next PR? As the code submitted this time has already been reviewed. And I will update the doc for all the changes in hugegraph-doc.

Fine & LGTM

…vertex & support kout dfs mode (apache#2295) - Support label & property filtering for both edge and vertex and the filtering is implemented in Kout Post and Kneighbor - Post Apis, reducing unnecessary graph searches through pruning - Support Kout dfs mode in Kout Post Api Originally only edge label filtering was supported, now label and property filtering for edge and vertex is supported. - add classes VEStepEntity and VEStep to support serialization in request - add class Steps to support filtering of edge and vertex in runtime(core) - add new method edgesOfVertex(Id source, Steps steps) to support label and property filtering for both edge and vertex in HugeTraverser.java --------- Co-authored-by: imbajin <[email protected]>

…vertex & support kout dfs mode (#2295) - Support label & property filtering for both edge and vertex and the filtering is implemented in Kout Post and Kneighbor - Post Apis, reducing unnecessary graph searches through pruning - Support Kout dfs mode in Kout Post Api Originally only edge label filtering was supported, now label and property filtering for edge and vertex is supported. - add classes VEStepEntity and VEStep to support serialization in request - add class Steps to support filtering of edge and vertex in runtime(core) - add new method edgesOfVertex(Id source, Steps steps) to support label and property filtering for both edge and vertex in HugeTraverser.java --------- Co-authored-by: imbajin <[email protected]>

javeme reviewed Aug 30, 2023

View reviewed changes

DanGuge added 11 commits August 31, 2023 19:51

Add Steps to track both edge and vertex labels && properties

040a128

Add VESteps to serialize json parameter in request

45886af

Add Steps parser to parse VESteps

format TraversalUtil.java, GraphTransaction.java, Query.java

5582cde

change Step(only with edge info) to Steps(with both edge and vertex i…

0e74170

…nfo) in Kneighbor

* change Step(only with edge info) to Steps(with both edge and vertex…

07e86e4

… info) in Kout * add Kout dfs algorithm

fix: remove unused method & rename variable names

c188653

fix: name style & code style

5f42209

fix: Kout dfs vertex counter

338f272

fix: modify Kout Post and Kneighbor Post tests

96c3329

fix: update Steps.java license

28e09a5

fix: format code

68ce908

Main Changes: 1. move class NestedIterator from HugeTraverser.java to separate file 2. rollback to origin code

DanGuge force-pushed the ve_kout_dfs branch from 8662451 to 68ce908 Compare August 31, 2023 12:44

fix: format code

f887005

javeme reviewed Aug 31, 2023

View reviewed changes

fix: rollback code style

bec5d79

DanGuge force-pushed the ve_kout_dfs branch from e6cece8 to bec5d79 Compare August 31, 2023 14:48

fix: format code style

be72db5

javeme reviewed Sep 8, 2023

View reviewed changes

fix: format code & change name style

360b96d

javeme previously approved these changes Sep 9, 2023

View reviewed changes

DanGuge requested a review from imbajin September 9, 2023 06:19

imbajin reviewed Sep 9, 2023

View reviewed changes

fix: Apply suggestions from code review

e403442

Co-authored-by: imbajin <[email protected]>

DanGuge dismissed javeme’s stale review via e403442 September 9, 2023 14:35

fix: format code

8ed4b07

zyxxoo previously approved these changes Sep 10, 2023

View reviewed changes

javeme reviewed Sep 10, 2023

View reviewed changes

fix: rename 'algorithm' to 'traverse_mode'

3e340ca

DanGuge dismissed zyxxoo’s stale review via 3e340ca September 10, 2023 05:01

javeme reviewed Sep 10, 2023

View reviewed changes

fix: rename 'traverse_mode' to 'traverseMode'

bcae6b8

javeme approved these changes Sep 10, 2023

View reviewed changes

imbajin approved these changes Sep 10, 2023

View reviewed changes

imbajin changed the title ~~feat(api&core): support label & property filtering for both edge and vertex & support kout dfs mode~~ feat(api-core): support label & property filtering for both edge and vertex & support kout dfs mode Sep 10, 2023

imbajin merged commit 8e99d85 into apache:master Sep 10, 2023

		public static final String TRAVERSE_MODE_BFS = "breadth_first_search";
		public static final String TRAVERSE_MODE_DFS = "depth_first_search";

Conversation

DanGuge commented Aug 25, 2023

Purpose of the PR

Main Changes

Label & property filtering for edge and vertex

Kout dfs mode

Verifying these changes

Initialize the Graph

Label & property filtering for edge and vertex in Kout Post

Kout Post dfs mode

Does this PR potentially affect the following parts?

Documentation Status

Uh oh!

codecov bot commented Aug 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

javeme Aug 31, 2023

Choose a reason for hiding this comment

Uh oh!

DanGuge Aug 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

imbajin Sep 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DanGuge Sep 1, 2023

Choose a reason for hiding this comment

Uh oh!

imbajin Sep 1, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

javeme left a comment

Choose a reason for hiding this comment

Uh oh!

imbajin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

imbajin Sep 9, 2023

Choose a reason for hiding this comment

Uh oh!

DanGuge Sep 9, 2023

Choose a reason for hiding this comment

Uh oh!

DanGuge commented Sep 9, 2023

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Aug 25, 2023 •

edited

Loading

DanGuge Aug 31, 2023 •

edited

Loading

imbajin Sep 1, 2023 •

edited

Loading