Skip to content

Upgrade to JsonPath 2.7.0 #7819

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
Jackie-Jiang merged 2 commits into from
Feb 1, 2022
Merged

Upgrade to JsonPath 2.7.0 #7819

Jackie-Jiang merged 2 commits into from
Feb 1, 2022

Conversation

@richardstartin
Copy link
Member

@richardstartin richardstartin commented Nov 23, 2021

By default, when a JSON path doesn't match a document, the JSONPath library throws an exception, but this can be disabled with configuration. These exceptions can be quite costly during ingestion and are worth avoiding:
Screenshot 2021-11-23 at 11 25 59

The JsonScalarExtractTransformFunction already uses this configuration.

@richardstartin
Copy link
Member Author

richardstartin commented Nov 23, 2021 •
edited
Loading

It turns out there's a bug in the library which prevents this configuration from being respected in all cases, which I have fixed in a PR to the library. Converting to draft until it's been released.

@richardstartin richardstartin marked this pull request as draft November 23, 2021 12:01
richardstartin
richardstartin commented Nov 23, 2021
View reviewed changes
pinot-common/src/main/java/org/apache/pinot/common/function/scalar/JsonFunctions.java Outdated
Comment on lines -179 to +180
Copy link
Member Author

@richardstartin richardstartin Nov 23, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a behaviour change - it used to throw on missing paths (which I believe was unintended) and now returns NaN

richardstartin
richardstartin commented Nov 23, 2021
View reviewed changes
...src/test/java/org/apache/pinot/common/function/scalar/ArrayAwareJacksonJsonProviderTest.java Outdated
Comment on lines -56 to -57
Copy link
Member Author

@richardstartin richardstartin Nov 23, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this bug was fixed

@codecov-commenter
Copy link

codecov-commenter commented Nov 23, 2021 •
edited
Loading

Codecov Report

Merging #7819 (19d3ea5) into master (4e595a2) will increase coverage by 1.38%.
The diff coverage is 79.31%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master    #7819      +/-   ##
============================================
+ Coverage     69.99%   71.38%   +1.38%     
- Complexity     4303     4307       +4     
============================================
  Files          1616     1617       +1     
  Lines         83752    83830      +78     
  Branches      12514    12520       +6     
============================================
+ Hits          58619    59838    +1219     
+ Misses        21109    19922    -1187     
- Partials       4024     4070      +46
Flag Coverage Δ
integration1 28.90% <13.79%> (?)
integration2 27.74% <13.79%> (+0.22%) ⬆️
unittests1 68.00% <79.31%> (+0.05%) ⬆️
unittests2 14.18% <0.00%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...he/pinot/common/function/scalar/JsonFunctions.java 85.00% <76.00%> (+4.64%) ⬆️
...re/common/evaluators/DefaultJsonPathEvaluator.java 30.68% <100.00%> (+4.18%) ⬆️
...ntroller/helix/core/minion/CronJobScheduleJob.java 0.00% <0.00%> (-59.10%) ⬇️
...org/apache/pinot/core/util/ListenerConfigUtil.java 77.00% <0.00%> (-8.37%) ⬇️
...a/org/apache/pinot/spi/env/PinotConfiguration.java 92.85% <0.00%> (-2.27%) ⬇️
...elix/core/periodictask/ControllerPeriodicTask.java 70.90% <0.00%> (-1.82%) ⬇️
...main/java/org/apache/pinot/core/util/TlsUtils.java 75.29% <0.00%> (-1.74%) ⬇️
...java/org/apache/pinot/spi/stream/StreamConfig.java 90.76% <0.00%> (-1.38%) ⬇️
...pache/pinot/spi/stream/StreamConfigProperties.java 100.00% <0.00%> (ø)
... and 115 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4e595a2...19d3ea5. Read the comment docs.

@richardstartin richardstartin changed the title prevent exceptions from being thrown when jsonpaths don't match inputs Upgrade to JsonPath 2.7.0 Jan 31, 2022
@richardstartin richardstartin marked this pull request as ready for review January 31, 2022 11:34
@richardstartin richardstartin force-pushed the jsonpath-exceptions branch 2 times, most recently from f8ea01f to ed3996e Compare January 31, 2022 12:38
Jackie-Jiang
Jackie-Jiang reviewed Jan 31, 2022
View reviewed changes
Copy link
Contributor

@Jackie-Jiang Jackie-Jiang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for contributing the fix to the json-path library!
In JsonFunctions.java I can still see some method throwing JsonProcessingException. Should it be eliminated completely?

@richardstartin
Copy link
Member Author

richardstartin commented Jan 31, 2022

Thanks for contributing the fix to the json-path library!
In JsonFunctions.java I can still see some method throwing JsonProcessingException. Should it be eliminated completely?

No, the values may not be valid json, or it may not be possible to stringify. The exceptions not thrown now were thrown because a jsonpath didn’t match the document.

@Jackie-Jiang Jackie-Jiang merged commit 3f76154 into apache:master Feb 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Jackie-Jiang Jackie-Jiang Jackie-Jiang left review comments

@mayankshriv mayankshriv mayankshriv approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@richardstartin @codecov-commenter @mayankshriv @Jackie-Jiang