Skip to content

Add Spark connector option for passing Pinot query options #10443

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
xiangfu0 merged 1 commit into from
Mar 20, 2023
Merged

Add Spark connector option for passing Pinot query options #10443

xiangfu0 merged 1 commit into from
Mar 20, 2023

Conversation

@cbalci
Copy link
Contributor

@cbalci cbalci commented Mar 19, 2023

Adding Spark connector option for passing Pinot query options.

Example:

val data = spark.read
  .format("pinot")
  .option("table", "myTable")
  .option("tableType", "OFFLINE")
  .option("queryOptions", "enableNullHandling=true,maxExecutionThreads=1") // <- new
  .load()

Also includes a bugfix which allows the Spark connector to return null values instead of their placeholders.

Testing

  • Unit tests are updated
  • Ran the suite of integration tests (ExampleSparkPinotConnectorTest) against a Pinot cluster successfully

feature bugfix
release-notes (Spark connector now accepts Pinot query options)

@codecov-commenter
Copy link

codecov-commenter commented Mar 19, 2023 •
edited
Loading

Codecov Report

Merging #10443 (522e852) into master (d9c4315) will increase coverage by 6.07%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##             master   #10443      +/-   ##
============================================
+ Coverage     64.21%   70.29%   +6.07%     
- Complexity     6089     6107      +18     
============================================
  Files          2007     2061      +54     
  Lines        109281   111752    +2471     
  Branches      16692    16985     +293     
============================================
+ Hits          70177    78553    +8376     
+ Misses        33993    27672    -6321     
- Partials       5111     5527     +416
Flag Coverage Δ
integration1 24.37% <ø> (?)
integration2 24.27% <ø> (?)
unittests1 67.95% <ø> (-0.05%) ⬇️
unittests2 13.91% <100.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...ctor/spark/common/PinotDataSourceReadOptions.scala 85.48% <100.00%> (+1.55%) ⬆️
...nector/spark/common/query/ScanQueryGenerator.scala 96.87% <100.00%> (+0.32%) ⬆️

... and 446 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@xiangfu0 xiangfu0 added feature bugfix release-notes Referenced by PRs that need attention when compiling the next release notes labels Mar 20, 2023
@xiangfu0 xiangfu0 merged commit cb1069f into apache:master Mar 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@xiangfu0 xiangfu0 xiangfu0 approved these changes

Assignees

No one assigned

Labels

bugfix feature release-notes Referenced by PRs that need attention when compiling the next release notes

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@cbalci @codecov-commenter @xiangfu0