Draft commit with the IndexService draft #1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
5 tasks
This was referenced Jan 27, 2023
…o index-spi-index-service
- Mailbox ownership model: * ---------------------------------------------------------------------------- * (Operator Layer) * MailboxSendOperator ---------> MailboxReceiveOperator * | | * | | * ------------------|----------------------------------|---------------------- * | | * (MailboxService) | | ( WAIT ON INIT ) * \_/ \_/ * SendingMailbox ReceivingMailbox * ------------------|---------------------------------/^\--------------------- * (Physical Layer) | | * (e.g. GRPC) | | ( INITIALIZE ) * | | * \_/ | * StreamObserver -------------------> StreamObserver * ---------------------------------------------------------------------------- - Work items done in this PR * clean up MailboxService API to sanitize cancel/release. * completely move SendMailbox management out of MailboxService * make sending mailbox lazy-open GRPC observers * add queue offer timeout to avoid infinite OOM blow up * handle close/cancel on leaf-stage operator + minor log change * enhanced in-mem transfer stream
* adding dispatch cancellation * fix cancel point inside scheduling to avoid put-back in schedule queue --------- Co-authored-by: Rong Rong <[email protected]>
…10409) * [multistage] Fix Predicate Pushdow by Using Rule Collection * Fix tests && minor refactors * Add tests for IN/Not-In * Address feedback
Co-authored-by: Rong Rong <[email protected]>
Co-authored-by: Kartik Khare <[email protected]>
…pache#10413) * Refactor: Pass context object instead on individual arguments to operator * Remove no-context operator constructor and replace with new constructor in tests * Refactor MultiStageOperator to accept only context * Replace arguments with operator context in constructors used in only testing * Rename OperatorExecutionContext to OpChainExecutionContext * Refactor MailboxSendOperator for context * Remove redundant constructors and use context based constructor for MailboxReceiveOperator * Remove no context constructor from MultiStageOperator * Use boxed timeout value in mailbox receive * Make context as first argument --------- Co-authored-by: Kartik Khare <[email protected]> Co-authored-by: Kartik Khare <[email protected]>
…field from the response (apache#10419) * Adding dropResults parameter in queryOptions to drop the resultTable field from the response. * Incorporated review feedbacks * Incorporated review feedbacks
…e as it will be sender and receiver aware (apache#10408) - MailboxSendOperator will be modified later to add sort support. - Modify SortOperator to avoid sorting if the input is already sorted. It should still apply offset + limit
…o index-spi-index-service
gortiz
force-pushed
the
index-spi-index-service
branch
from
aec56aa to
3abed16
Compare
* Implement Pinot Spark connector for Spark3 * Address documentation related comments * Add docstrings for main classes * Add sample command for running example jobs in cluster mode
gortiz
pushed a commit
that referenced
this pull request
May 6, 2025
* Addition of initial spi change checker code * Fixes to yaml file, mainly excluding artifacts.zip code to test run further * changing shell file permissions * add "exit 1" to shell script and mess with TableConfig method signature * changing shell file permissions again for some reason * changing git diff checker file path * second git diff checker file path change * removing unnecessary code from files, fixed issue with running GitDiffChecker, added functionality for displaying line number, and reverted temporary change to TableConfig.java * permission changes * changing permissions * fixing compilation error * fixing parameterization of commits * trial and error yml file #1 * commit for testing that config file correctly has parameters, changing sh file so that "No incorrect..." message only displayed once, and adding blank line check to GitDiffChecker * GitDiffChecker: add case to skip ---. otherwise, change parameters to work with pull requests. * testing * testing #2 * Revert "testing #2" This reverts commit 270937f. * yml: change main to master. shell: change main to master and change conditional to reflect return type change. java: switch from returning line number to string of code, as previous logic did not work if multiple "chunks" of code were changed, and added annotation logic that excluded json-related annotations. * yml: comment out apache/pinot condition for now. shell: change error message and put text file in my module. java: slightly altered regex after rethinking it, removed System.out.println, i think it's not necessary * java: slightly altered regex to account for interface definitions using semicolons and not curly braces * changing "no incorrect spi changes" value to "0", so if the method returns nothing, there isn't an accidental test passing * minor change to annotation regex, \n changed to $ (end of string) * fix to pom.xml * Added logic for outputting line number along with original file code snippet * per testing on another branch, slightly updating line number logic * Removed my outdated custom Java implementation of pinot-spi change checking. Switched to the japicmp plugin, with some modifications to the compatibility of checks that japicmp performs. With this, contributors will be able to see if they made incompatible SPI changes when running mvn clean, rather than waiting until they make a PR. Adding a .jar of pinot-spi that japicmp will use for comparisons. * Per Tianle's comment, added a comment in the pom file explaining our justification for using a baseline jar for comparison, and that we need to eventually transition away from it. * Removed code that made annotation changes/deletions incompatible. Fixed pom file so that all pinot-spi files are checked * updated baseline .jar to match updates from apache:master
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is the first draft PR related to apache#10183. The whole PR is in apache#10184. It isn't intended to be merged as it is and it doesn't require to compile or pass the checks. It has been created in order to make it easier to understand the PR with all the changes.
This is the partial PR with changes since apache#10191. To read the PR with all changes on master see apache#10192.
It shows the basics of the proposed
IndexServicewithout actually modifying the code and also includes aBloomIndexPluginas an example. Here we use Java Service Provider Interface to discover all services that implementIndexPlugin, although the procedure to discover the instances is subject of discussion.