Skip to content

Fix bug preventing numeric casts for MV columns in filters using the multi-stage query engine #13425

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
Jackie-Jiang merged 2 commits into from
Jun 22, 2024
Merged

Fix bug preventing numeric casts for MV columns in filters using the multi-stage query engine #13425

Jackie-Jiang merged 2 commits into from
Jun 22, 2024

Conversation

@yashmayya
Copy link
Contributor

@yashmayya yashmayya commented Jun 18, 2024

  • Queries like SELECT * FROM mytable WHERE ARRAY_TO_MV(CAST(mv_col AS BIGINT ARRAY)) = 0; fail when run with the multi-stage query engine with an error like:
Unable to execute query plan for request: 11345490000000000 on server: 192.168.29.25@{61393,61394}, ERROR: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Caught exception while submitting request: 11345490000000000, stage: 1
...
...
Caused by: java.lang.RuntimeException: Caught exception while submitting request: 11345490000000000, stage: 1
...
...
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: No enum constant org.apache.pinot.spi.data.FieldSpec.DataType.LONG_ARRAY
...
	at org.apache.pinot.query.service.server.QueryServer.lambda$submit$1(QueryServer.java:141)
...

Caused by: java.lang.IllegalArgumentException: No enum constant org.apache.pinot.spi.data.FieldSpec.DataType.LONG_ARRAY
	at java.base/java.lang.Enum.valueOf(Enum.java:273)
	at org.apache.pinot.spi.data.FieldSpec$DataType.valueOf(FieldSpec.java:474)
	at org.apache.pinot.core.query.optimizer.filter.NumericalFilterOptimizer.getDataType(NumericalFilterOptimizer.java:403)
	at org.apache.pinot.core.query.optimizer.filter.NumericalFilterOptimizer.optimizeChild(NumericalFilterOptimizer.java:88)
org.apache.pinot.query.service.dispatch.QueryDispatcher.submit(QueryDispatcher.java:198)
org.apache.pinot.query.service.dispatch.QueryDispatcher.submitAndReduce(QueryDispatcher.java:95)
org.apache.pinot.broker.requesthandler.MultiStageBrokerRequestHandler.handleRequest(MultiStageBrokerRequestHandler.java:216)
org.apache.pinot.broker.requesthandler.BaseBrokerRequestHandler.handleRequest(BaseBrokerRequestHandler.java:133)
  • That's because the NumericalFilterOptimizer wasn't handling the ARRAY Calcite types that represent MV fields in the v2 query engine.
  • Since MV columns aren't actual arrays and can be queried directly just like SV columns (where rows are matched if any value in the MV field matches the predicate) of the corresponding type, this bug can be fixed by simply stripping the _ARRAY suffix from the target type.

@yashmayya yashmayya added bugfix multi-stage Related to the multi-stage query engine labels Jun 18, 2024
@yashmayya yashmayya requested review from Jackie-Jiang and gortiz and removed request for Jackie-Jiang June 18, 2024 13:27
@yashmayya yashmayya changed the title Fix bug preventing numeric casts for multi-value columns in the multi-stage query engine Fix bug preventing numeric casts for MV columns in filters using the multi-stage query engine Jun 18, 2024
@yashmayya yashmayya force-pushed the filter-mv-cast-numeric-v2 branch from a0af1f3 to 39043c0 Compare June 19, 2024 03:59
@yashmayya yashmayya force-pushed the filter-mv-cast-numeric-v2 branch from 39043c0 to d3a3275 Compare June 21, 2024 06:52
@yashmayya yashmayya requested a review from Jackie-Jiang June 21, 2024 06:53
Jackie-Jiang
Jackie-Jiang approved these changes Jun 21, 2024
View reviewed changes
Copy link
Contributor

@Jackie-Jiang Jackie-Jiang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@Jackie-Jiang Jackie-Jiang merged commit 5760d10 into apache:master Jun 22, 2024
suyashpatel98 pushed a commit to suyashpatel98/pinot that referenced this pull request Jul 6, 2024
@yashmayya @suyashpatel98
Fix bug preventing numeric casts for MV columns in filters using the …
f802803 
…multi-stage query engine (apache#13425)
@npawar npawar added the v1v2 label Aug 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Jackie-Jiang Jackie-Jiang Jackie-Jiang approved these changes

@gortiz gortiz Awaiting requested review from gortiz

Assignees

No one assigned

Labels

bugfix multi-stage Related to the multi-stage query engine v1v2

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@yashmayya @gortiz @Jackie-Jiang @npawar