I feel this might catch a real bug, thus we should fix the bug instead of disabling the test

@walterddr quick question, is this regarding the following failure:

Error:  Failures: 
Error:    QueryRunnerTest.testSqlWithExceptionMsgChecker:204 Exception should contain: For input string:! but found: Received error query execution result block: {1000=io.grpc.StatusRuntimeException: CANCELLED: client cancelled
java.lang.RuntimeException: io.grpc.StatusRuntimeException: CANCELLED: client cancelled
	at org.apache.pinot.query.mailbox.channel.MailboxContentStreamObserver.createErrorContent(MailboxContentStreamObserver.java:163)
	at org.apache.pinot.query.mailbox.channel.MailboxContentStreamObserver.onError(MailboxContentStreamObserver.java:143)
	at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:288)
	at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:378)
...
Caused by: io.grpc.StatusRuntimeException: CANCELLED: client cancelled
	at io.grpc.Status.asRuntimeException(Status.java:530)
	at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:291)
	... 8 more} expected [true] but found [false]
[INFO] 
Error:  Tests run: 1020, Failures: 1, Errors: 0, Skipped: 0

My PR has run into this a couple of times, but when I run this test locally it passes

@walterddr quick question, is this regarding the following failure:

Error:  Failures: 
Error:    QueryRunnerTest.testSqlWithExceptionMsgChecker:204 Exception should contain: For input string:! but found: Received error query execution result block: {1000=io.grpc.StatusRuntimeException: CANCELLED: client cancelled
java.lang.RuntimeException: io.grpc.StatusRuntimeException: CANCELLED: client cancelled
	at org.apache.pinot.query.mailbox.channel.MailboxContentStreamObserver.createErrorContent(MailboxContentStreamObserver.java:163)
	at org.apache.pinot.query.mailbox.channel.MailboxContentStreamObserver.onError(MailboxContentStreamObserver.java:143)
	at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:288)
	at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:378)
...
Caused by: io.grpc.StatusRuntimeException: CANCELLED: client cancelled
	at io.grpc.Status.asRuntimeException(Status.java:530)
	at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:291)
	... 8 more} expected [true] but found [false]
[INFO] 
Error:  Tests run: 1020, Failures: 1, Errors: 0, Skipped: 0

My PR has run into this a couple of times, but when I run this test locally it passes

Correct. i wasn't able to repro even running 100+ times in a loop locally. CC @ankitsultana it seems like the issue surfaced after #10322 but i am not 100% sure that's the issue.

Will take a look and get back. I have seen this error on my local as well once in the past (and for the least() query too).

Found the issue: if an OpChain runs into a error, then it tries to send the error-block to the receiver and then return.

If the OpChain returns a error-block, we issue a cancel. This has a race with the error-block sent upstream because sends are async.

Will raise a fix shortly. Thanks folks for reporting this.

Raised: #10425

And more follow-up items in: #10424

Main thing is the design doc which I have been punting for some time. I'll try to close it this week. I was planning to discuss all these scenarios in the same but couldn't get time to write it down.

repush another attempt for fixing the issue

Codecov Report

Merging #10417 (0f4cfa8) into master (11b6bcd) will decrease coverage by 35.24%.
The diff coverage is 0.00%.

@@              Coverage Diff              @@
##             master   #10417       +/-   ##
=============================================
- Coverage     63.23%   27.99%   -35.24%     
+ Complexity     5068       58     -5010     
=============================================
  Files          2028     2028               
  Lines        110632   110635        +3     
  Branches      16847    16847               
=============================================
- Hits          69955    30977    -38978     
- Misses        35505    76575    +41070     
+ Partials       5172     3083     -2089     

Flags with carried forward coverage won't be shown. Click here to find out more.

... and 1170 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

will close this PR for now. it showed the proper root cause and several fixes