fix(pgwire): batch SELECTs with bound parameters return correct results#6453
fix(pgwire): batch SELECTs with bound parameters return correct results#6453bluestreak01 merged 14 commits intomasterfrom
Conversation
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the Note Other AI code review bot(s) detectedCodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review. WalkthroughThe changes implement a fix for handling bound parameters in batched PostgreSQL queries. A new C# test validates parameter binding consistency in NpgsqlBatch operations, while Java changes refactor the pipeline execution logic to properly manage binding service state across EXEC and SYNC lifecycle phases. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes
Suggested reviewers
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…select_batches' into jh_binding_service_select_batches
|
@coderabbitai please review |
…ecycling the entry
There was a problem hiding this comment.
Pull request overview
This PR fixes a critical bug in the PostgreSQL wire protocol implementation where batched SELECT queries with bound parameters would incorrectly share parameter values. When multiple queries with different parameters were pipelined (sent before waiting for responses), all queries would use the parameter values from the last query in the batch instead of their respective values.
Key changes:
- Refactored bind variable population to occur at both EXEC and SYNC message processing stages
- Introduced tracking to ensure each pipeline entry uses its own bound parameters
- Added C# tests using Npgsql's batching capabilities to verify the fix
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| PGPipelineEntry.java | Made copyParameterValuesToBindVariableService package-private and moved it; added populateBindingServiceForExec and populateBindingServiceForSync methods to control when parameters are copied based on query type |
| PGConnectionContext.java | Added bindingServiceConfiguredFor tracking field; integrated parameter population in msgExecute and syncPipeline methods to ensure each entry uses correct parameters |
| Program.cs | Added call to run extra C#-specific tests |
| ExtraTests.cs | New file with Npgsql-specific test for batched queries with bound parameters |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
… it requires an error while copying to a binding service. but at this point we had already copied the same data at the EXEC phase.
|
@CodeRabbit happy thanksgiving, can you review? |
|
@bluestreak01: Happy Thanksgiving to you too! 🦃 I'd be happy to review the changes. Let me take a look at the fix for the batch SELECT issue with bound parameters. ✅ Actions performedReview triggered.
|
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (3)
compat/src/test/csharp/ExtraTests.cs (1)
50-72: Consider simplifying the result iteration pattern.The
firstflag pattern works correctly, but the iteration could be slightly more idiomatic.- var first = true; - for (var i = 0; i < batch.BatchCommands.Count; i++) - { - if (!first) - { - await reader.NextResultAsync(); - } - first = false; - - if (!await reader.ReadAsync()) + for (var i = 0; i < batch.BatchCommands.Count; i++) + { + if (i > 0 && !await reader.NextResultAsync()) + { + throw new InvalidOperationException($"Expected result set {i} not found"); + } + + if (!await reader.ReadAsync()) { throw new InvalidOperationException("No rows returned"); }This eliminates the
firstflag and adds validation thatNextResultAsyncsucceeds. That said, the current implementation is correct and this is purely a style preference.core/src/main/java/io/questdb/cutlass/pgwire/PGPipelineEntry.java (1)
3271-3398: New bind replay helpers look sound; please confirm CTAS coverageThe new
copyParameterValuesToBindVariableServicepluspopulateBindingServiceForExec/Synccorrectly:
- Clear and redefine the
BindVariableServiceper pipeline entry.- Walk the arena safely using
msgBindParameterValueCountand per-parameter format codes.- Respect
PG_UNSPECIFIEDentries (unused gaps) and null values (valueSize == -1).- Use native types from
Numbers.decodeLowInt(...)so QuestDB functions get the exact types inferred at compile time.Two points to double‑check:
CTAS and other DDL that can contain expressions –
populateBindingServiceForExeccurrently coversEXPLAIN/SELECT/PSEUDO_SELECT/INSERT/INSERT_AS_SELECT/UPDATE/ALTERbut notCREATE_TABLE_AS_SELECT. If CTAS statements can reference$nbind variables in their SELECT part (which seems likely), you probably want to includeCompiledQuery.CREATE_TABLE_AS_SELECTin this switch so CTAS EXEC runs with the correct bind values.SYNC helper scope –
populateBindingServiceForSyncintentionally only handlesEXPLAIN/SELECT/PSEUDO_SELECT, which matches the cases where re-compilation or cursor recreation may happen during SYNC. That looks reasonable and avoids unnecessary rebinds for pure DML/DDL.If CTAS does use bind parameters in practice, consider extending the EXEC helper accordingly:
- return switch (sqlType) { + return switch (sqlType) { // these query types use binding variables at the EXEC time - case CompiledQuery.EXPLAIN, CompiledQuery.SELECT, CompiledQuery.PSEUDO_SELECT, CompiledQuery.INSERT, - CompiledQuery.INSERT_AS_SELECT, CompiledQuery.UPDATE, CompiledQuery.ALTER -> { + case CompiledQuery.EXPLAIN, CompiledQuery.SELECT, CompiledQuery.PSEUDO_SELECT, + CompiledQuery.INSERT, CompiledQuery.INSERT_AS_SELECT, + CompiledQuery.UPDATE, CompiledQuery.ALTER, + CompiledQuery.CREATE_TABLE_AS_SELECT -> { copyParameterValuesToBindVariableService( sqlExecutionContext, bindVariableCharacterStore, directUtf8String, binarySequenceParamsPool ); yield true; } default -> false; };Also applies to: 3444-3487
core/src/main/java/io/questdb/cutlass/pgwire/PGConnectionContext.java (1)
1390-1411: SYNC-side rebinding and error handling fix the pipelined bind bug; minor error-message nuanceThe updated
syncPipeline()loop looks correct and addresses the pipelined-SELECT bug:
- Before each
msgSynccall, you now (re)populate theBindVariableServicefor the currentpipelineCurrentEntrywhenbindingServiceConfiguredForpoints to a different entry andpopulateBindingServiceForSync(...)returnstrue. This ensures each entry’s SYNC phase sees its own bound parameter values even when EXECs were interleaved for multiple entries.- After an entry is fully processed (or in error/close cases), you clear
bindingServiceConfiguredForwhen it was pointing at that entry, avoiding stale ownership across reused pooled entries.- The
catch (PGMessageProcessingException | SqlException e)path now:
- Captures the textual error into the entry’s error sink.
- Invokes
msgSync(...)once to emit the error response.- Breaks out of the inner
while (true)loop, avoiding the infinite retry behavior noted in the earlier review.One small refinement you might consider (non-blocking):
- For
SqlException(which implementsFlyweightMessageContainer), usinggetFlyweightMessage()andgetPosition()(mirroring the pattern inPGPipelineEntry.msgExecute’s catch) would produce more consistent error messages and preserve position info, instead of relying one.getMessage()stringification.Example sketch:
- } catch (PGMessageProcessingException | SqlException e) { - pipelineCurrentEntry.getErrorMessageSink().put(e.getMessage()); + } catch (PGMessageProcessingException | SqlException e) { + if (e instanceof FlyweightMessageContainer f) { + int pos = f.getPosition(); + if (pos >= 0) { + pipelineCurrentEntry.setErrorMessagePosition(pos); + } + pipelineCurrentEntry.getErrorMessageSink().put(f.getFlyweightMessage()); + } else { + pipelineCurrentEntry.getErrorMessageSink().put(e.getMessage()); + } pipelineCurrentEntry.msgSync( sqlExecutionContext, pendingWriters, responseUtf8Sink ); break; }This keeps the current control flow but improves diagnostics for SQL errors.
Also applies to: 1435-1454
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
compat/src/test/csharp/ExtraTests.cs(1 hunks)compat/src/test/csharp/Program.cs(1 hunks)core/src/main/java/io/questdb/cutlass/pgwire/PGConnectionContext.java(7 hunks)core/src/main/java/io/questdb/cutlass/pgwire/PGPipelineEntry.java(3 hunks)
🧰 Additional context used
🧬 Code graph analysis (3)
compat/src/test/csharp/Program.cs (1)
compat/src/test/csharp/ExtraTests.cs (1)
ExtraTests(10-82)
core/src/main/java/io/questdb/cutlass/pgwire/PGConnectionContext.java (1)
core/src/main/java/io/questdb/griffin/SqlException.java (1)
SqlException(37-268)
core/src/main/java/io/questdb/cutlass/pgwire/PGPipelineEntry.java (1)
core/src/main/java/io/questdb/std/Numbers.java (1)
Numbers(44-3523)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (34)
- GitHub Check: New pull request (Coverage Report Coverage Report)
- GitHub Check: New pull request (SelfHosted Running tests with cover on linux-other)
- GitHub Check: New pull request (SelfHosted Running tests with cover on linux-pgwire)
- GitHub Check: New pull request (SelfHosted Running tests with cover on linux-cairo-sub)
- GitHub Check: New pull request (SelfHosted Running tests with cover on linux-cairo-root)
- GitHub Check: New pull request (SelfHosted Running tests with cover on linux-fuzz2)
- GitHub Check: New pull request (SelfHosted Running tests with cover on linux-fuzz1)
- GitHub Check: New pull request (SelfHosted Running tests with cover on linux-griffin-sub)
- GitHub Check: New pull request (SelfHosted Other tests on linux-x86-graal)
- GitHub Check: New pull request (SelfHosted Running tests with cover on linux-griffin-root)
- GitHub Check: New pull request (Rust Test and Lint on linux-jdk17)
- GitHub Check: New pull request (SelfHosted Other tests on linux-arm64)
- GitHub Check: New pull request (SelfHosted Other tests on linux-x64-zfs)
- GitHub Check: New pull request (Hosted Running tests on windows-other-2)
- GitHub Check: New pull request (Hosted Running tests on windows-other-1)
- GitHub Check: New pull request (Hosted Running tests on windows-pgwire)
- GitHub Check: New pull request (Hosted Running tests on windows-cairo-2)
- GitHub Check: New pull request (Hosted Running tests on windows-cairo-1)
- GitHub Check: New pull request (Hosted Running tests on windows-fuzz2)
- GitHub Check: New pull request (Hosted Running tests on windows-fuzz1)
- GitHub Check: New pull request (Hosted Running tests on windows-griffin-sub)
- GitHub Check: New pull request (Hosted Running tests on windows-griffin-base)
- GitHub Check: New pull request (Hosted Running tests on mac-other)
- GitHub Check: New pull request (Hosted Running tests on mac-pgwire)
- GitHub Check: New pull request (Hosted Running tests on mac-cairo-fuzz)
- GitHub Check: New pull request (Hosted Running tests on mac-cairo)
- GitHub Check: New pull request (SelfHosted Griffin tests on linux-x64-zfs)
- GitHub Check: New pull request (SelfHosted Griffin tests on linux-arm64)
- GitHub Check: New pull request (Hosted Running tests on mac-griffin)
- GitHub Check: New pull request (SelfHosted Griffin tests on linux-x86-graal)
- GitHub Check: New pull request (SelfHosted Cairo tests on linux-arm64)
- GitHub Check: New pull request (SelfHosted Cairo tests on linux-x86-graal)
- GitHub Check: New pull request (SelfHosted Cairo tests on linux-x64-zfs)
- GitHub Check: New pull request (Check Changes Check changes)
🔇 Additional comments (6)
compat/src/test/csharp/Program.cs (1)
58-59: LGTM!The integration point is well-placed at the end of
RunTests, ensuring all YAML-based tests complete before running the Npgsql-specific extra tests. The call is properly awaited, and error handling is delegated toExtraTests.RunAll()which follows the sameEnvironment.Exit(1)pattern used elsewhere in this file.compat/src/test/csharp/ExtraTests.cs (3)
5-15: Well-documented purpose.The class documentation clearly explains why these tests exist separately from the YAML-based tests. The
RunAll()pattern provides a clean extension point for adding additional Npgsql-specific tests in the future.
37-46: Effective test design for the regression.The approach of embedding the index directly in SQL (
{i} AS UnboundIndex) and comparing it with the bound parameter value is a clever way to detect parameter binding issues. The batch size of 5 is sufficient for catching the regression while keeping test execution fast.
76-80: Consistent error handling pattern.The exception handling follows the same pattern used in
Program.cs, ensuring the test suite fails with exit code 1 on any error. This is appropriate for a regression test.core/src/main/java/io/questdb/cutlass/pgwire/PGPipelineEntry.java (1)
597-662: Exec path now consistently (re)populates bind variables before dispatch – looks correctSwitching to
populateBindingServiceForExec(...)here centralizes bind population for all EXEC-time users and keepsmsgExecute’s switch focused on SQL-type dispatch. Control flow and error handling are preserved, and thefinallystill clears only the character store, not the bind values themselves, which is important for later SYNC usage.core/src/main/java/io/questdb/cutlass/pgwire/PGConnectionContext.java (1)
46-47: Binding-service ownership tracking is well-integrated
- Importing
SqlExceptionis required by the newsyncPipelinecatch.bindingServiceConfiguredForis correctly:
- Initialized to
nullin the constructor.- Reset in
clear().- Set to the current
pipelineCurrentEntryonmsgExecute, just before delegating to the entry’smsgExecute(...).This makes it straightforward to know which entry last configured the shared
BindVariableServiceand is a good basis for the SYNC-side rebinding logic.Also applies to: 180-181, 347-348, 917-946
[PR Coverage check]😍 pass : 59 / 74 (79.73%) file detail
|
|
Fixed a bug where queries with bound parameters executed in a batch would return incorrect results. When multiple SELECT statements with different parameter values were sent together (pipelining), all queries would incorrectly use the parameter values from the last query in the batch instead of their own values.
This affected PostgreSQL clients that use pipelining, such as Npgsql (.NET), where multiple queries are sent before waiting for responses. Each query now correctly uses its own bound parameter values.
I could not find a way to trigger this behaviour from a Java test using PGJDBC/R2DBC-Postgres. PGJDBC does not allow batch SELECTs at all. R2DBC supports it in the API, but then it sends SYNC for each SELECT anyway: pgjdbc/r2dbc-postgresql#527
The C# test/reproducer contributed by @nwoolmer, 🙇
Fixes #6123
Tandem: https://github.com/questdb/questdb-enterprise/pull/795
Supersedes #6401