Skip to content

fix(pgwire): batch SELECTs with bound parameters return correct results#6453

Merged
bluestreak01 merged 14 commits intomasterfrom
jh_binding_service_select_batches
Nov 28, 2025
Merged

fix(pgwire): batch SELECTs with bound parameters return correct results#6453
bluestreak01 merged 14 commits intomasterfrom
jh_binding_service_select_batches

Conversation

@jerrinot
Copy link
Copy Markdown
Contributor

@jerrinot jerrinot commented Nov 25, 2025

Fixed a bug where queries with bound parameters executed in a batch would return incorrect results. When multiple SELECT statements with different parameter values were sent together (pipelining), all queries would incorrectly use the parameter values from the last query in the batch instead of their own values.

This affected PostgreSQL clients that use pipelining, such as Npgsql (.NET), where multiple queries are sent before waiting for responses. Each query now correctly uses its own bound parameter values.

I could not find a way to trigger this behaviour from a Java test using PGJDBC/R2DBC-Postgres. PGJDBC does not allow batch SELECTs at all. R2DBC supports it in the API, but then it sends SYNC for each SELECT anyway: pgjdbc/r2dbc-postgresql#527

The C# test/reproducer contributed by @nwoolmer, 🙇

Fixes #6123
Tandem: https://github.com/questdb/questdb-enterprise/pull/795
Supersedes #6401

@coderabbitai
Copy link
Copy Markdown

coderabbitai bot commented Nov 25, 2025

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Note

Other AI code review bot(s) detected

CodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review.

Walkthrough

The changes implement a fix for handling bound parameters in batched PostgreSQL queries. A new C# test validates parameter binding consistency in NpgsqlBatch operations, while Java changes refactor the pipeline execution logic to properly manage binding service state across EXEC and SYNC lifecycle phases.

Changes

Cohort / File(s) Summary
C# Test Suite
compat/src/test/csharp/ExtraTests.cs, compat/src/test/csharp/Program.cs
New test class ExtraTests with RunAll entry point validates batched parameter binding by executing 5 NpgsqlBatch commands with parameters and asserting bound/unbound indices match. Program.cs integrates the new test into the main test runner.
PostgreSQL Pipeline State Management
core/src/main/java/io/questdb/cutlass/pgwire/PGConnectionContext.java
Adds bindingServiceConfiguredFor field to track binding service state per pipeline entry. Implements logic to repopulate binding service during sync operations, reset on clear, and manage state transitions across pipeline progression. Includes exception handling for SqlException.
PostgreSQL Pipeline Binding Logic
core/src/main/java/io/questdb/cutlass/pgwire/PGPipelineEntry.java
Introduces populateBindingServiceForExec and populateBindingServiceForSync methods to conditionally populate binding variables based on compiled query type. Refactors msgExecute to use populateBindingServiceForExec instead of direct copyParameterValuesToBindVariableService invocation.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

  • The Java binding service state management requires careful verification of state transitions across pipeline entry lifecycle, particularly around EXEC/SYNC phase boundaries
  • Ensure bindingServiceConfiguredFor tracking correctly handles concurrent batch operations and doesn't retain stale state
  • Validate that populateBindingServiceForExec and populateBindingServiceForSync methods correctly identify which CompiledQuery types require binding population during each phase
  • Verify the C# test adequately reproduces the issue from the linked bug report with sufficient batch size and parameter variation

Suggested reviewers

  • puzpuzpuz

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)
Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 17.65% which is insufficient. The required threshold is 80.00%. You can run @coderabbitai generate docstrings to improve docstring coverage.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately describes the main fix: resolving bound parameters in batched SELECT queries returning incorrect results.
Description check ✅ Passed The description clearly relates to the changeset, explaining the bug fix and providing context about the affected clients and testing approach.
Linked Issues check ✅ Passed The PR addresses issue #6123 by fixing bound parameter handling in batched queries. The changes maintain correct binding state across pipeline entries during EXEC and SYNC phases.
Out of Scope Changes check ✅ Passed All changes are scoped to fixing batched parameter binding: Java binding service state management, test validation in C#, and integration in the test runner.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@jerrinot jerrinot changed the title fix(pgwire): fix batch selects fix(pgwire): fix batch selects [DO NOT MERGE] Nov 25, 2025
@jerrinot jerrinot added Bug Incorrect or unexpected behavior Postgres Wire Issues or changes relating to Postgres wire protocol DO NOT MERGE These changes should not be merged to main branch labels Nov 25, 2025
@jerrinot jerrinot marked this pull request as draft November 25, 2025 17:36
@jerrinot jerrinot marked this pull request as ready for review November 26, 2025 18:29
@jerrinot jerrinot changed the title fix(pgwire): fix batch selects [DO NOT MERGE] fix(pgwire): batch selects with bound parameters return correct results Nov 26, 2025
@jerrinot jerrinot removed the DO NOT MERGE These changes should not be merged to main branch label Nov 26, 2025
@jerrinot
Copy link
Copy Markdown
Contributor Author

@coderabbitai please review

@jerrinot jerrinot requested a review from Copilot November 26, 2025 19:04
@jerrinot jerrinot changed the title fix(pgwire): batch selects with bound parameters return correct results fix(pgwire): batch SELECTs with bound parameters return correct results Nov 26, 2025
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a critical bug in the PostgreSQL wire protocol implementation where batched SELECT queries with bound parameters would incorrectly share parameter values. When multiple queries with different parameters were pipelined (sent before waiting for responses), all queries would use the parameter values from the last query in the batch instead of their respective values.

Key changes:

  • Refactored bind variable population to occur at both EXEC and SYNC message processing stages
  • Introduced tracking to ensure each pipeline entry uses its own bound parameters
  • Added C# tests using Npgsql's batching capabilities to verify the fix

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File Description
PGPipelineEntry.java Made copyParameterValuesToBindVariableService package-private and moved it; added populateBindingServiceForExec and populateBindingServiceForSync methods to control when parameters are copied based on query type
PGConnectionContext.java Added bindingServiceConfiguredFor tracking field; integrated parameter population in msgExecute and syncPipeline methods to ensure each entry uses correct parameters
Program.cs Added call to run extra C#-specific tests
ExtraTests.cs New file with Npgsql-specific test for batched queries with bound parameters

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

… it requires an error while copying to a binding service. but at this point we had already copied the same data at the EXEC phase.
@bluestreak01
Copy link
Copy Markdown
Member

@CodeRabbit happy thanksgiving, can you review?

@coderabbitai
Copy link
Copy Markdown

coderabbitai bot commented Nov 26, 2025

@bluestreak01: Happy Thanksgiving to you too! 🦃 I'd be happy to review the changes. Let me take a look at the fix for the batch SELECT issue with bound parameters.

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Copy link
Copy Markdown

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (3)
compat/src/test/csharp/ExtraTests.cs (1)

50-72: Consider simplifying the result iteration pattern.

The first flag pattern works correctly, but the iteration could be slightly more idiomatic.

-            var first = true;
-            for (var i = 0; i < batch.BatchCommands.Count; i++)
-            {
-                if (!first)
-                {
-                    await reader.NextResultAsync();
-                }
-                first = false;
-
-                if (!await reader.ReadAsync())
+            for (var i = 0; i < batch.BatchCommands.Count; i++)
+            {
+                if (i > 0 && !await reader.NextResultAsync())
+                {
+                    throw new InvalidOperationException($"Expected result set {i} not found");
+                }
+
+                if (!await reader.ReadAsync())
                 {
                     throw new InvalidOperationException("No rows returned");
                 }

This eliminates the first flag and adds validation that NextResultAsync succeeds. That said, the current implementation is correct and this is purely a style preference.

core/src/main/java/io/questdb/cutlass/pgwire/PGPipelineEntry.java (1)

3271-3398: New bind replay helpers look sound; please confirm CTAS coverage

The new copyParameterValuesToBindVariableService plus populateBindingServiceForExec/Sync correctly:

  • Clear and redefine the BindVariableService per pipeline entry.
  • Walk the arena safely using msgBindParameterValueCount and per-parameter format codes.
  • Respect PG_UNSPECIFIED entries (unused gaps) and null values (valueSize == -1).
  • Use native types from Numbers.decodeLowInt(...) so QuestDB functions get the exact types inferred at compile time.

Two points to double‑check:

  1. CTAS and other DDL that can contain expressionspopulateBindingServiceForExec currently covers EXPLAIN/SELECT/PSEUDO_SELECT/INSERT/INSERT_AS_SELECT/UPDATE/ALTER but not CREATE_TABLE_AS_SELECT. If CTAS statements can reference $n bind variables in their SELECT part (which seems likely), you probably want to include CompiledQuery.CREATE_TABLE_AS_SELECT in this switch so CTAS EXEC runs with the correct bind values.

  2. SYNC helper scopepopulateBindingServiceForSync intentionally only handles EXPLAIN/SELECT/PSEUDO_SELECT, which matches the cases where re-compilation or cursor recreation may happen during SYNC. That looks reasonable and avoids unnecessary rebinds for pure DML/DDL.

If CTAS does use bind parameters in practice, consider extending the EXEC helper accordingly:

-        return switch (sqlType) {
+        return switch (sqlType) {
             // these query types use binding variables at the EXEC time
-            case CompiledQuery.EXPLAIN, CompiledQuery.SELECT, CompiledQuery.PSEUDO_SELECT, CompiledQuery.INSERT,
-                 CompiledQuery.INSERT_AS_SELECT, CompiledQuery.UPDATE, CompiledQuery.ALTER -> {
+            case CompiledQuery.EXPLAIN, CompiledQuery.SELECT, CompiledQuery.PSEUDO_SELECT,
+                 CompiledQuery.INSERT, CompiledQuery.INSERT_AS_SELECT,
+                 CompiledQuery.UPDATE, CompiledQuery.ALTER,
+                 CompiledQuery.CREATE_TABLE_AS_SELECT -> {
                 copyParameterValuesToBindVariableService(
                         sqlExecutionContext,
                         bindVariableCharacterStore,
                         directUtf8String,
                         binarySequenceParamsPool
                 );
                 yield true;
             }
             default -> false;
         };

Also applies to: 3444-3487

core/src/main/java/io/questdb/cutlass/pgwire/PGConnectionContext.java (1)

1390-1411: SYNC-side rebinding and error handling fix the pipelined bind bug; minor error-message nuance

The updated syncPipeline() loop looks correct and addresses the pipelined-SELECT bug:

  • Before each msgSync call, you now (re)populate the BindVariableService for the current pipelineCurrentEntry when bindingServiceConfiguredFor points to a different entry and populateBindingServiceForSync(...) returns true. This ensures each entry’s SYNC phase sees its own bound parameter values even when EXECs were interleaved for multiple entries.
  • After an entry is fully processed (or in error/close cases), you clear bindingServiceConfiguredFor when it was pointing at that entry, avoiding stale ownership across reused pooled entries.
  • The catch (PGMessageProcessingException | SqlException e) path now:
    • Captures the textual error into the entry’s error sink.
    • Invokes msgSync(...) once to emit the error response.
    • Breaks out of the inner while (true) loop, avoiding the infinite retry behavior noted in the earlier review.

One small refinement you might consider (non-blocking):

  • For SqlException (which implements FlyweightMessageContainer), using getFlyweightMessage() and getPosition() (mirroring the pattern in PGPipelineEntry.msgExecute’s catch) would produce more consistent error messages and preserve position info, instead of relying on e.getMessage() stringification.

Example sketch:

-                } catch (PGMessageProcessingException | SqlException e) {
-                    pipelineCurrentEntry.getErrorMessageSink().put(e.getMessage());
+                } catch (PGMessageProcessingException | SqlException e) {
+                    if (e instanceof FlyweightMessageContainer f) {
+                        int pos = f.getPosition();
+                        if (pos >= 0) {
+                            pipelineCurrentEntry.setErrorMessagePosition(pos);
+                        }
+                        pipelineCurrentEntry.getErrorMessageSink().put(f.getFlyweightMessage());
+                    } else {
+                        pipelineCurrentEntry.getErrorMessageSink().put(e.getMessage());
+                    }
                     pipelineCurrentEntry.msgSync(
                             sqlExecutionContext,
                             pendingWriters,
                             responseUtf8Sink
                     );
                     break;
                 }

This keeps the current control flow but improves diagnostics for SQL errors.

Also applies to: 1435-1454

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between da5ff81 and 270f303.

📒 Files selected for processing (4)
  • compat/src/test/csharp/ExtraTests.cs (1 hunks)
  • compat/src/test/csharp/Program.cs (1 hunks)
  • core/src/main/java/io/questdb/cutlass/pgwire/PGConnectionContext.java (7 hunks)
  • core/src/main/java/io/questdb/cutlass/pgwire/PGPipelineEntry.java (3 hunks)
🧰 Additional context used
🧬 Code graph analysis (3)
compat/src/test/csharp/Program.cs (1)
compat/src/test/csharp/ExtraTests.cs (1)
  • ExtraTests (10-82)
core/src/main/java/io/questdb/cutlass/pgwire/PGConnectionContext.java (1)
core/src/main/java/io/questdb/griffin/SqlException.java (1)
  • SqlException (37-268)
core/src/main/java/io/questdb/cutlass/pgwire/PGPipelineEntry.java (1)
core/src/main/java/io/questdb/std/Numbers.java (1)
  • Numbers (44-3523)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (34)
  • GitHub Check: New pull request (Coverage Report Coverage Report)
  • GitHub Check: New pull request (SelfHosted Running tests with cover on linux-other)
  • GitHub Check: New pull request (SelfHosted Running tests with cover on linux-pgwire)
  • GitHub Check: New pull request (SelfHosted Running tests with cover on linux-cairo-sub)
  • GitHub Check: New pull request (SelfHosted Running tests with cover on linux-cairo-root)
  • GitHub Check: New pull request (SelfHosted Running tests with cover on linux-fuzz2)
  • GitHub Check: New pull request (SelfHosted Running tests with cover on linux-fuzz1)
  • GitHub Check: New pull request (SelfHosted Running tests with cover on linux-griffin-sub)
  • GitHub Check: New pull request (SelfHosted Other tests on linux-x86-graal)
  • GitHub Check: New pull request (SelfHosted Running tests with cover on linux-griffin-root)
  • GitHub Check: New pull request (Rust Test and Lint on linux-jdk17)
  • GitHub Check: New pull request (SelfHosted Other tests on linux-arm64)
  • GitHub Check: New pull request (SelfHosted Other tests on linux-x64-zfs)
  • GitHub Check: New pull request (Hosted Running tests on windows-other-2)
  • GitHub Check: New pull request (Hosted Running tests on windows-other-1)
  • GitHub Check: New pull request (Hosted Running tests on windows-pgwire)
  • GitHub Check: New pull request (Hosted Running tests on windows-cairo-2)
  • GitHub Check: New pull request (Hosted Running tests on windows-cairo-1)
  • GitHub Check: New pull request (Hosted Running tests on windows-fuzz2)
  • GitHub Check: New pull request (Hosted Running tests on windows-fuzz1)
  • GitHub Check: New pull request (Hosted Running tests on windows-griffin-sub)
  • GitHub Check: New pull request (Hosted Running tests on windows-griffin-base)
  • GitHub Check: New pull request (Hosted Running tests on mac-other)
  • GitHub Check: New pull request (Hosted Running tests on mac-pgwire)
  • GitHub Check: New pull request (Hosted Running tests on mac-cairo-fuzz)
  • GitHub Check: New pull request (Hosted Running tests on mac-cairo)
  • GitHub Check: New pull request (SelfHosted Griffin tests on linux-x64-zfs)
  • GitHub Check: New pull request (SelfHosted Griffin tests on linux-arm64)
  • GitHub Check: New pull request (Hosted Running tests on mac-griffin)
  • GitHub Check: New pull request (SelfHosted Griffin tests on linux-x86-graal)
  • GitHub Check: New pull request (SelfHosted Cairo tests on linux-arm64)
  • GitHub Check: New pull request (SelfHosted Cairo tests on linux-x86-graal)
  • GitHub Check: New pull request (SelfHosted Cairo tests on linux-x64-zfs)
  • GitHub Check: New pull request (Check Changes Check changes)
🔇 Additional comments (6)
compat/src/test/csharp/Program.cs (1)

58-59: LGTM!

The integration point is well-placed at the end of RunTests, ensuring all YAML-based tests complete before running the Npgsql-specific extra tests. The call is properly awaited, and error handling is delegated to ExtraTests.RunAll() which follows the same Environment.Exit(1) pattern used elsewhere in this file.

compat/src/test/csharp/ExtraTests.cs (3)

5-15: Well-documented purpose.

The class documentation clearly explains why these tests exist separately from the YAML-based tests. The RunAll() pattern provides a clean extension point for adding additional Npgsql-specific tests in the future.


37-46: Effective test design for the regression.

The approach of embedding the index directly in SQL ({i} AS UnboundIndex) and comparing it with the bound parameter value is a clever way to detect parameter binding issues. The batch size of 5 is sufficient for catching the regression while keeping test execution fast.


76-80: Consistent error handling pattern.

The exception handling follows the same pattern used in Program.cs, ensuring the test suite fails with exit code 1 on any error. This is appropriate for a regression test.

core/src/main/java/io/questdb/cutlass/pgwire/PGPipelineEntry.java (1)

597-662: Exec path now consistently (re)populates bind variables before dispatch – looks correct

Switching to populateBindingServiceForExec(...) here centralizes bind population for all EXEC-time users and keeps msgExecute’s switch focused on SQL-type dispatch. Control flow and error handling are preserved, and the finally still clears only the character store, not the bind values themselves, which is important for later SYNC usage.

core/src/main/java/io/questdb/cutlass/pgwire/PGConnectionContext.java (1)

46-47: Binding-service ownership tracking is well-integrated

  • Importing SqlException is required by the new syncPipeline catch.
  • bindingServiceConfiguredFor is correctly:
    • Initialized to null in the constructor.
    • Reset in clear().
    • Set to the current pipelineCurrentEntry on msgExecute, just before delegating to the entry’s msgExecute(...).

This makes it straightforward to know which entry last configured the shared BindVariableService and is a good basis for the SYNC-side rebinding logic.

Also applies to: 180-181, 347-348, 917-946

@glasstiger
Copy link
Copy Markdown
Contributor

[PR Coverage check]

😍 pass : 59 / 74 (79.73%)

file detail

path covered line new line coverage
🔵 io/questdb/cutlass/pgwire/PGConnectionContext.java 6 10 60.00%
🔵 io/questdb/cutlass/pgwire/PGPipelineEntry.java 53 64 82.81%

@bluestreak01 bluestreak01 merged commit 360948a into master Nov 28, 2025
42 of 43 checks passed
@bluestreak01 bluestreak01 deleted the jh_binding_service_select_batches branch November 28, 2025 19:20
@questdb-butler
Copy link
Copy Markdown

⚠️ Enterprise CI Failed

The enterprise test suite failed for this PR.

Build: View Details
Tested Commit: ac38f26bae68eb8e5ad9fc3d0b49134d2ff2b227

Please investigate the failure before merging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Bug Incorrect or unexpected behavior Postgres Wire Issues or changes relating to Postgres wire protocol

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bound parameters in batches is broken

5 participants