Skip to content

feat(ilp): add support for configuring multiple possible database endpoints#6074

Merged
bluestreak01 merged 77 commits intomasterfrom
nw_ilp_multi_url
Sep 30, 2025
Merged

feat(ilp): add support for configuring multiple possible database endpoints#6074
bluestreak01 merged 77 commits intomasterfrom
nw_ilp_multi_url

Conversation

@nwoolmer
Copy link
Copy Markdown
Contributor

@nwoolmer nwoolmer commented Aug 22, 2025

tandem with https://github.com/questdb/questdb-enterprise/pull/669

This PR adds support to the Java ILP client (and the QuestDB server) to handle multiple possible endpoints. On first connection, the client will poll /settings and check if the instance is read-write or read-only. Additionally, if this changes behind the scenes after the sender has been initialised, then the sender will automatically retry HTTP sends on 404 or 421 responses it receives.

@nwoolmer nwoolmer added ILP Issues or changes relating to Influx Line Protocol Friction tandem labels Aug 22, 2025
@coderabbitai
Copy link
Copy Markdown

coderabbitai bot commented Aug 22, 2025

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Walkthrough

Adds write-acceptance signaling to configs and settings, introduces HTTP 421 handling, gates ILP writes when not accepting, extends HTTP client API and errno reporting, implements multi-host HTTP line sender with retry/rotation and reset(), and updates tests accordingly. Minor utilities added (IntList factory, Rnd sync), and TCP path guarded.

Changes

Cohort / File(s) Summary
Config: accepting writes/reads flags
core/src/main/java/io/questdb/PropServerConfiguration.java, core/src/main/java/io/questdb/cairo/CairoConfiguration.java, core/src/main/java/io/questdb/cairo/CairoConfigurationWrapper.java, core/src/main/java/io/questdb/cairo/DefaultCairoConfiguration.java
Added ACCEPTING_READS/WRITES constants and isAcceptingWrites API with default true; wrapper delegates; PropCairoConfiguration computes from HTTP read-only context.
HTTP status 421 support
core/src/main/java/io/questdb/cutlass/http/HttpResponseSink.java
Added HTTP_MISDIRECTED_REQUEST (421) and status text mapping.
HTTP server ILP enablement
core/src/main/java/io/questdb/cutlass/http/HttpServer.java
Removed read-only-security check from ILP HTTP enablement condition.
Settings exposure
core/src/main/java/io/questdb/cutlass/http/processors/SettingsProcessor.java
Settings JSON now includes accepting.writes (from config) and accepting.reads (true).
ILP HTTP processor gating
core/src/main/java/io/questdb/cutlass/http/processors/LineHttpProcessorImpl.java, core/src/main/java/io/questdb/cutlass/http/processors/LineHttpProcessorState.java
Added NOT_ACCEPTING_WRITES status (421) and early rejection when engine not accepting writes.
HTTP client send/errno
core/src/main/java/io/questdb/cutlass/http/client/HttpClient.java, core/src/main/java/io/questdb/cutlass/http/client/HttpClientException.java
Request.send(host,port,timeout) introduced; timeout-only overload delegates. Enhanced exceptions with errno and propagation across failures (connect, non-blocking, TLS).
Line Sender: multi-host HTTP + reset/retry
core/src/main/java/io/questdb/client/Sender.java, core/src/main/java/io/questdb/cutlass/line/AbstractLineSender.java, core/src/main/java/io/questdb/cutlass/line/LineSenderException.java, core/src/main/java/io/questdb/cutlass/line/http/AbstractLineHttpSender.java, core/src/main/java/io/questdb/cutlass/line/http/LineHttpSenderV1.java, core/src/main/java/io/questdb/cutlass/line/http/LineHttpSenderV2.java
Sender builder supports multiple hosts/ports; HTTP sender rotates addresses, handles 421/404, blacklists, tracks accepting.writes from /settings; reset() added (HTTP-only supported); exceptions carry retryable flag; V1/V2 constructors extended for multi-host.
TCP sender minor
core/src/main/java/io/questdb/cutlass/line/LineTcpSenderV2.java
Formatting-only change.
Utils additions
core/src/main/java/io/questdb/std/IntList.java, core/src/main/java/io/questdb/std/Rnd.java
Added IntList.createWithValues(...) and synchronized Rnd.nextIntSync(int).
Tests: settings flags
core/src/test/java/io/questdb/test/ServerMainHttpAuthTest.java, core/src/test/java/io/questdb/test/cutlass/http/SettingsEndpointTest.java
Updated/added expectations for accepting.writes and accepting.reads in settings payloads.
Tests: multi-host HTTP + failover
core/src/test/java/io/questdb/test/cutlass/http/line/LineHttpMultiUrlTest.java
New suite covering two-server scenarios, read-only/read-write, failover, retries, and WAL drainage.
Tests: builder and messages
core/src/test/java/io/questdb/test/client/LineSenderBuilderTest.java
Adjusted messages for multi-address validation; added TCP auto-flush interval unsupported test; typo fix.
Tests: sender reset/retry flows
core/src/test/java/io/questdb/test/cutlass/http/line/LineHttpSenderMockServerTest.java, core/src/test/java/io/questdb/test/cutlass/http/line/LineHttpSenderTest.java, core/src/test/java/io/questdb/test/cutlass/http/line/LineHttpsSenderTest.java
Added sender.reset() after errors; renamed/added tests to cover retriable errors and protocol version retry.
Test utility
core/src/test/java/io/questdb/test/TestServerMain.java
Added printSql helper to execute SQL and capture output.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  participant App as Application
  participant Builder as Sender.Builder
  participant LHS as AbstractLineHttpSender
  participant HC as HttpClient
  participant S1 as Server A
  participant S2 as Server B

  App->>Builder: address(host1), port(p1), address(host2), port(p2)
  Builder-->>App: build() -> HTTP sender (multi-host)

  App->>LHS: table(...).symbol(...).flush()
  alt No protocol/version yet
    LHS->>HC: GET /settings on host[current]
    HC-->>LHS: 200 JSON { protocol, accepting.writes }
    LHS-->>App: proceed
  end

  LHS->>HC: POST /imp (lines) to host[current]
  alt 204 No Content
    HC-->>LHS: 204
    LHS-->>App: ok
  else 421 Misdirected/Not accepting
    HC-->>LHS: 421
    LHS->>LHS: rotate to next host
    LHS->>HC: retry POST on next host
    HC-->>LHS: 204
    LHS-->>App: ok
  else 5xx retriable
    HC-->>LHS: 5xx
    LHS-->>App: throw LineSenderException(retryable=true)
  end
Loading
sequenceDiagram
  autonumber
  participant Client as HTTP Client
  participant Req as Request
  participant Net as Network
  participant TLS as TLS
  Client->>Req: send(host, port, timeout)
  Req->>Net: connect(host, port)
  alt connect error
    Net-->>Req: errno
    Req-->>Client: HttpClientException(errno)
  else connect ok
    Req->>TLS: start if enabled
    alt TLS error
      TLS-->>Req: errno
      Req-->>Client: HttpClientException(errno)
    else TLS ok
      Req->>Net: write request, read response
      Net-->>Req: status/headers/body
      Req-->>Client: ResponseHeaders
    end
  end
Loading
sequenceDiagram
  autonumber
  participant Sender as Line HTTP Sender
  participant Server as HTTP Server
  participant Proc as LineHttpProcessor
  participant Engine as Cairo Engine

  Sender->>Server: POST /imp (lines)
  Server->>Proc: onHeadersReady()
  Proc->>Engine: config.isAcceptingWrites()
  alt Not accepting
    Engine-->>Proc: false
    Proc-->>Server: Status NOT_ACCEPTING_WRITES (421)
    Server-->>Sender: 421
  else Accepting
    Engine-->>Proc: true
    Proc-->>Server: proceed to ingest
    Server-->>Sender: 204
  end
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~75 minutes

Suggested labels

Core

Suggested reviewers

  • jerrinot

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)
Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 6.99% which is insufficient. The required threshold is 80.00%. You can run @coderabbitai generate docstrings to improve docstring coverage.
✅ Passed checks (2 passed)
Check name Status Explanation
Title Check ✅ Passed The title accurately and concisely summarizes the primary change—adding ILP support for multiple database endpoints—without noise or ambiguity, and it maps directly to the multi-host client/server changes in the diff.
Description Check ✅ Passed The description clearly describes the intent and key behaviors added by the PR—multi-endpoint support, initial /settings polling to detect read-write vs read-only, and automatic retries on 404/421—and aligns with the code changes across client and server files.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share
🧪 Early access (Sonnet 4.5): enabled

We are currently testing the Sonnet 4.5 model, which is expected to improve code review quality. However, this model may lead to increased noise levels in the review comments. Please disable the early access features if the noise level causes any inconvenience.

Note:

  • Public repositories are always opted into early access features.
  • You can enable or disable early access features from the CodeRabbit UI or by updating the CodeRabbit configuration file.

Comment @coderabbitai help to get the list of available commands and usage tips.

# Conflicts:
#	core/src/main/java/io/questdb/cutlass/line/http/AbstractLineHttpSender.java
#	core/src/main/java/io/questdb/cutlass/line/http/LineHttpSenderV1.java
#	core/src/main/java/io/questdb/cutlass/line/http/LineHttpSenderV2.java
@jerrinot jerrinot self-requested a review September 8, 2025 14:49
@puzpuzpuz puzpuzpuz self-requested a review September 30, 2025 15:23
@glasstiger
Copy link
Copy Markdown
Contributor

[PR Coverage check]

😍 pass : 225 / 250 (90.00%)

file detail

path covered line new line coverage
🔵 io/questdb/cutlass/line/AbstractLineSender.java 0 1 00.00%
🔵 io/questdb/cutlass/http/DefaultHttpServerConfiguration.java 0 1 00.00%
🔵 io/questdb/cutlass/http/client/HttpClient.java 3 7 42.86%
🔵 io/questdb/cutlass/http/processors/LineHttpProcessorImpl.java 7 9 77.78%
🔵 io/questdb/cutlass/line/LineSenderException.java 8 9 88.89%
🔵 io/questdb/client/Sender.java 41 46 89.13%
🔵 io/questdb/cutlass/line/http/AbstractLineHttpSender.java 129 140 92.14%
🔵 io/questdb/std/IntList.java 4 4 100.00%
🔵 io/questdb/PropServerConfiguration.java 24 24 100.00%
🔵 io/questdb/cutlass/http/processors/LineHttpProcessorState.java 2 2 100.00%
🔵 io/questdb/cutlass/http/HttpResponseSink.java 1 1 100.00%
🔵 io/questdb/cutlass/line/http/LineHttpSenderV1.java 2 2 100.00%
🔵 io/questdb/cutlass/http/HttpServer.java 1 1 100.00%
🔵 io/questdb/cutlass/http/HttpServerConfigurationWrapper.java 1 1 100.00%
🔵 io/questdb/cutlass/line/http/LineHttpSenderV2.java 2 2 100.00%

@bluestreak01 bluestreak01 merged commit d004904 into master Sep 30, 2025
35 checks passed
@bluestreak01 bluestreak01 deleted the nw_ilp_multi_url branch September 30, 2025 16:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Friction ILP Issues or changes relating to Influx Line Protocol

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants