feat(core): Java ILP client nanos precision by glasstiger · Pull Request #6220 · questdb/questdb

glasstiger · 2025-10-02T23:54:51Z

Protocol v2:

If the timestamp passed to the client in nanos, it should send it as nanos on the wire.
Micros (and everything else) should be sent as micros.
This way if the user sends nanos for TIMESTAMP_NS and micros for TIMESTAMP columns, the data gets ingested without conversion and/or handling of ArithmeticExceptions.

Protocol v1:

Unchanged, always sends timestamp columns with micros precision, and sends nanos for the designated timestamp.
Data always gets stored with micros precision, unless the designated timestamp column is created manually with TIMESTAMP_NS type.

Server changes:

The unit used to send the data on the wire now drives the column type if a table or column is created via ILP.
If the data is sent in nanos, the column type will be TIMESTAMP_NS.
If the data is sent in micros, the column type will be TIMESTAMP.
If the client does not send designated timestamp on the row which creates the table or column, the server falls back to the precision param sent in the query string, or to the config property line.timestamp.default.column.type in case of TCP protocol, and those will determine the type of the designated timestamp.

Bugfixes:

Also have to check for long overflow when ingesting into TIMESTAMP_NS column. Currently if there is an overflow, we insert incorrect timestamps instead of reporting an error.
Also fixing the reset() method on the HTTP sender. It did not reset the state to EMPTY.

coderabbitai · 2025-10-02T23:54:57Z

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Walkthrough

Refactors timestamp emission in line protocol senders by centralizing formatting into putTimestamp(...) helpers, adds an overload to write Instant-based timestamp columns, introduces HTTP sender backoff configuration fields and a public isNotFound(...) utility, extends tests to cover cross-unit timestamp ingestion, and updates related binary test fixtures.

Changes

Cohort / File(s)	Summary
Timestamp handling refactor (senders) `core/src/main/java/io/questdb/cutlass/line/AbstractLineTcpSender.java`, `core/src/main/java/io/questdb/cutlass/line/http/AbstractLineHttpSender.java`	Replaced inline timestamp formatting with private putTimestamp(long, ChronoUnit) and putTimestamp(Instant) helpers; updated at(...) and timestampColumn(...) to use centralized helpers.
HTTP sender API and config `core/src/main/java/io/questdb/cutlass/line/http/AbstractLineHttpSender.java`	Added initialBackoffMillis and maxBackoffMillis fields; added public static isNotFound(DirectUtf8Sequence).
Tests for cross-unit timestamps `core/src/test/java/io/questdb/test/cutlass/http/line/LineHttpSenderTest.java`	Added tests for TIMESTAMP and TIMESTAMP_NS ingestion across units; added helper; updated imports and minor formatting.
Updated test fixtures `core/src/test/resources/io/questdb/test/cutlass/line/interop/client-protocol-version-2-test.json`	Replaced multiple base64Content blocks with new encoded payloads; statuses unchanged.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  actor Caller
  participant Sender as AbstractLine*Sender
  participant TS as putTimestamp(...)
  participant Buf as Output Buffer

  Caller->>Sender: at(value, unit) / at(Instant)
  Sender->>TS: putTimestamp(value, unit or Instant)
  TS->>Buf: write timestamp + unit suffix
  Buf-->>Sender: appended
  Sender-->>Caller: return (fluent)

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	You can run `@coderabbitai generate docstrings` to improve docstring coverage.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title Check	✅ Passed	The title succinctly highlights the introduction of nanosecond precision support in the Java ILP client, which is the primary focus of the pull request. It clearly indicates a new feature (“feat(core)”) and identifies the component affected (“Java ILP client”) along with the enhancement (“nanos precision”). This phrasing is concise, specific to the main change, and avoids unnecessary details about microsecond handling or overflow checks that are secondary. A teammate scanning the history will immediately understand the core improvement being delivered.
Description Check	✅ Passed	The pull request description clearly outlines the changes to timestamp handling—sending nanoseconds as nanoseconds and all other timestamps as microseconds—alongside related bugfixes, which directly corresponds to the code modifications in the diff around centralized timestamp helpers and overflow checks.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

bluestreak01 · 2025-10-03T13:03:02Z

@CodeRabbit review

coderabbitai · 2025-10-03T13:03:27Z

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

core/src/main/java/io/questdb/cutlass/line/AbstractLineTcpSender.java

core/src/main/java/io/questdb/cutlass/line/tcp/TableStructureAdapter.java

amunra · 2025-10-09T14:50:04Z

I've tested this PR against the upcoming c-questdb-client and py-questdb-client nanos changes and the server now behaves as expected.

…if no designated ts is sent by the client

core/src/main/java/io/questdb/cairo/MillsTimestampDriver.java

core/src/main/java/io/questdb/cairo/MicrosTimestampDriver.java

core/src/main/java/io/questdb/cutlass/line/http/AbstractLineHttpSender.java

core/src/main/java/io/questdb/std/datetime/microtime/Micros.java

core/src/main/java/io/questdb/PropServerConfiguration.java

core/src/test/java/io/questdb/test/cutlass/line/tcp/LineTcpInsertOtherTypesTest.java

bluestreak01

this PR needs a little cleanup

glasstiger · 2025-10-24T15:12:09Z

[PR Coverage check]

😍 pass : 229 / 242 (94.63%)

file detail

	path	covered line	new line	coverage
🔵	io/questdb/cutlass/http/DefaultHttpServerConfiguration.java	0	2	00.00%
🔵	io/questdb/cutlass/line/LineUdpSender.java	1	4	25.00%
🔵	io/questdb/cutlass/line/tcp/TableUpdateDetails.java	2	3	66.67%
🔵	io/questdb/cairo/TimestampDriver.java	8	10	80.00%
🔵	io/questdb/cutlass/line/http/AbstractLineHttpSender.java	10	11	90.91%
🔵	io/questdb/cutlass/line/LineTcpSenderV2.java	22	24	91.67%
🔵	io/questdb/cutlass/line/tcp/LineTcpParser.java	17	18	94.44%
🔵	io/questdb/cutlass/line/tcp/LineTcpMeasurementEvent.java	20	21	95.24%
🔵	io/questdb/cutlass/line/tcp/LineWalAppender.java	7	7	100.00%
🔵	io/questdb/PropServerConfiguration.java	5	5	100.00%
🔵	io/questdb/cutlass/line/tcp/DefaultColumnTypes.java	38	38	100.00%
🔵	io/questdb/cutlass/line/tcp/LineTcpReceiverConfigurationWrapper.java	1	1	100.00%
🔵	io/questdb/cutlass/line/LineUtils.java	3	3	100.00%
🔵	io/questdb/cutlass/line/http/LineHttpSenderV1.java	10	10	100.00%
🔵	io/questdb/std/datetime/microtime/Micros.java	3	3	100.00%
🔵	io/questdb/cutlass/line/AbstractLineSender.java	1	1	100.00%
🔵	io/questdb/cairo/MicrosTimestampDriver.java	13	13	100.00%
🔵	io/questdb/cairo/MillsTimestampDriver.java	10	10	100.00%
🔵	io/questdb/cutlass/line/http/LineHttpSenderV2.java	24	24	100.00%
🔵	io/questdb/cutlass/line/udp/LineUdpParserImpl.java	1	1	100.00%
🔵	io/questdb/cairo/NanosTimestampDriver.java	10	10	100.00%
🔵	io/questdb/cutlass/line/tcp/LineProtocolException.java	2	2	100.00%
🔵	io/questdb/cutlass/line/tcp/TableStructureAdapter.java	13	13	100.00%
🔵	io/questdb/cutlass/line/LineTcpSenderV1.java	8	8	100.00%

feat(core): client to send nanos as nanos, and everything else as micros

95a6fbc

Merge branch 'master' into ia_sender_nanos

36fffe1

bluestreak01 previously approved these changes Oct 3, 2025

View reviewed changes

glasstiger added the DO NOT MERGE These changes should not be merged to main branch label Oct 3, 2025

bluestreak01 self-requested a review October 3, 2025 17:27

error when conversion fails with long overflow

09cf47d

glasstiger dismissed bluestreak01’s stale review via 09cf47d October 3, 2025 20:42

glasstiger and others added 2 commits October 3, 2025 21:43

Merge branch 'master' into ia_sender_nanos

6009021

more tests

333444c

glasstiger removed the DO NOT MERGE These changes should not be merged to main branch label Oct 6, 2025

Merge branch 'master' into ia_sender_nanos

6746b5d

ideoma changed the title ~~feat(core): client to send nanos as nanos, and everything else as micros~~ refactor(core): client to send nanos as nanos, and everything else as micros Oct 6, 2025

ideoma self-requested a review October 6, 2025 13:22

ideoma reviewed Oct 6, 2025

View reviewed changes

core/src/main/java/io/questdb/cutlass/line/AbstractLineTcpSender.java Outdated Show resolved Hide resolved

glasstiger and others added 5 commits October 9, 2025 02:32

fix exception name

44d287a

remove warning

561c95a

deprecate 'line.timestamp.default.column.type'

31b0f44

create table/add column via ILP tests

30aca35

Merge branch 'master' into ia_sender_nanos

137ddf9

glasstiger changed the title ~~refactor(core): client to send nanos as nanos, and everything else as micros~~ refactor(core): client to send nanos as nanos and everything else as micros Oct 9, 2025

glasstiger changed the title ~~refactor(core): client to send nanos as nanos and everything else as micros~~ refactor(core): Java ILP nanos related changes Oct 9, 2025

ideoma reviewed Oct 9, 2025

View reviewed changes

core/src/main/java/io/questdb/cutlass/line/tcp/TableStructureAdapter.java Show resolved Hide resolved

glasstiger changed the title ~~refactor(core): Java ILP nanos related changes~~ feat(core): Java ILP nanos related changes Oct 9, 2025

amunra mentioned this pull request Oct 9, 2025

BREAKING CHANGE: designated and non-designated timestamp feature parity for micros and nanos questdb/c-questdb-client#121

Merged

glasstiger and others added 2 commits October 9, 2025 18:05

config 'line.timestamp.default.column.type' to determine column type …

bd5ca4a

…if no designated ts is sent by the client

Merge branch 'master' into ia_sender_nanos

9196ca4

glasstiger and others added 4 commits October 17, 2025 14:54

Merge remote-tracking branch 'origin/master' into ia_sender_nanos

aed265a

fix compile error

6cae595

Merge branch 'master' into ia_sender_nanos

b9006ec

Merge branch 'master' into ia_sender_nanos

642e76d

ideoma previously approved these changes Oct 22, 2025

View reviewed changes

glasstiger added DO NOT MERGE These changes should not be merged to main branch and removed DO NOT MERGE These changes should not be merged to main branch labels Oct 22, 2025