Skip to content

fix: correct millis-to-micros conversion for Avro timestamp-millis in Document conversion - #122

Merged
orfeon merged 1 commit into
mercari:mainfrom
kratos0718:fix/avro-timestamp-millis-to-document
Aug 16, 2026
Merged

orfeon merged 1 commit into
mercari:mainfrom
kratos0718:fix/avro-timestamp-millis-to-document

Conversation

@kratos0718

Copy link
Copy Markdown
Contributor

Follow-up to #119, in the Avro → Firestore/Datastore Document path.

Problem

DateTimeUtil.toProtoTimestamp(Long) interprets its argument as epoch microseconds:

public static com.google.protobuf.Timestamp toProtoTimestamp(final Long epocMicros) {
    long second = epocMicros / 1000_000L;
    long nano   = epocMicros % 1000_000L * 1000L;

AvroToDocumentConverter passes it epoch seconds on the timestamp-millis branch:

if(LogicalTypes.timestampMillis().equals(schema.getLogicalType())) {
    yield ...setTimestampValue(DateTimeUtil.toProtoTimestamp(l/1000L)).build();   // l is epoch millis
} else if(LogicalTypes.timestampMicros().equals(schema.getLogicalType())) {
    yield ...setTimestampValue(DateTimeUtil.toProtoTimestamp(l)).build();         // correct
}

The value ends up wrong by a factor of 10^6. 2026-08-07T10:15:30.123Z (epoch millis 1786097730123) becomes 1786 seconds — 1970-01-01T00:29:46Z. Every timestamp-millis field written through this path collapses to just after the epoch, silently.

The adjacent timestamp-micros branch passing l straight through confirms microseconds is the expected unit. Of the call sites of toProtoTimestamp outside DateTimeUtil, this is the only one that divides.

The path is reachable from MElement and ElementToDocumentConverter, and timestamp-millis is a first-class logical type here (REQUIRED_LOGICAL_TIMESTAMP_MILLI_TYPE / NULLABLE_LOGICAL_TIMESTAMP_MILLI_TYPE).

Fix

l/1000L → l * 1000L, converting millis to the micros the callee expects.

Tests

Adds AvroToDocumentConverterTest with three cases:

test before after
testTimestampMillisField FAIL — expected:<1786097730> but was:<1786> pass
testTimestampMillisAndMicrosAgree FAIL — expected seconds 1786097730, was 1786 pass
testTimestampMicrosField pass pass

testTimestampMicrosField passes both before and after by design — it guards the already-correct branch against over-correction.

before fix:  Tests run: 3, Failures: 2
after fix:   Tests run: 3, Failures: 0

Note

Building from a clean clone needed org.apache.kafka:kafka-clients added locally — Beam ships it at provided scope so it is not pulled transitively by beam-sdks-java-io-kafka (same thing reported in the second half of #108). That is unrelated to this change and is not included here; I only added it locally to run the tests. Happy to open a separate PR if useful.

… Document conversion

DateTimeUtil.toProtoTimestamp(Long) expects epoch microseconds, but the
timestamp-millis branch of AvroToDocumentConverter divided the epoch
millis value by 1000, passing epoch seconds instead. The resulting
Firestore/Datastore timestamp was off by a factor of 10^6, collapsing
every timestamp-millis field to a value just after the epoch.

The adjacent timestamp-micros branch already passes its value through
unchanged, confirming microseconds is the expected unit.

Adds regression tests covering the timestamp-millis path, the
timestamp-micros path, and agreement between the two encodings of the
same instant.

@orfeon orfeon left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your contribution!

@orfeon
orfeon merged commit b3fcd9b into mercari:main Aug 16, 2026
1 check passed
kratos0718 added a commit to kratos0718/portfolio that referenced this pull request Sep 20, 2026
Merged-PR stat was showing 17 while the actual count is 22 - the
OpenSource.tsx list was missing mercari/pipeline#122, apache/maven-
help-plugin#424, grokability/snipe-it#19514, phasespace-labs/
palinode#132, and the newly-merged apache/maven-shared-jar#165.
Added all five with real quotes/stars pulled from each PR, and
updated the Landing and OpenSource stat badges and bio/tagline
text in portfolioData.ts to match.
kratos0718 added a commit to kratos0718/portfolio that referenced this pull request Nov 12, 2026
Merged-PR stat was showing 17 while the actual count is 22 - the
OpenSource.tsx list was missing mercari/pipeline#122, apache/maven-
help-plugin#424, grokability/snipe-it#19514, phasespace-labs/
palinode#132, and the newly-merged apache/maven-shared-jar#165.
Added all five with real quotes/stars pulled from each PR, and
updated the Landing and OpenSource stat badges and bio/tagline
text in portfolioData.ts to match.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants