Skip to content

Commit 417bc6c

Browse files
GaoleMenggcf-owl-bot[bot]
GaoleMeng
and
gcf-owl-bot[bot]
authored
fix: add precision overwritten to 9 digit if the passed in JSON type is FLOAT or DOUBLE (#1932)
* feat: Split writer into connection worker and wrapper, this is a prerequisite for multiplexing client * feat: add connection worker pool skeleton, used for multiplexing client * feat: add Load api for connection worker for multiplexing client * feat: add multiplexing support to connection worker. We will treat every new stream name as a switch of destinationt * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat: port the multiplexing client core algorithm and basic tests also fixed a tiny bug inside fake bigquery write impl for getting thre response from offset * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat: wire multiplexing connection pool to stream writer * feat: some fixes for multiplexing client * feat: fix some todos, and reject the mixed behavior of passed in client or not * feat: fix the bug that we may peek into the write_stream field but it's possible the proto schema does not contain this field * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat: fix the bug that we may peek into the write_stream field but it's possible the proto schema does not contain this field * feat: add getInflightWaitSeconds implementation * feat: Add schema comparision in connection loop to ensure schema update for the same stream name can be notified * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat: add schema update support to multiplexing * fix: fix windows build bug: windows Instant resolution is different with linux * fix: fix another failing tests for windows build * fix: fix another test failure for Windows build * feat: Change new thread for each retry to be a thread pool to avoid create/tear down too much threads if lots of retries happens * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * fix: add back the background executor provider that's accidentally removed * feat: throw error when use connection pool for explicit stream * fix: Add precision truncation to the passed in value from JSON float and double type. * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * modify the bom version * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
1 parent 2d648cf commit 417bc6c

3 files changed

Lines changed: 70 additions & 5 deletions

File tree

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,7 @@ If you are using Maven without BOM, add this to your dependencies:
4949
If you are using Gradle 5.x or later, add this to your dependencies:
5050

5151
```Groovy
52-
implementation platform('com.google.cloud:libraries-bom:26.3.0')
52+
implementation platform('com.google.cloud:libraries-bom:26.4.0')
5353
5454
implementation 'com.google.cloud:google-cloud-bigquerystorage'
5555
```

‎google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/JsonToProtoMessage.java‎

Lines changed: 14 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@
2828
import com.google.protobuf.Message;
2929
import com.google.protobuf.UninitializedMessageException;
3030
import java.math.BigDecimal;
31+
import java.math.RoundingMode;
3132
import java.time.LocalDate;
3233
import java.util.List;
3334
import java.util.logging.Logger;
@@ -49,6 +50,7 @@
4950
*/
5051
public class JsonToProtoMessage {
5152
private static final Logger LOG = Logger.getLogger(JsonToProtoMessage.class.getName());
53+
private static int NUMERIC_SCALE = 9;
5254
private static ImmutableMap<FieldDescriptor.Type, String> FieldTypeToDebugMessage =
5355
new ImmutableMap.Builder<FieldDescriptor.Type, String>()
5456
.put(FieldDescriptor.Type.BOOL, "boolean")
@@ -315,10 +317,15 @@ private static void fillField(
315317
new BigDecimal(((Number) val).longValue())));
316318
return;
317319
} else if (val instanceof Float || val instanceof Double) {
320+
// In JSON, the precision passed in is machine dependent. We should round the number
321+
// before passing to backend.
322+
BigDecimal bigDecimal = new BigDecimal(String.valueOf(val));
323+
if (bigDecimal.scale() > 9) {
324+
bigDecimal = bigDecimal.setScale(NUMERIC_SCALE, RoundingMode.HALF_UP);
325+
}
318326
protoMsg.setField(
319327
fieldDescriptor,
320-
BigDecimalByteStringEncoder.encodeToNumericByteString(
321-
new BigDecimal(String.valueOf(val))));
328+
BigDecimalByteStringEncoder.encodeToNumericByteString(bigDecimal));
322329
return;
323330
} else if (val instanceof BigDecimal) {
324331
protoMsg.setField(
@@ -559,10 +566,13 @@ private static void fillRepeatedField(
559566
new BigDecimal(((Number) val).longValue())));
560567
added = true;
561568
} else if (val instanceof Float || val instanceof Double) {
569+
BigDecimal bigDecimal = new BigDecimal(String.valueOf(val));
570+
if (bigDecimal.scale() > 9) {
571+
bigDecimal = bigDecimal.setScale(NUMERIC_SCALE, RoundingMode.HALF_UP);
572+
}
562573
protoMsg.addRepeatedField(
563574
fieldDescriptor,
564-
BigDecimalByteStringEncoder.encodeToNumericByteString(
565-
new BigDecimal(String.valueOf(val))));
575+
BigDecimalByteStringEncoder.encodeToNumericByteString(bigDecimal));
566576
added = true;
567577
} else if (val instanceof BigDecimal) {
568578
protoMsg.addRepeatedField(

‎google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1/JsonToProtoMessageTest.java‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020

2121
import com.google.cloud.bigquery.storage.test.JsonTest.*;
2222
import com.google.cloud.bigquery.storage.test.SchemaTest.*;
23+
import com.google.common.collect.ImmutableList;
2324
import com.google.common.collect.ImmutableMap;
2425
import com.google.protobuf.ByteString;
2526
import com.google.protobuf.Descriptors.Descriptor;
@@ -685,6 +686,60 @@ public void testDouble() throws Exception {
685686
assertEquals(expectedProto, protoMsg);
686687
}
687688

689+
@Test
690+
public void testDoubleHighPrecision() throws Exception {
691+
TableSchema tableSchema =
692+
TableSchema.newBuilder()
693+
.addFields(
694+
TableFieldSchema.newBuilder()
695+
.setName("numeric")
696+
.setType(TableFieldSchema.Type.NUMERIC)
697+
.build())
698+
.build();
699+
TestNumeric expectedProto =
700+
TestNumeric.newBuilder()
701+
.setNumeric(
702+
BigDecimalByteStringEncoder.encodeToNumericByteString(
703+
new BigDecimal("3.400500513")))
704+
.build();
705+
JSONObject json = new JSONObject();
706+
json.put("numeric", 3.400500512978076);
707+
DynamicMessage protoMsg =
708+
JsonToProtoMessage.convertJsonToProtoMessage(
709+
TestNumeric.getDescriptor(), tableSchema, json);
710+
assertEquals(expectedProto, protoMsg);
711+
}
712+
713+
@Test
714+
public void testDoubleHighPrecision_RepeatedField() throws Exception {
715+
TableSchema tableSchema =
716+
TableSchema.newBuilder()
717+
.addFields(
718+
0,
719+
TableFieldSchema.newBuilder()
720+
.setName("bignumeric")
721+
.setType(TableFieldSchema.Type.NUMERIC)
722+
.setMode(TableFieldSchema.Mode.REPEATED)
723+
.build())
724+
.build();
725+
TestBignumeric expectedProto =
726+
TestBignumeric.newBuilder()
727+
.addBignumeric(
728+
BigDecimalByteStringEncoder.encodeToNumericByteString(
729+
new BigDecimal("3.400500513")))
730+
.addBignumeric(
731+
BigDecimalByteStringEncoder.encodeToNumericByteString(new BigDecimal("0.1")))
732+
.addBignumeric(
733+
BigDecimalByteStringEncoder.encodeToNumericByteString(new BigDecimal("0.12")))
734+
.build();
735+
JSONObject json = new JSONObject();
736+
json.put("bignumeric", ImmutableList.of(3.400500512978076, 0.10000000000055, 0.12));
737+
DynamicMessage protoMsg =
738+
JsonToProtoMessage.convertJsonToProtoMessage(
739+
TestBignumeric.getDescriptor(), tableSchema, json);
740+
assertEquals(expectedProto, protoMsg);
741+
}
742+
688743
@Test
689744
public void testTimestamp() throws Exception {
690745
TableSchema tableSchema =

0 commit comments

Comments
 (0)