Skip to content

Commit 2bb12b2

Browse files
xinyiZzzYour Name
authored andcommitted
[chore](arrow-flight-sql) Add Arrow Flight Sql demo for Java (#45306)
### What problem does this PR solve? # How to use: 1. mvn clean install -U 2. mvn package 3. java --add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED -cp java-0.1.jar doris.arrowflight.demo.Main "sql" "fe_ip" "fe_arrow_flight_port" "fe_query_port" # What can this demo do: This is a java demo for doris arrow flight sql, you can use this to test various connection methods for sending queries to the doris arrow flight server, help you understand how to use arrow flight sql and test performance. You should install maven prior to run this demo. # Performance test Section 6.2 of #25514 is the performance test results of the Doris Arrow Flight SQL using java. # Output ``` WARNING: Unknown module: org.apache.arrow.memory.core specified to --add-opens ************************************* | FlightAdbcDriver | ************************************* FlightAdbcDriver > loadArrowBatch SLF4J(W): No SLF4J providers were found. SLF4J(W): Defaulting to no-operation (NOP) logger implementation SLF4J(W): See https://www.slf4j.org/codes.html#noProviders for further details. > Schema<l_shipdate: Date(DAY) not null> > 1994-08-17, > batchCount: 25, rowCount: 100000 > cost: 1704 ms. FlightAdbcDriver > loadArrowBatchToString > Schema<l_shipdate: Date(DAY) not null> > 1992-01-02, > batchCount: 25, rowCount: 100000 > cost: 1692 ms. ************************************* | FlightJdbcDriver | ************************************* FlightJdbcDriver > loadArrowBatch > Schema<l_shipdate: Date(DAY) not null> > 1997-01-30, > batchCount: 98, rowCount: 100000 > cost: 1840 ms. FlightJdbcDriver > loadArrowBatchToString > Schema<l_shipdate: Date(DAY) not null> > 1992-01-02, > batchCount: 98, rowCount: 100000 > cost: 1712 ms. ************************************* | JdbcDriverManager | ************************************* JdbcDriverManager > jdbc:mysql > loadJdbcResult > rowCount: 100000, columnCount: 1 > cost: 11431 ms. JdbcDriverManager > jdbc:mysql > loadJdbcResultToString > 1992-01-02, > rowCount: 100000, columnCount: 1 resultSize: 100000 > cost: 5164 ms. JdbcDriverManager > jdbc:arrow-flight-sql > loadJdbcResultToString > rowCount: 100000, columnCount: 1 > cost: 1736 ms. JdbcDriverManager > jdbc:arrow-flight-sql > loadJdbcResultToString > 1997-01-29, > rowCount: 100000, columnCount: 1 resultSize: 100000 > cost: 2442 ms. ************************************* | FlightSqlClient | ************************************* FlightSqlClient > getFlightInfoFromDorisFe > Schema<l_shipdate: Date(DAY) not null> > 1994-08-15, > batchCount: 25, rowCount: 100000 FlightSqlClient > constructDummyFlightInfo, don't be afraid! expected to get error `INVALID_ARGUMENT: Malformed ticket` org.apache.arrow.flight.FlightRuntimeException: INVALID_ARGUMENT: Malformed ticket, size: 1 at org.apache.arrow.flight.CallStatus.toRuntimeException(CallStatus.java:121) at org.apache.arrow.flight.grpc.StatusUtils.fromGrpcRuntimeException(StatusUtils.java:161) at org.apache.arrow.flight.grpc.StatusUtils.fromThrowable(StatusUtils.java:182) at org.apache.arrow.flight.FlightStream$Observer.onError(FlightStream.java:489) at org.apache.arrow.flight.FlightClient$1.onError(FlightClient.java:371) at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:481) at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) at org.apache.arrow.flight.grpc.ClientInterceptorAdapter$FlightClientCallListener.onClose(ClientInterceptorAdapter.java:118) at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:564) at io.grpc.internal.ClientCallImpl.access$100(ClientCallImpl.java:72) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:729) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:710) at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:842) Process finished with exit code 0 ```
1 parent 40fb69f commit 2bb12b2

11 files changed

Lines changed: 911 additions & 0 deletions

File tree

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
<!--
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
10+
http://www.apache.org/licenses/LICENSE-2.0
11+
12+
Unless required by applicable law or agreed to in writing,
13+
software distributed under the License is distributed on an
14+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
KIND, either express or implied. See the License for the
16+
specific language governing permissions and limitations
17+
under the License.
18+
-->
19+
20+
# How to use:
21+
22+
1. mvn clean install -U
23+
2. mvn package
24+
3. java --add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED -cp java-0.1.jar doris.arrowflight.demo.Main "sql" "fe_ip" "fe_arrow_flight_port" "fe_query_port"
25+
26+
# What can this demo do:
27+
28+
This is a java demo for doris arrow flight sql, you can use this to test various connection
29+
methods for sending queries to the doris arrow flight server, help you understand how to use arrow flight sql
30+
and test performance. You should install maven prior to run this demo.
31+
32+
# Performance test
33+
34+
Section 6.2 of https://github.com/apache/doris/issues/25514 is the performance test
35+
results of the doris arrow flight sql using java.
Lines changed: 186 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,186 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<!--
3+
Licensed to the Apache Software Foundation (ASF) under one
4+
or more contributor license agreements. See the NOTICE file
5+
distributed with this work for additional information
6+
regarding copyright ownership. The ASF licenses this file
7+
to you under the Apache License, Version 2.0 (the
8+
"License"); you may not use this file except in compliance
9+
with the License. You may obtain a copy of the License at
10+
11+
http://www.apache.org/licenses/LICENSE-2.0
12+
13+
Unless required by applicable law or agreed to in writing,
14+
software distributed under the License is distributed on an
15+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
16+
KIND, either express or implied. See the License for the
17+
specific language governing permissions and limitations
18+
under the License.
19+
-->
20+
<project xmlns="http://maven.apache.org/POM/4.0.0"
21+
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
22+
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
23+
<modelVersion>4.0.0</modelVersion>
24+
25+
<groupId>doris.arrowflight.demo</groupId>
26+
<artifactId>java</artifactId>
27+
<version>0.1</version>
28+
29+
<properties>
30+
<maven.compiler.source>17</maven.compiler.source>
31+
<maven.compiler.target>17</maven.compiler.target>
32+
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
33+
<!--
34+
Notice !!! Arrow and ADBC versions cannot be combined arbitrarily.
35+
36+
1. For ADBC 0.15.0, please use Arrow 18.0.0 and after, not compatible with previous versions of Arrow.
37+
2. Try not to use Arrow 17.0.0, may get the following error:
38+
```
39+
Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 2 out of bounds for length 2
40+
at com.google.protobuf.GeneratedMessageV3$FieldAccessorTable.ensureFieldAccessorsInitialized(GeneratedMessageV3.java:2094)
41+
at org.apache.arrow.flight.sql.impl.FlightSql$ActionCreatePreparedStatementRequest.internalGetFieldAccessorTable(FlightSql.java:16332)
42+
at com.google.protobuf.GeneratedMessageV3.getDescriptorForType(GeneratedMessageV3.java:139)
43+
at com.google.protobuf.Any.pack(Any.java:61)
44+
at org.apache.arrow.flight.sql.FlightSqlClient.prepare(FlightSqlClient.java:767)
45+
at org.apache.arrow.flight.sql.FlightSqlClient.prepare(FlightSqlClient.java:746)
46+
at org.apache.arrow.driver.jdbc.client.ArrowFlightSqlClientHandler.prepare(ArrowFlightSqlClientHandler.java:310)
47+
```
48+
similar issue: https://github.com/protocolbuffers/protobuf/issues/15762
49+
3. A more stable version is Arrow 15.0.2 and ADBC 0.12.0, but we always hope to embrace the future with new versions!
50+
-->
51+
<arrow.version>18.1.0</arrow.version>
52+
<adbc.version>0.15.0</adbc.version>
53+
<log4j.version>2.17.1</log4j.version>
54+
</properties>
55+
<dependencies>
56+
<!-- If Maven cannot find the Arrow or Adbc version of the dependency, update the Maven central repository.
57+
Arrow and Adbc are updated frequently, and we always try to use the latest version as much as possible. -->
58+
<dependency>
59+
<groupId>org.apache.arrow.adbc</groupId>
60+
<artifactId>adbc-driver-jdbc</artifactId>
61+
<version>${adbc.version}</version>
62+
</dependency>
63+
<dependency>
64+
<groupId>org.apache.arrow.adbc</groupId>
65+
<artifactId>adbc-driver-flight-sql</artifactId>
66+
<version>${adbc.version}</version>
67+
</dependency>
68+
<dependency>
69+
<groupId>mysql</groupId>
70+
<artifactId>mysql-connector-java</artifactId>
71+
<version>8.0.33</version>
72+
</dependency>
73+
<dependency>
74+
<groupId>org.apache.arrow</groupId>
75+
<artifactId>arrow-memory-core</artifactId>
76+
<version>${arrow.version}</version>
77+
</dependency>
78+
<dependency>
79+
<groupId>org.apache.arrow</groupId>
80+
<artifactId>arrow-memory-netty</artifactId>
81+
<version>${arrow.version}</version>
82+
</dependency>
83+
<dependency>
84+
<groupId>org.apache.arrow</groupId>
85+
<artifactId>arrow-vector</artifactId>
86+
<version>${arrow.version}</version>
87+
</dependency>
88+
<dependency>
89+
<groupId>org.apache.arrow</groupId>
90+
<artifactId>arrow-vector</artifactId>
91+
<version>${arrow.version}</version>
92+
</dependency>
93+
<dependency>
94+
<groupId>org.apache.arrow</groupId>
95+
<artifactId>flight-core</artifactId>
96+
<version>${arrow.version}</version>
97+
</dependency>
98+
<dependency>
99+
<groupId>org.apache.arrow</groupId>
100+
<artifactId>flight-sql</artifactId>
101+
<version>${arrow.version}</version>
102+
</dependency>
103+
104+
<dependency>
105+
<groupId>org.apache.arrow.adbc</groupId>
106+
<artifactId>adbc-core</artifactId>
107+
<version>${adbc.version}</version>
108+
</dependency>
109+
<dependency>
110+
<groupId>org.apache.arrow.adbc</groupId>
111+
<artifactId>adbc-driver-manager</artifactId>
112+
<version>${adbc.version}</version>
113+
</dependency>
114+
<dependency>
115+
<groupId>org.apache.arrow.adbc</groupId>
116+
<artifactId>adbc-sql</artifactId>
117+
<version>${adbc.version}</version>
118+
</dependency>
119+
<dependency>
120+
<groupId>org.apache.arrow</groupId>
121+
<artifactId>flight-sql-jdbc-core</artifactId>
122+
<version>${arrow.version}</version>
123+
</dependency>
124+
<dependency>
125+
<groupId>junit</groupId>
126+
<artifactId>junit</artifactId>
127+
<version>4.13.1</version>
128+
<scope>test</scope>
129+
</dependency>
130+
<dependency>
131+
<groupId>org.junit.jupiter</groupId>
132+
<artifactId>junit-jupiter</artifactId>
133+
<version>RELEASE</version>
134+
<scope>test</scope>
135+
</dependency>
136+
</dependencies>
137+
138+
<build>
139+
<plugins>
140+
<!-- Java Compiler -->
141+
<plugin>
142+
<groupId>org.apache.maven.plugins</groupId>
143+
<artifactId>maven-compiler-plugin</artifactId>
144+
<version>3.10.1</version>
145+
<configuration>
146+
<source>${maven.compiler.source}</source>
147+
<target>${maven.compiler.target}</target>
148+
<encoding>${project.build.sourceEncoding}</encoding>
149+
</configuration>
150+
</plugin>
151+
<plugin>
152+
<groupId>org.apache.maven.plugins</groupId>
153+
<artifactId>maven-shade-plugin</artifactId>
154+
<version>3.4.1</version>
155+
<executions>
156+
<execution>
157+
<phase>package</phase>
158+
<goals>
159+
<goal>shade</goal>
160+
</goals>
161+
<configuration>
162+
<filters>
163+
<filter>
164+
<artifact>*:*</artifact>
165+
<excludes>
166+
<exclude>META-INF/*.SF</exclude>
167+
<exclude>META-INF/*.DSA</exclude>
168+
<exclude>META-INF/*.RSA</exclude>
169+
</excludes>
170+
</filter>
171+
</filters>
172+
<transformers>
173+
<transformer
174+
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
175+
</transformer>
176+
<transformer
177+
implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
178+
</transformers>
179+
</configuration>
180+
</execution>
181+
</executions>
182+
</plugin>
183+
</plugins>
184+
</build>
185+
186+
</project>
Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
package doris.arrowflight.demo;
19+
20+
import org.apache.arrow.vector.ValueVector;
21+
import org.apache.arrow.vector.VectorSchemaRoot;
22+
import org.apache.arrow.vector.ipc.ArrowReader;
23+
import org.apache.arrow.vector.types.pojo.Field;
24+
25+
import java.io.IOException;
26+
import java.time.LocalDate;
27+
import java.time.format.DateTimeFormatter;
28+
import java.util.ArrayList;
29+
import java.util.List;
30+
31+
32+
/**
33+
* Iterate over each batch in ArrowReader.
34+
* ArrowReader is the iterator returned by ADBC Client when executing a query.
35+
*/
36+
public class ArrowBatchReader {
37+
38+
@FunctionalInterface
39+
public interface LoadArrowBatchFunc {
40+
void load(ArrowReader reader) throws IOException;
41+
}
42+
43+
/**
44+
* Print one row in VectorSchemaRoot, if the output format is incorrect, may need to modify
45+
* the output method of different types of ValueVector.
46+
*/
47+
public static void printRow(VectorSchemaRoot root, int rowIndex) {
48+
if (root == null || rowIndex < 0 || rowIndex >= root.getRowCount()) {
49+
System.out.println("Invalid row index: " + rowIndex);
50+
return;
51+
}
52+
53+
System.out.print("> ");
54+
for (Field field : root.getSchema().getFields()) {
55+
ValueVector vector = root.getVector(field.getName());
56+
if (vector != null) {
57+
if (vector instanceof org.apache.arrow.vector.DateDayVector) {
58+
DateTimeFormatter formatter = DateTimeFormatter.ofPattern("yyyy-MM-dd");
59+
int dayOffset = ((org.apache.arrow.vector.DateDayVector) vector).get(rowIndex);
60+
LocalDate date = LocalDate.ofEpochDay(dayOffset);
61+
System.out.print(date.format(formatter));
62+
} else if (vector instanceof org.apache.arrow.vector.BitVector) {
63+
System.out.print(((org.apache.arrow.vector.BitVector) vector).get(rowIndex) == 1);
64+
} else {
65+
// other types field
66+
System.out.print(vector.getObject(rowIndex).toString());
67+
}
68+
System.out.print(", ");
69+
}
70+
}
71+
System.out.println();
72+
}
73+
74+
/**
75+
* Iterate over each batch in ArrowReader with the least cost, only record the number of rows and batches,
76+
* usually used to test performance.
77+
*/
78+
public static LoadArrowBatchFunc loadArrowBatch = reader -> {
79+
int rowCount = 0;
80+
int batchCount = 0;
81+
while (reader.loadNextBatch()) {
82+
VectorSchemaRoot root = reader.getVectorSchemaRoot();
83+
if (batchCount == 0) {
84+
System.out.println("> " + root.getSchema().toString());
85+
printRow(root, 1); // only print first line
86+
}
87+
rowCount += root.getRowCount();
88+
batchCount += 1;
89+
}
90+
System.out.println("> batchCount: " + batchCount + ", rowCount: " + rowCount);
91+
};
92+
93+
/**
94+
* Iterate over each batch in ArrowReader and convert the batch to String, this will take more time.
95+
*/
96+
public static LoadArrowBatchFunc loadArrowBatchToString = reader -> {
97+
int rowCount = 0;
98+
List<String> result = new ArrayList<>();
99+
while (reader.loadNextBatch()) {
100+
VectorSchemaRoot root = reader.getVectorSchemaRoot();
101+
if (result.size() == 0) {
102+
System.out.println("> " + root.getSchema().toString());
103+
printRow(root, 0); // only print first line
104+
}
105+
result.add(root.contentToTSVString());
106+
rowCount += root.getRowCount();
107+
}
108+
System.out.println("> batchCount: " + result.size() + ", rowCount: " + rowCount);
109+
};
110+
}
Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
package doris.arrowflight.demo;
19+
20+
public class Configuration {
21+
public String sql = ""; // require
22+
public String ip = "127.0.0.1"; // require
23+
public String arrowFlightPort = "9090"; // require
24+
public String mysqlPort = "9030";
25+
public int retryTimes = 2; // The first execution is cold run
26+
public String user = "root";
27+
public String password = "";
28+
29+
Configuration(String[] args) {
30+
for (int i = 0; i < args.length; i++) {
31+
switch (i) {
32+
case 0 -> sql = args[i];
33+
case 1 -> ip = args[i];
34+
case 2 -> arrowFlightPort = args[i];
35+
case 3 -> mysqlPort = args[i];
36+
case 4 -> retryTimes = Integer.parseInt(args[i]);
37+
case 5 -> user = args[i];
38+
case 6 -> password = args[i];
39+
}
40+
}
41+
}
42+
}

0 commit comments

Comments
 (0)