Skip to content

Commit 24c0486

Browse files
authored
Merge pull request feast-dev#16 from farfetch-external/algattik/spark-ingestion
Closes KE-609 - read data from Kafka Closes KE-636 - write data into ADLS Gen2 Closes KE-655 - write data into Redis Added a Spark ingestion job reading from Kafka into Delta Lake storage and Redis. Created an integration test running against local Kafka, Redis and Spark and ingesting 128 random features into Redis and 128 random features into Delta, with different data types, and checking the result. As Spark only runs on Java 8, the Integration test is skipped in the CI build (that runs under Java 11), but is run in the e2e tests, when the emulator container is built (including the ingestion jar). Delta tables are automatically created and partitioned by day. ```bash # run SparkIngestionTest.java $ ls /var/folders/67/59_hhx6d5lz0wg__x35g8wbw0000gn/T/junit9193296653579758585/bXlwcm9qZWN0/bXlwcm9qZWN0L2ZlYXR1cmVfc2V0X2Zvcl9kZWx0YQ==/event_timestamp_day=2020-06-08/part-00002-abd1fbe1-c773-4d83-972d-5193c75885e5.c000.snappy.parquet /var/folders/67/59_hhx6d5lz0wg__x35g8wbw0000gn/T/junit9193296653579758585/bXlwcm9qZWN0/bXlwcm9qZWN0L2ZlYXR1cmVfc2V0X2Zvcl9kZWx0YQ==/event_timestamp_day=2020-06-08/part-00004-440cca76-fdc1-437a-93f2-d6739296cfe4.c000.snappy.parquet /var/folders/67/59_hhx6d5lz0wg__x35g8wbw0000gn/T/junit9193296653579758585/bXlwcm9qZWN0/bXlwcm9qZWN0L2ZlYXR1cmVfc2V0X2Zvcl9kZWx0YQ==/event_timestamp_day=2020-06-08/part-00005-bb670850-3001-493b-9000-30b9b41f81e6.c000.snappy.parquet ... # bXlwcm9qZWN0 is base64 for "myproject" # bXlwcm9...9kZWx0YQ== is base64 for "myproject/feature_set_for_delta" ``` ```python >>> import pandas as pd >>> df=pd.read_parquet("/var/folders/67/59_hhx6d5lz0wg__x35g8wbw0000gn/T/junit9193296653579758585/bXlwcm9qZWN0/bXlwcm9qZWN0L2ZlYXR1cmVfc2V0X2Zvcl9kZWx0YQ==") >>> df.iloc[1] event_timestamp 2020-06-08 07:47:04.931000 created_timestamp 2020-06-08 07:47:43.915000 ingestion_id testjob entity_id_primary -2101962939 entity_id_secondary iAkKJCqcry6NTS4 f_BYTES b'Smts8audYXVOYTw' f_STRING gK1chlMFi2Btbdd f_INT32 -951335350 f_INT64 3659296699309908912 f_DOUBLE 0.697594 f_FLOAT 0.879062 f_BOOL True f_STRING_LIST [Xx92gwGy2PUQCLl] f_INT32_LIST [-1400465827] f_INT64_LIST [6251409099521094342] f_DOUBLE_LIST [0.11672805668860675] f_FLOAT_LIST [0.6432213] f_BOOL_LIST [False] event_timestamp_day 2020-06-08 Name: 1, dtype: object ``` The spark-ingestion uses several classes copied and adapted from feast-ingestion and feast-storage-connector-redis. To reduce merge conflicts downstream, I've kept those classes as close as possible to the original. When we approach PR submission into public Feast, we can work on creating shared projects for both ingestion modes. A few data types have been disabled in the tests as they give a difference when checking for equality, although based on manual inspection they seem ok. Need to debug later on.
2 parents 3e49cb4 + 802b24a commit 24c0486

19 files changed

Lines changed: 2423 additions & 9 deletions

File tree

‎Makefile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ test-java:
4747
mvn test
4848

4949
test-java-with-coverage:
50-
mvn test jacoco:report-aggregate
50+
mvn -B test jacoco:report-aggregate
5151

5252
build-java:
5353
mvn clean verify

‎infra/docker/databricks-emulator/Dockerfile‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,8 +17,9 @@ RUN wget -q https://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/spark-$
1717
RUN cd spark/jars && for jar in \
1818
org/apache/spark/spark-sql-kafka-0-10_2.11/2.4.5/spark-sql-kafka-0-10_2.11-2.4.5.jar \
1919
org/apache/kafka/kafka-clients/2.3.0/kafka-clients-2.3.0.jar \
20+
io/delta/delta-core_2.11/0.6.1/delta-core_2.11-0.6.1.jar \
2021
; do \
21-
wget https://repo1.maven.org/maven2/$jar; \
22+
wget -q https://repo1.maven.org/maven2/$jar; \
2223
done
2324

2425
RUN mv spark/conf/log4j.properties.template spark/conf/log4j.properties
@@ -47,7 +48,7 @@ COPY LICENSE .m[2] .m2/
4748
# the existing .m2 directory to $FEAST_REPO_ROOT/.m2
4849
#
4950
ENV MAVEN_OPTS="-Dmaven.repo.local=/build/.m2/repository -DdependencyLocationsEnabled=false"
50-
RUN mvn --also-make --projects spark/databricks-emulator -Drevision=$REVISION \
51+
RUN mvn --also-make --projects spark/databricks-emulator,spark/spark-ingestion-job -Drevision=$REVISION \
5152
--batch-mode clean package
5253

5354
# ============================================================
@@ -61,6 +62,7 @@ ENV SPARK_HOME /spark
6162

6263
COPY --from=builder /spark /spark
6364
COPY --from=builder /build/spark/databricks-emulator/target/databricks-emulator-$REVISION.jar /opt/databricks-emulator.jar
65+
COPY --from=builder /build/spark/spark-ingestion-job/target/spark-ingestion-job-$REVISION.jar /opt/spark-ingestion-job.jar
6466
CMD ["java",\
6567
"-Xms2048m",\
6668
"-Xmx2048m",\

‎infra/scripts/test-docker-compose-databricks.sh‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,3 +51,6 @@ ${PROJECT_ROOT_DIR}/infra/scripts/wait-for-it.sh ${JUPYTER_DOCKER_CONTAINER_IP_A
5151

5252
# Run e2e tests for Redis
5353
docker exec feast_jupyter_1 bash -c 'cd feast/tests/e2e/ && pytest -s basic-ingest-redis-serving.py --core_url core:6565 --serving_url=online-serving:6566 --databricks'
54+
55+
# Run ingestion tests for FF Data Science scenarios
56+
docker exec feast_jupyter_1 bash -c 'cd feast/tests/ds_scenarios/ && pytest -s test-ingest.py --core_url core:6565 --serving_url=online-serving:6566'

‎pom.xml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,7 @@
6565
<!-- Force log4j2 to 2.11+ to support objectMessageAsJsonObject -->
6666
<log4jVersion>2.12.1</log4jVersion>
6767
<spark.version>2.4.5</spark.version>
68+
<scala.version>2.11.12</scala.version>
6869
<scala.compat.version>2.11</scala.compat.version>
6970
</properties>
7071

‎protos/feast/core/Store.proto‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,8 @@ message Store {
9898
CASSANDRA = 3;
9999

100100
REDIS_CLUSTER = 4;
101+
102+
DELTA = 5;
101103
}
102104

103105
message RedisConfig {
@@ -130,6 +132,10 @@ message Store {
130132
int32 max_retries = 3;
131133
}
132134

135+
message DeltaConfig {
136+
string path = 1;
137+
}
138+
133139
message Subscription {
134140
// Name of project that the feature sets belongs to. This can be one of
135141
// - [project_name]
@@ -167,5 +173,6 @@ message Store {
167173
BigQueryConfig bigquery_config = 12;
168174
CassandraConfig cassandra_config = 13;
169175
RedisClusterConfig redis_cluster_config = 14;
176+
DeltaConfig delta_config = 15;
170177
}
171178
}

‎spark/databricks-emulator/pom.xml‎

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -103,11 +103,5 @@
103103
<artifactId>mockito-core</artifactId>
104104
<scope>test</scope>
105105
</dependency>
106-
107-
<dependency>
108-
<groupId>com.jayway.jsonpath</groupId>
109-
<artifactId>json-path-assert</artifactId>
110-
<scope>test</scope>
111-
</dependency>
112106
</dependencies>
113107
</project>

‎spark/spark-ingestion-job/pom.xml‎

Lines changed: 146 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,4 +15,150 @@
1515
<version>${revision}</version>
1616
<relativePath>../..</relativePath>
1717
</parent>
18+
19+
<build>
20+
<plugins>
21+
<plugin>
22+
<groupId>org.apache.maven.plugins</groupId>
23+
<artifactId>maven-compiler-plugin</artifactId>
24+
<configuration>
25+
<!-- Spark 2 only runs on Java 8 -->
26+
<release>8</release>
27+
</configuration>
28+
</plugin>
29+
<plugin>
30+
<groupId>org.apache.maven.plugins</groupId>
31+
<artifactId>maven-surefire-plugin</artifactId>
32+
<configuration>
33+
<argLine>-Xmx1024m -XX:MaxPermSize=256m</argLine>
34+
</configuration>
35+
</plugin>
36+
<plugin>
37+
<groupId>org.codehaus.mojo</groupId>
38+
<artifactId>exec-maven-plugin</artifactId>
39+
<executions>
40+
<execution>
41+
<goals>
42+
<goal>exec</goal>
43+
</goals>
44+
<configuration>
45+
<executable>spark-submit</executable>
46+
<arguments>
47+
<argument>--master</argument>
48+
<argument>local</argument>
49+
<argument>${project.build.directory}/${project.artifactId}-${project.version}.jar
50+
</argument>
51+
</arguments>
52+
</configuration>
53+
</execution>
54+
</executions>
55+
</plugin>
56+
<!-- Use the shade plugin to remove all the provided artifacts (such as spark itself) from the jar -->
57+
<plugin>
58+
<groupId>org.apache.maven.plugins</groupId>
59+
<artifactId>maven-shade-plugin</artifactId>
60+
<version>3.1.1</version>
61+
<configuration>
62+
<transformers>
63+
<transformer
64+
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
65+
<mainClass>feast.spark.ingestion.SparkIngestion</mainClass>
66+
</transformer>
67+
</transformers>
68+
<!-- Relocate packages that conflict with JARs distributed in Spark. -->
69+
<relocations>
70+
<relocation>
71+
<pattern>com.google.code.gson</pattern>
72+
<shadedPattern>vendor.feast.com.google.code.gson</shadedPattern>
73+
</relocation>
74+
<relocation>
75+
<pattern>com.google.guava</pattern>
76+
<shadedPattern>vendor.feast.com.google.guava</shadedPattern>
77+
</relocation>
78+
<relocation>
79+
<pattern>com.google.protobuf</pattern>
80+
<shadedPattern>vendor.feast.com.google.protobuf</shadedPattern>
81+
</relocation>
82+
</relocations>
83+
</configuration>
84+
<executions>
85+
<execution>
86+
<phase>package</phase>
87+
<goals>
88+
<goal>shade</goal>
89+
</goals>
90+
</execution>
91+
</executions>
92+
</plugin>
93+
</plugins>
94+
</build>
95+
96+
<dependencies>
97+
98+
<dependency>
99+
<groupId>dev.feast</groupId>
100+
<artifactId>datatypes-java</artifactId>
101+
<version>${project.version}</version>
102+
</dependency>
103+
104+
<dependency>
105+
<groupId>org.apache.spark</groupId>
106+
<artifactId>spark-sql_${scala.compat.version}</artifactId>
107+
<version>${spark.version}</version>
108+
<scope>provided</scope>
109+
</dependency>
110+
111+
<dependency>
112+
<groupId>org.apache.spark</groupId>
113+
<artifactId>spark-sql-kafka-0-10_${scala.compat.version}</artifactId>
114+
<version>${spark.version}</version>
115+
<scope>provided</scope>
116+
</dependency>
117+
118+
<dependency>
119+
<groupId>io.delta</groupId>
120+
<artifactId>delta-core_${scala.compat.version}</artifactId>
121+
<version>0.6.1</version>
122+
<scope>provided</scope>
123+
</dependency>
124+
125+
<!-- To run actual Redis for ingestion integration test -->
126+
<dependency>
127+
<groupId>com.github.kstyrc</groupId>
128+
<artifactId>embedded-redis</artifactId>
129+
<scope>test</scope>
130+
</dependency>
131+
132+
<!-- To run actual Kafka for ingestion integration test -->
133+
<dependency>
134+
<groupId>org.apache.kafka</groupId>
135+
<artifactId>kafka_${scala.compat.version}</artifactId>
136+
<scope>test</scope>
137+
</dependency>
138+
139+
<dependency>
140+
<groupId>org.apache.kafka</groupId>
141+
<artifactId>kafka-clients</artifactId>
142+
<version>1.0.2</version>
143+
<scope>test</scope>
144+
</dependency>
145+
146+
<dependency>
147+
<groupId>io.lettuce</groupId>
148+
<artifactId>lettuce-core</artifactId>
149+
</dependency>
150+
151+
<dependency>
152+
<groupId>org.hamcrest</groupId>
153+
<artifactId>hamcrest-library</artifactId>
154+
<scope>test</scope>
155+
</dependency>
156+
157+
<dependency>
158+
<groupId>junit</groupId>
159+
<artifactId>junit</artifactId>
160+
<scope>test</scope>
161+
</dependency>
162+
163+
</dependencies>
18164
</project>
Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
/*
2+
* SPDX-License-Identifier: Apache-2.0
3+
* Copyright 2018-2020 The Feast Authors
4+
*
5+
* Licensed under the Apache License, Version 2.0 (the "License");
6+
* you may not use this file except in compliance with the License.
7+
* You may obtain a copy of the License at
8+
*
9+
* https://www.apache.org/licenses/LICENSE-2.0
10+
*
11+
* Unless required by applicable law or agreed to in writing, software
12+
* distributed under the License is distributed on an "AS IS" BASIS,
13+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
* See the License for the specific language governing permissions and
15+
* limitations under the License.
16+
*/
17+
package feast.ingestion.transform.fn;
18+
19+
import feast.proto.types.FeatureRowProto.FeatureRow;
20+
import java.io.Serializable;
21+
22+
/**
23+
* * Class copied and adapted from feast-ingestion ProcessFeatureRowDoFn.
24+
*
25+
* <p>TODO move common ingestion code into a shared project.
26+
*/
27+
public class ProcessFeatureRowDoFn implements Serializable {
28+
29+
private String defaultProject;
30+
31+
public ProcessFeatureRowDoFn(String defaultProject) {
32+
this.defaultProject = defaultProject;
33+
}
34+
35+
public FeatureRow processElement(FeatureRow featureRow) {
36+
String featureSetId = stripVersion(featureRow.getFeatureSet());
37+
featureSetId = applyDefaultProject(featureSetId);
38+
featureRow = featureRow.toBuilder().setFeatureSet(featureSetId).build();
39+
return featureRow;
40+
}
41+
42+
// For backward compatibility. Will be deprecated eventually.
43+
private String stripVersion(String featureSetId) {
44+
String[] split = featureSetId.split(":");
45+
return split[0];
46+
}
47+
48+
private String applyDefaultProject(String featureSetId) {
49+
String[] split = featureSetId.split("/");
50+
if (split.length == 1) {
51+
return defaultProject + "/" + featureSetId;
52+
}
53+
return featureSetId;
54+
}
55+
}
Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
/*
2+
* SPDX-License-Identifier: Apache-2.0
3+
* Copyright 2018-2019 The Feast Authors
4+
*
5+
* Licensed under the Apache License, Version 2.0 (the "License");
6+
* you may not use this file except in compliance with the License.
7+
* You may obtain a copy of the License at
8+
*
9+
* https://www.apache.org/licenses/LICENSE-2.0
10+
*
11+
* Unless required by applicable law or agreed to in writing, software
12+
* distributed under the License is distributed on an "AS IS" BASIS,
13+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
* See the License for the specific language governing permissions and
15+
* limitations under the License.
16+
*/
17+
package feast.ingestion.utils;
18+
19+
import com.google.protobuf.InvalidProtocolBufferException;
20+
import com.google.protobuf.util.JsonFormat;
21+
import feast.proto.core.FeatureSetProto.FeatureSet;
22+
import feast.proto.core.FeatureSetProto.FeatureSetSpec;
23+
import feast.proto.core.StoreProto.Store;
24+
import feast.proto.core.StoreProto.Store.Subscription;
25+
import java.util.ArrayList;
26+
import java.util.List;
27+
import java.util.regex.Pattern;
28+
29+
/**
30+
* * Class copied and adapted from feast-ingestion SpecUtil.
31+
*
32+
* <p>TODO move common ingestion code into a shared project.
33+
*/
34+
public class SpecUtil {
35+
36+
public static String getFeatureSetReference(FeatureSetSpec featureSetSpec) {
37+
return String.format("%s/%s", featureSetSpec.getProject(), featureSetSpec.getName());
38+
}
39+
40+
/** Get only feature set specs that matches the subscription */
41+
public static List<FeatureSet> getSubscribedFeatureSets(
42+
List<Subscription> subscriptions, List<FeatureSet> featureSets) {
43+
List<FeatureSet> subscribed = new ArrayList<>();
44+
for (FeatureSet featureSet : featureSets) {
45+
for (Subscription sub : subscriptions) {
46+
// If configuration missing, fail
47+
if (sub.getProject().isEmpty() || sub.getName().isEmpty()) {
48+
throw new IllegalArgumentException(
49+
String.format("Subscription is missing arguments: %s", sub.toString()));
50+
}
51+
52+
// If all wildcards, subscribe to everything
53+
if (sub.getProject().equals("*") || sub.getName().equals("*")) {
54+
subscribed.add(featureSet);
55+
break;
56+
}
57+
58+
// Match project name
59+
if (!featureSet.getSpec().getProject().equals(sub.getProject())) {
60+
continue;
61+
}
62+
63+
// Convert wildcard to regex
64+
String subName = sub.getName();
65+
if (!sub.getName().contains(".*")) {
66+
subName = subName.replace("*", ".*");
67+
}
68+
69+
// Match feature set name to pattern
70+
Pattern pattern = Pattern.compile(subName);
71+
if (!pattern.matcher(featureSet.getSpec().getName()).matches()) {
72+
continue;
73+
}
74+
subscribed.add(featureSet);
75+
}
76+
}
77+
return subscribed;
78+
}
79+
80+
public static List<FeatureSet> parseFeatureSetSpecJsonList(List<String> jsonList)
81+
throws InvalidProtocolBufferException {
82+
List<FeatureSet> featureSets = new ArrayList<>();
83+
for (String json : jsonList) {
84+
FeatureSetSpec.Builder builder = FeatureSetSpec.newBuilder();
85+
JsonFormat.parser().merge(json, builder);
86+
featureSets.add(FeatureSet.newBuilder().setSpec(builder.build()).build());
87+
}
88+
return featureSets;
89+
}
90+
91+
public static List<Store> parseStoreJsonList(List<String> jsonList)
92+
throws InvalidProtocolBufferException {
93+
List<Store> stores = new ArrayList<>();
94+
for (String json : jsonList) {
95+
Store.Builder builder = Store.newBuilder();
96+
JsonFormat.parser().merge(json, builder);
97+
stores.add(builder.build());
98+
}
99+
return stores;
100+
}
101+
}

0 commit comments

Comments
 (0)