Skip to content

Commit 1ae84df

Browse files
algattikalgattik
authored
Merge pull request feast-dev#12 from farfetch-external/algattik/merge_2020-06-01
Rebase to Feast v0.5.0
2 parents a7621a5 + 14e6e83 commit 1ae84df

374 files changed

Lines changed: 14593 additions & 9474 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.dockerignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,3 @@
11
docs
2+
!docs/coverage
23
charts

‎.github/CODEOWNERS‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
# Each line is a file pattern followed by one or more owners.
22
# https://help.github.com/en/articles/about-code-owners
33

4-
* @zhilingc @pradithya @woop @davidheryanto @khorshuheng
5-
/core/ @zhilingc @pradithya
6-
/ingestion/ @zhilingc @pradithya
7-
/serving/ @zhilingc @pradithya
8-
/cli/ @zhilingc @pradithya
4+
* @zhilingc @woop @davidheryanto @khorshuheng @pyalex
5+
/core/ @zhilingc
6+
/ingestion/ @zhilingc
7+
/serving/ @zhilingc
8+
/cli/ @zhilingc

‎.github/pull_request_template.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
<!-- Thanks for sending a pull request! Here are some tips for you:
22
3-
1. Ensure that your code follows our code conventions: https://github.com/gojek/feast/blob/master/docs/contributing.md#code-conventions
4-
2. Run unit tests and ensure that they are passing: https://github.com/gojek/feast/blob/master/docs/contributing.md#running-unit-tests
5-
3. If your change introduces any API changes, make sure to update the integration tests scripts here: https://github.com/gojek/feast/tree/master/tests/e2e
3+
1. Ensure that your code follows our code conventions: https://github.com/feast-dev/feast/blob/master/docs/contributing.md#code-conventions
4+
2. Run unit tests and ensure that they are passing: https://github.com/feast-dev/feast/blob/master/docs/contributing.md#running-unit-tests
5+
3. If your change introduces any API changes, make sure to update the integration tests scripts here: https://github.com/feast-dev/feast/tree/master/tests/e2e
66
4. Make sure documentation is updated for your PR!
77
5. Make sure you have signed the CLA https://cla.developers.google.com/clas
88

.github/workflows/integration-tests.yml renamed to .github/workflows/docker_build_test_push.yml

Lines changed: 7 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -25,67 +25,39 @@ jobs:
2525
# Checkout sources from Git
2626
- uses: actions/checkout@v2
2727

28-
# Build base CI image, with dependencies such as Go and Python.
29-
# This image seldom changes, so thanks to layer caching this usually completes in seconds.
30-
- name: Build CI image
31-
uses: docker/build-push-action@v1
32-
with:
33-
username: ${{ secrets.CONTAINERREGISTRY_USERNAME }}
34-
password: ${{ secrets.CONTAINERREGISTRY_PASSWORD }}
35-
repository: ${{ secrets.CONTAINERREGISTRY_IMAGENAMEBASE }}-ci
36-
registry: ${{ secrets.CONTAINERREGISTRY_URL }}
37-
dockerfile: infra/docker/ci/Dockerfile
38-
tags: v${{ github.sha }}
39-
4028
# Cache the Maven repository across runs.
4129
- name: Cache Maven repository
4230
id: cache-maven
4331
uses: actions/cache@v1
4432
with:
4533
path: .m2
46-
key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
34+
key: ${{ runner.os }}-mavenr-${{ hashFiles('**/pom.xml') }}
4735
restore-keys: |
48-
${{ runner.os }}-maven-
36+
${{ runner.os }}-mavenr-
4937
5038
# Generate the Maven cache if needed by running a throwaway build of Feast.
5139
# --user: run as the host user, to avoid writing .m2 files to the host as root.
5240
# -e MAVEN_CONFIG: See https://github.com/carlossg/docker-maven#running-as-non-root
5341
# -v $PWD:/build: mount the current directory to write .m2 files onto the host.
5442
- name: Generate Maven cache
5543
if: steps.cache-maven.outputs.cache-hit != 'true'
56-
run: docker run --rm --user $(id -u):$(id -g) -e MAVEN_CONFIG=/build/.m2 -v $PWD:/build $IMAGE bash -c "cd /build && mvn -Dmaven.repo.local=/build/.m2/repository -Dgpg.skip=true -B clean verify -DskipTests"
57-
env:
58-
IMAGE: ${{ secrets.CONTAINERREGISTRY_URL }}/${{ secrets.CONTAINERREGISTRY_IMAGENAMEBASE }}-ci:v${{ github.sha }}
44+
run: docker run --rm --user $(id -u):$(id -g) -e MAVEN_CONFIG=/build/.m2 -v $PWD:/build maven:3.6-jdk-11 bash -c "cp -R /build /tmp && mvn -f /tmp/build -Dmaven.repo.local=/build/.m2/repository -Dgpg.skip=true -B clean verify -DskipTests"
5945

6046
# Build images in parallel.
61-
- name: Build Feast runtime and end-to-end test images
62-
run: |
63-
set -x
64-
docker-compose -f infra/docker-compose/docker-compose.yml build --parallel end-to-end-tests core online-serving
47+
- name: Build Feast runtime images
48+
run: docker-compose -f infra/docker-compose/docker-compose.yml -f infra/docker-compose/docker-compose.online.yml build --parallel core online-serving
6549
env:
6650
FEAST_VERSION: v${{ github.sha }}
67-
FEAST_CI_IMAGE: ${{ secrets.CONTAINERREGISTRY_URL }}/${{ secrets.CONTAINERREGISTRY_IMAGENAMEBASE }}-ci
6851
FEAST_CORE_IMAGE: ${{ secrets.CONTAINERREGISTRY_URL }}/${{ secrets.CONTAINERREGISTRY_IMAGENAMEBASE }}-core
6952
FEAST_SERVING_IMAGE: ${{ secrets.CONTAINERREGISTRY_URL }}/${{ secrets.CONTAINERREGISTRY_IMAGENAMEBASE }}-serving
7053

71-
# "docker-compose up" starts docker containers.
72-
# "--abort-on-container-exit" ensures that the containers are shutdown when end-to-end
73-
# tests complete, and that the task exit status indicates failure if there are test failures.
74-
# Images fetched from a repository, such as Kafka, will not be built, so this
75-
# step will only build the Feast Core image.
76-
- name: Run Feast end-to-end test with Docker compose
77-
run: |
78-
set -x
79-
docker-compose -f infra/docker-compose/docker-compose.yml up --abort-on-container-exit end-to-end-tests
54+
- name: test docker compose
55+
run: ./infra/scripts/test-docker-compose.sh
8056
env:
8157
COMPOSE_PROJECT_NAME: feast
8258
FEAST_VERSION: v${{ github.sha }}
83-
FEAST_CI_IMAGE: ${{ secrets.CONTAINERREGISTRY_URL }}/${{ secrets.CONTAINERREGISTRY_IMAGENAMEBASE }}-ci
8459
FEAST_CORE_IMAGE: ${{ secrets.CONTAINERREGISTRY_URL }}/${{ secrets.CONTAINERREGISTRY_IMAGENAMEBASE }}-core
8560
FEAST_SERVING_IMAGE: ${{ secrets.CONTAINERREGISTRY_URL }}/${{ secrets.CONTAINERREGISTRY_IMAGENAMEBASE }}-serving
86-
FEAST_CORE_CONFIG: direct-runner.yml
87-
FEAST_ONLINE_SERVING_CONFIG: online-serving.yml
88-
FEAST_ONLINE_STORE_CONFIG: redis-store.yml
8961

9062
# Login to Azure Container Registry.
9163
- name: Login to Azure Container Registry

‎.github/workflows/docker_compose_tests.yml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
name: docker compose tests
2+
3+
on:
4+
push:
5+
branches:
6+
- master
7+
8+
jobs:
9+
basic-redis-e2e-tests-docker-compose:
10+
runs-on: ubuntu-latest
11+
name: basic redis e2e tests on docker compose
12+
steps:
13+
- uses: actions/checkout@v2
14+
- name: test docker compose
15+
run: ./infra/scripts/test-docker-compose.sh

.github/workflows/unit-tests.yml renamed to .github/workflows/unit_tests.yml

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -8,22 +8,26 @@ jobs:
88
container: gcr.io/kf-feast/feast-ci:latest
99
name: unit test java
1010
steps:
11-
- uses: actions/checkout@v1
11+
- uses: actions/checkout@v2
1212
- uses: actions/cache@v1
1313
with:
1414
path: ~/.m2/repository
1515
key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
1616
restore-keys: |
1717
${{ runner.os }}-maven-
1818
- name: test java
19-
run: make test-java
19+
run: make test-java-with-coverage
20+
- uses: actions/upload-artifact@v2
21+
with:
22+
name: java-coverage-report
23+
path: ${{ github.workspace }}/docs/coverage/java/target/site/jacoco-aggregate/
2024

2125
unit-test-python:
2226
runs-on: ubuntu-latest
2327
container: gcr.io/kf-feast/feast-ci:latest
2428
name: unit test python
2529
steps:
26-
- uses: actions/checkout@v1
30+
- uses: actions/checkout@v2
2731
- name: install python
2832
run: make install-python
2933
- name: test python
@@ -34,7 +38,7 @@ jobs:
3438
container: gcr.io/kf-feast/feast-ci:latest
3539
name: unit test go
3640
steps:
37-
- uses: actions/checkout@v1
41+
- uses: actions/checkout@v2
3842
- name: install dependencies
3943
run: make compile-protos-go
4044
- name: test go

‎.gitignore‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -179,11 +179,12 @@ dmypy.json
179179
.flattened-pom.xml
180180

181181
sdk/python/docs/html
182-
sdk/python/feast/core/
183-
sdk/python/feast/serving/
184-
sdk/python/feast/storage/
185-
sdk/python/feast/types/
186-
sdk/python/tensorflow_metadata
182+
183+
# Generated python code
184+
*_pb2.py
185+
*_pb2.pyi
186+
*_pb2_grpc.py
187+
187188

188189
### Farfetch ###
189190

‎.helmdocsignore‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
infra/charts/feast/charts/postgresql
2+
infra/charts/feast/charts/kafka
3+
infra/charts/feast/charts/redis
4+
infra/charts/feast/charts/prometheus-statsd-exporter
5+
infra/charts/feast/charts/prometheus
6+
infra/charts/feast/charts/grafana

‎.prow/config.yaml‎

Lines changed: 114 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ deck:
4141
tide:
4242
queries:
4343
- repos:
44-
- gojek/feast
44+
- feast-dev/feast
4545
labels:
4646
- lgtm
4747
- approved
@@ -51,16 +51,17 @@ tide:
5151
- do-not-merge/invalid-owners-file
5252
- do-not-merge/work-in-progress
5353
- needs-rebase
54+
- needs-kind
5455
merge_method:
55-
gojek/feast: squash
56+
feast-dev/feast: squash
5657
blocker_label: merge-blocker
5758
squash_label: tide/squash
5859

5960
# presubmits and postsubmits configure ProwJobs:
6061
# https://github.com/kubernetes/test-infra/blob/6571843b1aa7bd6cf577a7a8b9e9971241f424d5/prow/jobs.md
6162

6263
presubmits:
63-
gojek/feast:
64+
feast-dev/feast:
6465
- name: test-core-and-ingestion
6566
decorate: true
6667
spec:
@@ -142,6 +143,7 @@ presubmits:
142143

143144
- name: test-end-to-end
144145
decorate: true
146+
always_run: true
145147
spec:
146148
containers:
147149
- image: maven:3.6-jdk-11
@@ -153,6 +155,20 @@ presubmits:
153155
skip_branches:
154156
- ^v0\.(3|4)-branch$
155157

158+
- name: test-end-to-end-redis-cluster
159+
decorate: true
160+
always_run: true
161+
spec:
162+
containers:
163+
- image: maven:3.6-jdk-11
164+
command: ["infra/scripts/test-end-to-end-redis-cluster.sh"]
165+
resources:
166+
requests:
167+
cpu: "6"
168+
memory: "6144Mi"
169+
skip_branches:
170+
- ^v0\.(3|4)-branch$
171+
156172
- name: test-end-to-end-java-8
157173
decorate: true
158174
always_run: true
@@ -169,6 +185,7 @@ presubmits:
169185

170186
- name: test-end-to-end-batch
171187
decorate: true
188+
always_run: true
172189
spec:
173190
volumes:
174191
- name: service-account
@@ -187,6 +204,27 @@ presubmits:
187204
skip_branches:
188205
- ^v0\.(3|4)-branch$
189206

207+
- name: test-end-to-end-batch-fs-update
208+
decorate: true
209+
always_run: false
210+
spec:
211+
volumes:
212+
- name: service-account
213+
secret:
214+
secretName: feast-service-account
215+
containers:
216+
- image: maven:3.6-jdk-11
217+
command: ["infra/scripts/test-end-to-end-batch.sh", "-m", "fs_update"]
218+
resources:
219+
requests:
220+
cpu: "6"
221+
memory: "6144Mi"
222+
volumeMounts:
223+
- name: service-account
224+
mountPath: "/etc/service-account"
225+
skip_branches:
226+
- ^v0\.(3|4)-branch$
227+
190228
- name: test-end-to-end-batch-java-8
191229
decorate: true
192230
always_run: true
@@ -208,8 +246,79 @@ presubmits:
208246
branches:
209247
- ^v0\.(3|4)-branch$
210248

249+
- name: test-end-to-end-batch-dataflow
250+
decorate: true
251+
always_run: false
252+
spec:
253+
volumes:
254+
- name: service-account-df
255+
secret:
256+
secretName: feast-e2e-service-account
257+
- name: docker-socket
258+
hostPath:
259+
path: /var/run/docker.sock
260+
containers:
261+
- image: google/cloud-sdk:273.0.0
262+
command: ["infra/scripts/test-end-to-end-batch-dataflow.sh"]
263+
resources:
264+
requests:
265+
cpu: "6"
266+
memory: "6144Mi"
267+
volumeMounts:
268+
- name: service-account-df
269+
mountPath: "/etc/service-account-df"
270+
- name: docker-socket
271+
mountPath: /var/run/docker.sock
272+
securityContext:
273+
privileged: true
274+
skip_branches:
275+
- ^v0\.(3|4)-branch$
276+
277+
- name: publish-docker-images
278+
decorate: true
279+
always_run: true
280+
spec:
281+
containers:
282+
- image: google/cloud-sdk:273.0.0
283+
command:
284+
- bash
285+
- -c
286+
- |
287+
infra/scripts/download-maven-cache.sh \
288+
--archive-uri gs://feast-templocation-kf-feast/.m2.2019-10-24.tar \
289+
--output-dir $PWD/
290+
291+
infra/scripts/publish-docker-image.sh \
292+
--repository gcr.io/kf-feast/feast-core \
293+
--tag ${PULL_PULL_SHA:1} \
294+
--file infra/docker/core/Dockerfile \
295+
--google-service-account-file /etc/gcloud/service-account.json
296+
297+
infra/scripts/publish-docker-image.sh \
298+
--repository gcr.io/kf-feast/feast-serving \
299+
--tag ${PULL_PULL_SHA:1} \
300+
--file infra/docker/serving/Dockerfile \
301+
--google-service-account-file /etc/gcloud/service-account.json
302+
303+
volumeMounts:
304+
- name: docker-socket
305+
mountPath: /var/run/docker.sock
306+
- name: service-account
307+
mountPath: /etc/gcloud/service-account.json
308+
subPath: service-account.json
309+
readOnly: true
310+
securityContext:
311+
privileged: true
312+
volumes:
313+
- name: docker-socket
314+
hostPath:
315+
path: /var/run/docker.sock
316+
- name: service-account
317+
secret:
318+
secretName: feast-service-account
319+
211320
postsubmits:
212-
gojek/feast:
321+
feast-dev/feast:
213322
- name: publish-python-sdk
214323
decorate: true
215324
spec:
@@ -219,7 +328,7 @@ postsubmits:
219328
- sh
220329
- -c
221330
- |
222-
infra/scripts/publish-python-sdk.sh \
331+
make compile-protos-python && infra/scripts/publish-python-sdk.sh \
223332
--directory-path sdk/python --repository pypi
224333
volumeMounts:
225334
- name: pypirc

‎.prow/plugins.yaml‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
plugins:
2-
gojek/feast:
2+
feast-dev/feast:
33
- approve
44
- assign
55
- help
@@ -12,14 +12,22 @@ plugins:
1212
- wip
1313
- trigger
1414
- config-updater
15+
- require-matching-label
1516

1617
config_updater:
1718
maps:
1819
.prow/config.yaml:
1920
name: config
2021

2122
external_plugins:
22-
gojek/feast:
23+
feast-dev/feast:
2324
- name: needs-rebase
2425
events:
2526
- pull_request
27+
28+
require_matching_label:
29+
- missing_label: needs-kind
30+
org: feast-dev
31+
repo: feast
32+
prs: true
33+
regexp: ^kind/

0 commit comments

Comments
 (0)