Skip to content

Commit e4507ac

Browse files
kevjumbakevjumba
authored
fix: Remove hard-coded integration test setup for AWS & GCP (#2970)
* Fix Signed-off-by: Kevin Zhang <[email protected]> * Fix lint Signed-off-by: Kevin Zhang <[email protected]> * Fix Signed-off-by: Kevin Zhang <[email protected]> * see if fix works Signed-off-by: Kevin Zhang <[email protected]> * Fix lint Signed-off-by: Kevin Zhang <[email protected]> * Fix Signed-off-by: Kevin Zhang <[email protected]> * Fix Signed-off-by: Kevin Zhang <[email protected]> * Fix lint Signed-off-by: Kevin Zhang <[email protected]>
1 parent 5ae2a34 commit e4507ac

11 files changed

Lines changed: 111 additions & 40 deletions

File tree

‎.github/workflows/pr_integration_tests.yml‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -175,8 +175,6 @@ jobs:
175175
if: ${{ always() }} # this will guarantee that step won't be canceled and resources won't leak
176176
env:
177177
FEAST_SERVER_DOCKER_IMAGE_TAG: ${{ needs.build-docker-image.outputs.DOCKER_IMAGE_TAG }}
178-
FEAST_USAGE: "False"
179-
IS_TEST: "True"
180178
SNOWFLAKE_CI_DEPLOYMENT: ${{ secrets.SNOWFLAKE_CI_DEPLOYMENT }}
181179
SNOWFLAKE_CI_USER: ${{ secrets.SNOWFLAKE_CI_USER }}
182180
SNOWFLAKE_CI_PASSWORD: ${{ secrets.SNOWFLAKE_CI_PASSWORD }}

‎.github/workflows/unit_tests.yml‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -70,13 +70,12 @@ jobs:
7070
run: make install-python-ci-dependencies
7171
- name: Test Python
7272
env:
73-
IS_TEST: "True"
7473
SNOWFLAKE_CI_DEPLOYMENT: ${{ secrets.SNOWFLAKE_CI_DEPLOYMENT }}
7574
SNOWFLAKE_CI_USER: ${{ secrets.SNOWFLAKE_CI_USER }}
7675
SNOWFLAKE_CI_PASSWORD: ${{ secrets.SNOWFLAKE_CI_PASSWORD }}
7776
SNOWFLAKE_CI_ROLE: ${{ secrets.SNOWFLAKE_CI_ROLE }}
7877
SNOWFLAKE_CI_WAREHOUSE: ${{ secrets.SNOWFLAKE_CI_WAREHOUSE }}
79-
run: FEAST_USAGE=False pytest -n 8 --cov=./ --cov-report=xml --color=yes sdk/python/tests
78+
run: pytest -n 8 --cov=./ --cov-report=xml --color=yes sdk/python/tests
8079
- name: Upload coverage to Codecov
8180
uses: codecov/codecov-action@v1
8281
with:

‎CONTRIBUTING.md‎

Lines changed: 53 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -139,7 +139,7 @@ There are two sets of tests you can run:
139139
#### Local integration tests
140140
For this approach of running tests, you'll need to have docker set up locally: [Get Docker](https://docs.docker.com/get-docker/)
141141

142-
It leverages a file based offline store to test against emulated versions of Datastore, DynamoDB, and Redis, using ephemeral containers.
142+
It leverages a file based offline store to test against emulated versions of Datastore, DynamoDB, and Redis, using ephemeral containers.
143143

144144
These tests create new temporary tables / datasets locally only, and they are cleaned up. when the containers are torn down.
145145

@@ -161,17 +161,48 @@ To test across clouds, on top of setting up Redis, you also need GCP / AWS / Sno
161161
gcloud auth login
162162
gcloud auth application-default login
163163
```
164-
3. Export `GCLOUD_PROJECT=[your project]` to your .zshrc
164+
- When you run `gcloud auth application-default login`, you should see some output of the form:
165+
```
166+
Credentials saved to file: [$HOME/.config/gcloud/application_default_credentials.json]
167+
```
168+
- You should run `export GOOGLE_APPLICATION_CREDENTIALS="$HOME/.config/gcloud/application_default_credentials.json”` to add the application credentials to your .zshrc or .bashrc.
169+
3. Run `export GCLOUD_PROJECT=[your project]` to your .zshrc or .bashrc.
170+
4. Running `gcloud config list` should give you something like this:
171+
```sh
172+
$ gcloud config list
173+
[core]
174+
account = [your email]
175+
disable_usage_reporting = True
176+
project = [your project]
177+
178+
Your active configuration is: [default]
179+
```
180+
5. Export gcp specific environment variables. Namely,
181+
```sh
182+
export GCS_REGION='[your gcs region e.g US]'
183+
export GCS_STAGING_LOCATION='[your gcs staging location]'
184+
```
165185

166186
**AWS**
167187
1. TODO(adchia): flesh out setting up AWS login (or create helper script)
168-
2. Modify `RedshiftDataSourceCreator` to use your credentials
188+
2. To run the AWS Redshift and Dynamo integration tests you will have to export your own AWS credentials. Namely,
189+
190+
```sh
191+
export AWS_REGION='[your aws region]'
192+
export AWS_CLUSTER_ID='[your aws cluster id]'
193+
export AWS_USER='[your aws user]'
194+
export AWS_DB='[your aws database]'
195+
export AWS_STAGING_LOCATION='[your s3 staging location uri]'
196+
export AWS_IAM_ROLE='[redshift and s3 access role]'
197+
export AWS_LAMBDA_ROLE='[your aws lambda execution role]'
198+
export AWS_REGISTRY_PATH='[your aws registry path]'
199+
```
169200

170201
**Snowflake**
171-
1. See https://signup.snowflake.com/ to setup a trial.
202+
1. See https://signup.snowflake.com/ to setup a trial.
172203
2. Then to run successfully, you'll need some environment variables setup:
173204
```sh
174-
export SNOWFLAKE_CI_DEPLOYMENT='[snowflake_deployment]'
205+
export SNOWFLAKE_CI_DEPLOYMENT='[snowflake_deployment]'
175206
export SNOWFLAKE_CI_USER='[your user]'
176207
export SNOWFLAKE_CI_PASSWORD='[your pw]'
177208
export SNOWFLAKE_CI_ROLE='[your CI role e.g. SYSADMIN]'
@@ -180,12 +211,28 @@ export SNOWFLAKE_CI_WAREHOUSE='[your warehouse]'
180211

181212
Then run `make test-python-integration`. Note that for Snowflake / GCP / AWS, this will create new temporary tables / datasets.
182213

214+
#### Running specific provider tests or running your test against specific online or offline stores
215+
216+
1. If you don't need to have your test run against all of the providers(`gcp`, `aws`, and `snowflake`) or don't need to run against all of the online stores, you can tag your test with specific providers or stores that you need(`@pytest.mark.universal_online_stores` or `@pytest.mark.universal_online_stores` with the `only` parameter). The `only` parameter selects specific offline providers and online stores that your test will test against. Example:
217+
218+
```python
219+
# Only parametrizes this test with the sqlite online store
220+
@pytest.mark.universal_online_stores(only=["sqlite"])
221+
def test_feature_get_online_features_types_match():
222+
```
223+
224+
2. You can also filter tests to run by using pytest's cli filtering. Instead of using the make commands to test Feast, you can filter tests by name with the `-k` parameter. The parametrized integration tests are all uniquely identified by their provider and online store so the `-k` option can select only the tests that you need to run. For example, to run only Redshift related tests, you can use the following command:
225+
226+
```sh
227+
python -m pytest -n 8 --integration -k Redshift sdk/python/tests
228+
```
229+
183230
#### (Experimental) Run full integration tests against containerized services
184231
Test across clouds requires existing accounts on GCP / AWS / Snowflake, and may incur costs when using these services.
185232

186233
For this approach of running tests, you'll need to have docker set up locally: [Get Docker](https://docs.docker.com/get-docker/)
187234

188-
It's possible to run some integration tests against emulated local versions of these services, using ephemeral containers.
235+
It's possible to run some integration tests against emulated local versions of these services, using ephemeral containers.
189236
These tests create new temporary tables / datasets locally only, and they are cleaned up. when the containers are torn down.
190237

191238
The services with containerized replacements currently implemented are:

‎README.md‎

Lines changed: 5 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -135,11 +135,10 @@ pprint(feature_vector)
135135

136136
## 📦 Functionality and Roadmap
137137

138-
The list below contains the functionality that contributors are planning to develop for Feast
138+
The list below contains the functionality that contributors are planning to develop for Feast.
139139

140-
* Items below that are in development (or planned for development) will be indicated in parentheses.
141140
* We welcome contribution to all items in the roadmap!
142-
* Have questions about the roadmap? Go to the Slack channel to ask on #feast-development
141+
* Have questions about the roadmap? Go to the Slack channel to ask on #feast-development.
143142

144143
* **Data Sources**
145144
* [x] [Snowflake source](https://docs.feast.dev/reference/data-sources/snowflake)
@@ -185,9 +184,8 @@ The list below contains the functionality that contributors are planning to deve
185184
* [x] Kubernetes (See [guide](https://docs.feast.dev/how-to-guides/running-feast-in-production#4.3.-java-based-feature-server-deployed-on-kubernetes))
186185
* **Feature Serving**
187186
* [x] Python Client
188-
* [x] REST Feature Server (Python) (See [RFC](https://docs.google.com/document/d/1iXvFhAsJ5jgAhPOpTdB3j-Wj1S9x3Ev\_Wr6ZpnLzER4/edit))
189-
* [x] REST / gRPC Feature Server (Go) (Alpha release. See [docs](https://docs.feast.dev/reference/feature-servers/go-feature-retrieval)
190-
* [x] gRPC Feature Server (Java) (Alpha release. See [#1497](https://github.com/feast-dev/feast/issues/1497))
187+
* [x] [Python feature server](https://docs.feast.dev/reference/feature-servers/python-feature-server)
188+
* [x] [Go feature server](https://docs.feast.dev/reference/feature-servers/go-feature-server)
191189
* **Data Quality Management (See [RFC](https://docs.google.com/document/d/110F72d4NTv80p35wDSONxhhPBqWRwbZXG4f9mNEMd98/edit))**
192190
* [x] Data profiling and validation (Great Expectations)
193191
* **Feature Discovery and Governance**
@@ -196,7 +194,7 @@ The list below contains the functionality that contributors are planning to deve
196194
* [x] Model-centric feature tracking (feature services)
197195
* [x] Amundsen integration (see [Feast extractor](https://github.com/amundsen-io/amundsen/blob/main/databuilder/databuilder/extractor/feast_extractor.py))
198196
* [x] DataHub integration (see [DataHub Feast docs](https://datahubproject.io/docs/generated/ingestion/sources/feast/))
199-
* [x] Feast Web UI (Alpha release. See [documentation](https://docs.feast.dev/reference/alpha-web-ui))
197+
* [x] Feast Web UI (Alpha release. See [docs](https://docs.feast.dev/reference/alpha-web-ui))
200198

201199
## 🎓 Important Resources
202200

‎sdk/python/tests/conftest.py‎

Lines changed: 9 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@
1313
# limitations under the License.
1414
import logging
1515
import multiprocessing
16+
import os
1617
import socket
1718
from contextlib import closing
1819
from datetime import datetime, timedelta
@@ -24,13 +25,15 @@
2425
import pytest
2526
from _pytest.nodes import Item
2627

27-
from feast import FeatureStore
28-
from feast.wait import wait_retry_backoff
29-
from tests.data.data_creator import create_basic_driver_dataset
30-
from tests.integration.feature_repos.integration_test_repo_config import (
28+
os.environ["FEAST_USAGE"] = "False"
29+
os.environ["IS_TEST"] = "True"
30+
from feast import FeatureStore # noqa: E402
31+
from feast.wait import wait_retry_backoff # noqa: E402
32+
from tests.data.data_creator import create_basic_driver_dataset # noqa: E402
33+
from tests.integration.feature_repos.integration_test_repo_config import ( # noqa: E402
3134
IntegrationTestRepoConfig,
3235
)
33-
from tests.integration.feature_repos.repo_configuration import (
36+
from tests.integration.feature_repos.repo_configuration import ( # noqa: E402
3437
AVAILABLE_OFFLINE_STORES,
3538
AVAILABLE_ONLINE_STORES,
3639
OFFLINE_STORE_TO_PROVIDER_CONFIG,
@@ -39,7 +42,7 @@
3942
construct_test_environment,
4043
construct_universal_test_data,
4144
)
42-
from tests.integration.feature_repos.universal.data_sources.file import (
45+
from tests.integration.feature_repos.universal.data_sources.file import ( # noqa: E402
4346
FileDataSourceCreator,
4447
)
4548

‎sdk/python/tests/integration/feature_repos/repo_configuration.py‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -65,8 +65,6 @@
6565
)
6666

6767
DYNAMO_CONFIG = {"type": "dynamodb", "region": "us-west-2"}
68-
# Port 12345 will chosen as default for redis node configuration because Redis Cluster is started off of nodes
69-
# 6379 -> 6384. This causes conflicts in cli integration tests so we manually keep them separate.
7068
REDIS_CONFIG = {"type": "redis", "connection_string": "localhost:6379,db=0"}
7169
REDIS_CLUSTER_CONFIG = {
7270
"type": "redis",
@@ -390,7 +388,10 @@ def construct_test_environment(
390388

391389
feature_server = AwsLambdaFeatureServerConfig(
392390
enabled=True,
393-
execution_role_name="arn:aws:iam::402087665549:role/lambda_execution_role",
391+
execution_role_name=os.getenv(
392+
"AWS_LAMBDA_ROLE",
393+
"arn:aws:iam::402087665549:role/lambda_execution_role",
394+
),
394395
)
395396

396397
else:
@@ -402,9 +403,12 @@ def construct_test_environment(
402403
if (
403404
test_repo_config.python_feature_server and test_repo_config.provider == "aws"
404405
) or test_repo_config.registry_location == RegistryLocation.S3:
406+
aws_registry_path = os.getenv(
407+
"AWS_REGISTRY_PATH", "s3://feast-integration-tests/registries"
408+
)
405409
registry: Union[
406410
str, RegistryConfig
407-
] = f"s3://feast-integration-tests/registries/{project}/registry.db"
411+
] = f"{aws_registry_path}/{project}/registry.db"
408412
else:
409413
registry = RegistryConfig(
410414
path=str(Path(repo_dir_name) / "registry.db"),

‎sdk/python/tests/integration/feature_repos/universal/data_sources/bigquery.py‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
import os
12
import uuid
23
from typing import Dict, List, Optional
34

@@ -53,7 +54,10 @@ def teardown(self):
5354

5455
def create_offline_store_config(self):
5556
return BigQueryOfflineStoreConfig(
56-
location="US", gcs_staging_location="gs://feast-export/"
57+
location=os.getenv("GCS_REGION", "US"),
58+
gcs_staging_location=os.getenv(
59+
"GCS_STAGING_LOCATION", "gs://feast-export/"
60+
),
5761
)
5862

5963
def create_data_source(

‎sdk/python/tests/integration/feature_repos/universal/data_sources/redshift.py‎

Lines changed: 16 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
import os
12
import uuid
23
from typing import Dict, List, Optional
34

@@ -24,16 +25,23 @@ class RedshiftDataSourceCreator(DataSourceCreator):
2425

2526
def __init__(self, project_name: str, *args, **kwargs):
2627
super().__init__(project_name)
27-
self.client = aws_utils.get_redshift_data_client("us-west-2")
28-
self.s3 = aws_utils.get_s3_resource("us-west-2")
28+
self.client = aws_utils.get_redshift_data_client(
29+
os.getenv("AWS_REGION", "us-west-2")
30+
)
31+
self.s3 = aws_utils.get_s3_resource(os.getenv("AWS_REGION", "us-west-2"))
2932

3033
self.offline_store_config = RedshiftOfflineStoreConfig(
31-
cluster_id="feast-integration-tests",
32-
region="us-west-2",
33-
user="admin",
34-
database="feast",
35-
s3_staging_location="s3://feast-integration-tests/redshift/tests/ingestion",
36-
iam_role="arn:aws:iam::402087665549:role/redshift_s3_access_role",
34+
cluster_id=os.getenv("AWS_CLUSTER_ID", "feast-integration-tests"),
35+
region=os.getenv("AWS_REGION", "us-west-2"),
36+
user=os.getenv("AWS_USER", "admin"),
37+
database=os.getenv("AWS_DB", "feast"),
38+
s3_staging_location=os.getenv(
39+
"AWS_STAGING_LOCATION",
40+
"s3://feast-integration-tests/redshift/tests/ingestion",
41+
),
42+
iam_role=os.getenv(
43+
"AWS_IAM_ROLE", "arn:aws:iam::402087665549:role/redshift_s3_access_role"
44+
),
3745
)
3846

3947
def create_data_source(

‎sdk/python/tests/integration/registration/test_feature_store.py‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@
1111
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
1212
# See the License for the specific language governing permissions and
1313
# limitations under the License.
14+
import os
1415
import time
1516
from datetime import datetime, timedelta
1617
from tempfile import mkstemp
@@ -75,12 +76,17 @@ def feature_store_with_gcs_registry():
7576

7677
@pytest.fixture
7778
def feature_store_with_s3_registry():
79+
aws_registry_path = os.getenv(
80+
"AWS_REGISTRY_PATH", "s3://feast-integration-tests/registries"
81+
)
7882
return FeatureStore(
7983
config=RepoConfig(
80-
registry=f"s3://feast-integration-tests/registries/{int(time.time() * 1000)}/registry.db",
84+
registry=f"{aws_registry_path}/{int(time.time() * 1000)}/registry.db",
8185
project="default",
8286
provider="aws",
83-
online_store=DynamoDBOnlineStoreConfig(region="us-west-2"),
87+
online_store=DynamoDBOnlineStoreConfig(
88+
region=os.getenv("AWS_REGION", "us-west-2")
89+
),
8490
offline_store=FileOfflineStoreConfig(),
8591
)
8692
)

‎sdk/python/tests/integration/registration/test_registry.py‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@
1111
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
1212
# See the License for the specific language governing permissions and
1313
# limitations under the License.
14+
import os
1415
import time
1516
from datetime import timedelta
1617
from tempfile import mkstemp
@@ -63,8 +64,11 @@ def gcs_registry() -> Registry:
6364

6465
@pytest.fixture
6566
def s3_registry() -> Registry:
67+
aws_registry_path = os.getenv(
68+
"AWS_REGISTRY_PATH", "s3://feast-integration-tests/registries"
69+
)
6670
registry_config = RegistryConfig(
67-
path=f"s3://feast-integration-tests/registries/{int(time.time() * 1000)}/registry.db",
71+
path=f"{aws_registry_path}/{int(time.time() * 1000)}/registry.db",
6872
cache_ttl_seconds=600,
6973
)
7074
return Registry(registry_config, None)

0 commit comments

Comments
 (0)