Add 'postfixPath' to support time pattern in the middle of paths #858

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

enya-yx merged 14 commits into feathr-ai:main from enya-yx:enya/postfix-path

Dec 2, 2022

Collaborator

enya-yx commented Nov 15, 2022 •

edited

Loading

Description

Support time pattern in the middle of data sources paths
Modify scala source codes and add related test cases
Expose the API to python client
Add functions to copy directory for both databricks and synapse
Add test cases with 'hourly' and 'postfix-path' for both feature join and feature generation
Add some documents

How was this PR tested?

Test cases added into pipeline
Conducted more test cases locally

enya-yx added 2 commits

November 15, 2022 17:18


          Add 'postfixPath' to support time pattern in the middle of paths

ed32ccd


          quick fix

96e3d3a

enya-yx requested review from blrchen, jaymo001, windoze and xiaoyongzhu

November 15, 2022 09:28

xiaoyongzhu added the safe to test label

enya-yx and others added 4 commits

November 21, 2022 11:18


          Merge branch 'main' into enya/postfix-path

0799dd7


          Expose 'postfix_path' to datasource API

3b75e9c

Add postfix test cases for databricks


          quick change

f4dcdcf


          Merge branch 'main' into enya/postfix-path

xiaoyongzhu reviewed

View reviewed changes

feathr_project/feathr/spark_provider/_databricks_submission.py Show resolved Hide resolved

xiaoyongzhu reviewed

View reviewed changes

feathr_project/feathr/spark_provider/_databricks_submission.py Outdated Show resolved Hide resolved

xiaoyongzhu reviewed

View reviewed changes

feathr_project/test/test_azure_spark_e2e.py Outdated Show resolved Hide resolved

xiaoyongzhu reviewed

View reviewed changes

feathr_project/test/test_azure_spark_e2e.py Outdated Show resolved Hide resolved

xiaoyongzhu reviewed

View reviewed changes

feathr_project/feathr/utils/job_utils.py Show resolved Hide resolved

xiaoyongzhu reviewed

View reviewed changes

feathr_project/test/test_azure_spark_e2e.py Outdated Show resolved Hide resolved

Member

xiaoyongzhu commented Nov 23, 2022

Thanks @enya-yx for this PR, this looks good to me other than some minor comments on the duplicated code/naming conventions.


          Add more test cases & documents

abd2a08

enya-yx requested review from Yuqing-cat, edwincheung, jainr and mufajjul

November 24, 2022 13:59


          quick fix

458d74e

enya-yx mentioned this pull request

Support 'hourly' resolution for 'timePartitionPattern' #844

Closed

2 tasks

xiaoyongzhu reviewed

View reviewed changes

feathr_project/test/test_fixture.py Show resolved Hide resolved

xiaoyongzhu reviewed

View reviewed changes

feathr_project/test/test_time_partition_pattern_e2e.py Outdated Show resolved Hide resolved

xiaoyongzhu reviewed

View reviewed changes

feathr_project/feathr/definition/materialization_settings.py Show resolved Hide resolved

xiaoyongzhu reviewed

View reviewed changes

docs/how-to-guides/feathr-input-format.md Outdated Show resolved Hide resolved

xiaoyongzhu reviewed

View reviewed changes

feathr_project/feathr/spark_provider/_synapse_submission.py Outdated Show resolved Hide resolved

enya-yx and others added 6 commits

November 30, 2022 16:29


          quick change to merge with main

01372bb


          quick changes

b6979f8


          merge to main

11349f7


          move test data

8aa6180


          quick fix

e35c229


          quick change

db9146c

xiaoyongzhu approved these changes

View reviewed changes

Member

xiaoyongzhu left a comment

The test failures seem should to be irrelevant

windoze approved these changes

View reviewed changes

enya-yx merged commit 2ee97ce into feathr-ai:main

jaymo001 pushed a commit that referenced this pull request


          Add 'postfixPath' to support time pattern in the middle of paths (#858)

3c48645

* Add 'postfixPath' to support time pattern in the middle of paths
* Expose 'postfix_path' to datasource API
* Add test cases & documents

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

windoze windoze approved these changes

xiaoyongzhu xiaoyongzhu approved these changes

jaymo001 Awaiting requested review from jaymo001

blrchen Awaiting requested review from blrchen

edwincheung Awaiting requested review from edwincheung

jainr Awaiting requested review from jainr

mufajjul Awaiting requested review from mufajjul

Yuqing-cat Awaiting requested review from Yuqing-cat

Labels