Skip to content

feat: add scaffold script for directory + dataset.yaml setup #412

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
adlersantos merged 7 commits into from
Jul 29, 2022
Merged

feat: add scaffold script for directory + dataset.yaml setup #412

adlersantos merged 7 commits into from
Jul 29, 2022

Conversation

@alick-at-google
Copy link
Contributor

@alick-at-google alick-at-google commented Jul 12, 2022 •
edited by adlersantos
Loading

Description

Note: If you are adding or editing a dataset, please specify the dataset folder involved, e.g. datasets/google_trends

Checklist

  • (Required) This pull request is appropriately labeled
  • Please merge this pull request after it's approved

Use the sections below based on what's applicable to your PR and delete the rest:

Feature

  • I'm adding or editing a feature
  • I have added/revised tests for the feature

adlersantos
adlersantos reviewed Jul 14, 2022
View reviewed changes
adlersantos
adlersantos reviewed Jul 14, 2022
View reviewed changes
adlersantos
adlersantos reviewed Jul 14, 2022
View reviewed changes
@adlersantos
Copy link
Member

adlersantos commented Jul 14, 2022

When generating the resulting files, please add the license header:

public-datasets-pipelines/scripts/generate_dag.py

Lines 191 to 194 in 492b973

def write_to_file(contents: str, filepath: pathlib.Path):
license_header = pathlib.Path(TEMPLATE_PATHS["license"]).read_text() + "\n"
with open(filepath, "w") as file_:
file_.write(license_header + contents.replace(license_header, ""))

adlersantos
adlersantos reviewed Jul 14, 2022
View reviewed changes
adlersantos
adlersantos reviewed Jul 14, 2022
View reviewed changes
@adlersantos
Copy link
Member

adlersantos commented Jul 14, 2022

This is looking good for a 1st version! 👍🏼 Time to add tests.

adlersantos
adlersantos requested changes Jul 22, 2022
View reviewed changes
@adlersantos adlersantos added the feature request New feature or request label Jul 22, 2022
@adlersantos adlersantos changed the title feat: add generate pipeline directory + dataset.yaml script feat: add scaffold script for directory + dataset.yaml setup Jul 29, 2022
adlersantos
adlersantos approved these changes Jul 29, 2022
View reviewed changes
Copy link
Member

@adlersantos adlersantos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work!

@adlersantos adlersantos merged commit 5bf354b into GoogleCloudPlatform:main Jul 29, 2022
@release-please release-please bot mentioned this pull request Jul 29, 2022
Merged
vijay-google pushed a commit to vijay-google/public-datasets-pipelines that referenced this pull request Aug 17, 2022
@alick-at-google @vijay-google
feat: add scaffold script for directory + dataset.yaml setup (GoogleC…
3ce4ea6 
…loudPlatform#412)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@adlersantos adlersantos adlersantos approved these changes

Assignees

No one assigned

Labels

feature request New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@alick-at-google @adlersantos