Skip to content

Update EKS sample DAGs and docs #21508

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
ferruzzi wants to merge 19 commits into from
Closed

Update EKS sample DAGs and docs #21508

ferruzzi wants to merge 19 commits into from

Conversation

@ferruzzi
Copy link
Contributor

@ferruzzi ferruzzi commented Feb 11, 2022

Part of a project to standardize AWS provider package sample DAGs and documentation.

  • Adds entries in the docs for the three EKS State Sensors
  • All EKS sample DAGs are reformatted to be more consistent and to make the embedded snippets in the docs more approachable
  • There is an example DAG that shows off how to use Jinja templates but it didn't actually use them for much, so I made that a better demo.

related: #21475

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

@ferruzzi ferruzzi requested a review from mik-laj as a code owner February 11, 2022 00:26
@ferruzzi
Copy link
Contributor Author

ferruzzi commented Feb 11, 2022 •
edited
Loading

Fixing two of the three errors (my oops) but not sure about this one.

File "/opt/airflow/airflow/providers/amazon/aws/example_dags/example_eks_templated.py", line 58, in <module>
VPC_CONFIG = json.loads("{{ dag_run.conf['resources_vpc_config'] }}")
    <<snip>>
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)

dstandish and others added 17 commits February 11, 2022 08:54
@dstandish @ferruzzi
Allow blank lines after docstring (apache#21477)
0098070
@dstandish @ferruzzi
Fix typing of operator attrs for mypy (apache#21480)
c922e2b
@ferruzzi
Added SNS example DAG and rst (apache#21475)
b617496
@jhtimmins @ashb @ferruzzi
Simplify fab has access lookup (apache#21482)
adeba23 
Co-authored-by: Ash Berlin-Taylor <[email protected]>
@uranusjr @ferruzzi
Rewrite decorated task mapping (apache#21328)
c13aa11
@mhenc @ferruzzi
Move Zombie detection to SchedulerJob (apache#21181)
ce07146
@IKholopov @ferruzzi
Modernize DAG-related URL routes and rename "tree" to "grid" (apache#…
862ec76 
…20730)

Co-authored-by: Igor Kholopov <[email protected]>
@ashb @ferruzzi
Fix broken tests after URL restructure PR (apache#21489)
4fb5745
@pingzh @ferruzzi
Log memory usage in CgroupTaskRunner (apache#21481)
6c28d3a
@pingzh @ferruzzi
Fix Resources __eq__ check (apache#21442)
201cfd4
@eladkal @ferruzzi
Update link in Postgres connection doc (apache#21490)
9b3064f
@KevinYanesG @ferruzzi
Fix postgres hook import pipeline tutorial (apache#21491)
5899d4d
@davidpr91 @ferruzzi
Optionally raise an error if source file does not exist in GCSToGCSOp…
bffc442 
…erator (apache#21391)
@josh-fell @ferruzzi
Update Zendesk example DAG to use TaskFlow API (apache#21411)
039f13b
@Melodie97 @ferruzzi
Update docs/readme.rst (apache#21346)
855df82
@Melodie97 @ferruzzi
add how-to guide for pig operator (apache#21498)
60c02e0
@ferruzzi
Fixes missing params in tasks
3fbd376
@ferruzzi
Copy link
Contributor Author

ferruzzi commented Feb 11, 2022

oof. Sorry about that, I must have screwed up a rebase.

@ferruzzi ferruzzi closed this Feb 11, 2022
@ferruzzi ferruzzi mentioned this pull request Feb 11, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mik-laj mik-laj Awaiting requested review from mik-laj

@turbaszek turbaszek Awaiting requested review from turbaszek

@ryanahamilton ryanahamilton Awaiting requested review from ryanahamilton

@ashb ashb Awaiting requested review from ashb

@bbovenzi bbovenzi Awaiting requested review from bbovenzi

@kaxil kaxil Awaiting requested review from kaxil

@potiuk potiuk Awaiting requested review from potiuk

@jedcunningham jedcunningham Awaiting requested review from jedcunningham

@ephraimbuddy ephraimbuddy Awaiting requested review from ephraimbuddy

@XD-DENG XD-DENG Awaiting requested review from XD-DENG

Assignees

No one assigned

Labels

area:providers kind:documentation provider:amazon AWS/Amazon - related issues

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.