Skip to content

Add "@asset" to decorate a function as a DAG and an asset#41325

Merged
Lee-W merged 29 commits intoapache:mainfrom
astronomer:asset-decorator
Nov 14, 2024
Merged

Add "@asset" to decorate a function as a DAG and an asset#41325
Lee-W merged 29 commits intoapache:mainfrom
astronomer:asset-decorator

Conversation

@uranusjr
Copy link
Member

@uranusjr uranusjr commented Aug 8, 2024 •
edited by Lee-W
Loading

Why

As part of AIP-75

What

Add an asset decorator which can be used to decorate a function and create a DAG and an asset.

@asset(uri="s3://bucket/object", schedule=None)
def asset2_producer(self, context, asset1_producer):
    print(self)
    print(context["inlet_events"][asset1_producer])

In the example above, it create a DAG(dag_id="asset2_producer", schedule=None, ...) and an Asset(name="asset2_producer", uri="s3://bucket/object"). Inside the function, self can be used to access the asset with name asset2_producer, context is just like context in a normal task and asset1_producer is used to access an asset with name=asset1_producer

Closes: #42314

@uranusjr uranusjr force-pushed the asset-decorator branch 2 times, most recently from f0ea15c to 017cdd9 Compare August 9, 2024 10:47
@Lee-W Lee-W mentioned this pull request Sep 23, 2024
Closed
2 tasks
@github-actions
Copy link

github-actions bot commented Sep 24, 2024

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Sep 24, 2024
@uranusjr uranusjr removed the stale Stale PRs per the .github/workflows/stale.yml policy file label Sep 25, 2024
@Lee-W Lee-W force-pushed the asset-decorator branch 3 times, most recently from 91fb531 to b6bb01b Compare October 25, 2024 05:41
@Lee-W Lee-W force-pushed the asset-decorator branch 4 times, most recently from 2b7dbda to cc11458 Compare October 30, 2024 11:28
Lee-W
Lee-W reviewed Oct 30, 2024
View reviewed changes
@Lee-W
Copy link
Member

Lee-W commented Oct 30, 2024

@uranusjr I tried to make the @asset work and added an example DAG and some unit tests. The next thing I will work on is accessing self and context. But would be great if you could take a look in advance in case I'm doing something wrong. Thanks!

uranusjr
uranusjr commented Oct 31, 2024
View reviewed changes
@Lee-W Lee-W force-pushed the asset-decorator branch 5 times, most recently from 6257830 to 0ef189e Compare November 1, 2024 10:21
@Lee-W Lee-W marked this pull request as ready for review November 1, 2024 10:37
@Lee-W Lee-W changed the title [WIP] Implement asset definition creating a DAG Add "@asset" to decorate a function as a DAG and an asset Nov 1, 2024
@Lee-W Lee-W force-pushed the asset-decorator branch 2 times, most recently from 3ac1a6a to a7e4dcc Compare November 4, 2024 01:34
uranusjr
uranusjr commented Nov 4, 2024
View reviewed changes
uranusjr
uranusjr commented Nov 4, 2024
View reviewed changes
Lee-W and others added 20 commits November 14, 2024 16:07
@Lee-W
test(decorators/asset): add test cases to Test_AssetMainOperator
1d31d4d
@Lee-W
test(decorators/assets): remove unused fixtures
5e933f4
@Lee-W
docs(example_dag): add example dag for asset_decorator
c69045a
@Lee-W
feat(decorators/assets): allow passing self and context into asset
6c0ec47
@Lee-W
feat(decorators/assets): return actual asset in asset decorator
ad0c6d2
@Lee-W
refactor(decorators/assets): extract active assets fetching logic as …
290461a 
…_fetch_active_assets_by_name
@Lee-W
feat(decorators/assets): allow fethcing inlet events through AssetRef
00ecd55
@Lee-W
feat(decorators/assets): reorder import paths
574b602
@Lee-W
docs: update asset decorator example dag
8728714
@Lee-W
test: fix tests
6c2539a
@Lee-W
test(decorators/assets): extend test_determine_kwargs to cover active…
d42ce8c 
… asset
@Lee-W
fix: address easy to fix comments
17fce69
@Lee-W
fix: fix asset serialization
de73b46
@Lee-W
refactor(decorators/assets): postpone the attribute check to AssetDef…
0b76c81 
…inition instead of asset decorator
@uranusjr @Lee-W
Simplify group validators
69523dc 
The validate_identifier validator already checks the length, so we don't
need an extra one doing that.
@Lee-W
style(dag): remove _wrapped_definition
81ee77b
@Lee-W
style(decorators/assets): change types.FunctionType to Callable
7ccc9ab
@Lee-W
refactor(decorators/assets): make session in _fetch_active_assets_by_…
52c8b85 
…name required
@Lee-W
fix(decorators/asets): remove DAG.bulk_write_to_db and remove self ha…
7e4ecf8 
…ndling
@Lee-W
feat(utils/context): fetch asset_refs all at once
2060ead
@uranusjr
Copy link
Member Author

uranusjr commented Nov 14, 2024

GitHub seems to have changed something and now I can’t approve my own PR. So consider this as a manual approval. ✅

@Lee-W
Copy link
Member

Lee-W commented Nov 14, 2024

GitHub seems to have changed something and now I can’t approve my own PR. So consider this as a manual approval. ✅

Thanks! Let me merge it now

@Lee-W Lee-W merged commit b3362f8 into apache:main Nov 14, 2024
@Lee-W Lee-W deleted the asset-decorator branch November 14, 2024 09:14
This was referenced Nov 14, 2024
Closed
Merged
@potiuk
Copy link
Member

potiuk commented Nov 14, 2024

GitHub seems to have changed something and now I can’t approve my own PR. So consider this as a manual approval. ✅

They are really experimenting with Approvals it seems......

@vatsrahul1001 vatsrahul1001 mentioned this pull request Nov 19, 2024
Merged
@kaxil kaxil mentioned this pull request Jan 17, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Lee-W Lee-W Lee-W left review comments

@phanikumv phanikumv phanikumv approved these changes

@ashb ashb Awaiting requested review from ashb

@bolkedebruin bolkedebruin Awaiting requested review from bolkedebruin

@XD-DENG XD-DENG Awaiting requested review from XD-DENG

@sunank200 sunank200 Awaiting requested review from sunank200

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Implement skeleton @asset decorator

4 participants

@uranusjr @Lee-W @potiuk @phanikumv