Skip to content

Comments

Rename dataset related python variable names to asset#41348

Merged
uranusjr merged 138 commits intoapache:mainfrom
astronomer:rename-dataset-as-asset
Sep 30, 2024
Merged

Rename dataset related python variable names to asset#41348
uranusjr merged 138 commits intoapache:mainfrom
astronomer:rename-dataset-as-asset

Conversation

@Lee-W
Copy link
Member

@Lee-W Lee-W commented Aug 9, 2024 •
edited
Loading

Why

as part of AIP-74

part of #42307

What

Rename the variable names in Airflow. DB, API, and UI changes will be done in the following PRs

  • Rename module airflow.api_connexion.schemas.dataset_schema as airflow.api_connexion.schemas.asset_schema

    • Rename variable create_dataset_event_schema as create_asset_event_schema
    • Rename variable dataset_collection_schema as asset_collection_schema
    • Rename variable dataset_event_collection_schema as asset_event_collection_schema
    • Rename variable dataset_event_schema as asset_event_schema
    • Rename variable dataset_schema as asset_schema
    • Rename class TaskOutletDatasetReferenceSchema as TaskOutletAssetReferenceSchema
    • Rename class DagScheduleDatasetReferenceSchema as DagScheduleAssetReferenceSchema
    • Rename class DatasetAliasSchema as AssetAliasSchema
    • Rename class DatasetSchema as AssetSchema
    • Rename class DatasetCollection as AssetCollection
    • Rename class DatasetEventSchema as AssetEventSchema
    • Rename class DatasetEventCollection as AssetEventCollection
    • Rename class DatasetEventCollectionSchema as AssetEventCollectionSchema
    • Rename class CreateDatasetEventSchema as CreateAssetEventSchema

  • Rename module airflow.datasets as airflow.assets

    • Rename class DatasetAlias as AssetAlias

    • Rename class DatasetAll as AssetAll

    • Rename class DatasetAny as AssetAny

    • Rename function expand_alias_to_datasets as expand_alias_to_assets

    • Rename class DatasetAliasEvent as AssetAliasEvent

      • Rename method dest_dataset_uri as dest_asset_uri

    • Rename class BaseDataset as BaseAsset

      • Rename method iter_datasets as iter_assets
      • Rename method iter_dataset_aliases as iter_asset_aliases

    • Rename class Dataset as Asset

      • Rename method iter_datasets as iter_assets
      • Rename method iter_dataset_aliases as iter_asset_aliases

    • Rename class _DatasetBooleanCondition as _AssetBooleanCondition

      • Rename method iter_datasets as iter_assets
      • Rename method iter_dataset_aliases as iter_asset_aliases

  • Rename module airflow.datasets.manager as airflow.assets.manager

    • Rename variable dataset_manager as asset_manager

    • Rename function resolve_dataset_manager as resolve_asset_manager

    • Rename class DatasetManager as AssetManager

      • Rename method register_dataset_change as register_asset_change
      • Rename method create_datasets as create_assets
      • Rename method register_dataset_change as notify_asset_created
      • Rename method notify_dataset_changed as notify_asset_changed

  • Rename module airflow.models.dataset as airflow.models.asset

    • Rename class DatasetDagRunQueue as AssetDagRunQueue
    • Rename class DatasetEvent as AssetEvent
    • Rename class DatasetModel as AssetModel
    • Rename class DatasetAliasModel as AssetAliasModel
    • Rename class DagScheduleDatasetReference as DagScheduleAssetReference
    • Rename class TaskOutletDatasetReference as TaskOutletAssetReference
    • Rename class DagScheduleDatasetAliasReference as DagScheduleAssetAliasReference

  • Rename module airflow.api_ui.views.datasets as airflow.api_ui.views.assets

    • Rename variable dataset_router as asset_rounter

  • Rename module airflow.listeners.spec.dataset as airflow.listeners.spec.asset

    • Rename function on_dataset_created as on_asset_created
    • Rename function on_dataset_changed as on_asset_changed

  • Rename module airflow.timetables.datasets as airflow.timetables.assets

    • Rename class DatasetOrTimeSchedule as AssetOrTimeSchedule

  • Rename module airflow.serialization.pydantic.dataset as airflow.serialization.pydantic.asset

    • Rename class DagScheduleDatasetReferencePydantic as DagScheduleAssetReferencePydantic
    • Rename class TaskOutletDatasetReferencePydantic as TaskOutletAssetReferencePydantic
    • Rename class DatasetPydantic as AssetPydantic
    • Rename class DatasetEventPydantic as AssetEventPydantic

  • Rename module airflow.datasets.metadata as airflow.assets.metadata

  • In module airflow.jobs.scheduler_job_runner

    • and its class SchedulerJobRunner

      • Rename method _create_dag_runs_dataset_triggered as _create_dag_runs_asset_triggered
      • Rename method _orphan_unreferenced_datasets as _orphan_unreferenced_datasets

  • In module airflow.api_connexion.security

    • Rename decorator requires_access_dataset as requires_access_asset

  • In module airflow.auth.managers.models.resource_details

    • Rename class DatasetDetails as AssetDetails

  • In module airflow.auth.managers.base_auth_manager

    • Rename function is_authorized_dataset as is_authorized_asset

  • In module airflow.timetables.simple

    • Rename class DatasetTriggeredTimetable as AssetTriggeredTimetable

  • In module airflow.lineage.hook

    • Rename class DatasetLineageInfo as AssetLineageInfo

      • Rename attribute dataset as asset

    • In its class HookLineageCollector

      • Rename method create_dataset as create_asset
      • Rename method add_input_dataset as add_input_asset
      • Rename method add_output_dataset as add_output_asset
      • Rename method collected_datasets as collected_assets

  • In module airflow.models.dag

    • Rename function get_dataset_triggered_next_run_info as get_asset_triggered_next_run_info

    • In its class DagModel

      • Rename method get_dataset_triggered_next_run_info as get_asset_triggered_next_run_info

  • In module airflow.models.taskinstance

    • and its class TaskInstance

      • Rename method _register_dataset_changes as _register_asset_changes

  • In module airflow.providers_manager

    • and its class ProvidersManager

      • Rename method initialize_providers_dataset_uri_resources as initialize_providers_asset_uri_resources
      • Rename attribute _discover_dataset_uri_resources as _discover_asset_uri_resources
      • Rename property dataset_factories as asset_factories
      • Rename property dataset_uri_handlers as asset_uri_handlers
      • Rename property dataset_to_openlineage_converters as asset_to_openlineage_converters

  • In module airflow.security.permissions

    • Rename constant RESOURCE_DATASET as RESOURCE_ASSET

  • In module airflow.serialization.enums

    • and its class DagAttributeTypes

      • Rename attribute DATASET_EVENT_ACCESSORS as ASSET_EVENT_ACCESSORS
      • Rename attribute DATASET_EVENT_ACCESSOR as ASSET_EVENT_ACCESSOR
      • Rename attribute DATASET as ASSET
      • Rename attribute DATASET_ALIAS as ASSET_ALIAS
      • Rename attribute DATASET_ANY as ASSET_ANY
      • Rename attribute DATASET_ALL as ASSET_ALL

  • In module airflow.serialization.pydantic.taskinstance

    • and its class TaskInstancePydantic

      • Rename method _register_dataset_changes as _register_dataset_changes

  • In module airflow.serialization.serialized_objects

    • Rename function encode_dataset_condition as encode_asset_condition
    • Rename function decode_dataset_condition as decode_asset_condition

  • In module airflow.timetables.base

    • Rename class _NullDataset as _NullAsset

      • Rename method iter_datasets as iter_assets

  • In module airflow.utils.context

    • Rename class LazyDatasetEventSelectSequence as LazyAssetEventSelectSequence

  • In module airflow.www.auth

    • Rename function has_access_dataset as has_access_asset

  • Rename configuration core.strict_dataset_uri_validation as core.strict_asset_uri_validation, core.dataset_manager_class as core.asset_manager_class and core.dataset_manager_class as core.asset_manager_class

  • Rename example dags example_dataset_alias.py, example_dataset_alias_with_no_taskflow.py, example_datasets.py as example_asset_alias.py, example_asset_alias_with_no_taskflow.py, example_assets.py

  • Rename DagDependency name dataset-alias, dataset as asset-alias, asset

  • Rename context key triggering_dataset_events as triggering_asset_events

  • Rename resource key dataset-uris as asset-uris for providers amazon, common.io, mysql, fab, postgres, trino

  • In provider airflow.providers.amazon.aws

    • Rename package datasets as assets

      • In its module s3

        • Rename method create_dataset as create_asset
        • Rename method convert_dataset_to_openlineage as convert_asset_to_openlineage

    • and its module auth_manager.avp.entities

      • Rename attribute AvpEntities.DATASET as AvpEntities.ASSET

    • and its module auth_manager.auth_manager.aws_auth_manager

      • Rename function is_authorized_dataset as is_authorized_asset

  • In provider airflow.providers.common.io

    • Rename package datasets as assets

      • in its module file

        • Rename method create_dataset as create_asset
        • Rename method convert_dataset_to_openlineage as convert_asset_to_openlineage

  • In provider airflow.providers.fab

    • in its module auth_manager.fab_auth_manager

      • Rename function is_authorized_dataset as is_authorized_asset

  • In provider airflow.providers.openlineage

    • in its module utils.utils

      • Rename class DatasetInfo as AssetInfo
      • Rename function translate_airflow_dataset as translate_airflow_asset

  • Rename package airflow.providers.postgres.datasets as airflow.providers.postgres.assets

  • Rename package airflow.providers.mysql.datasets as airflow.providers.mysql.assets

  • Rename package airflow.providers.trino.datasets as airflow.providers.trino.assets

  • Add module airflow.providers.common.compat.assets

  • Add module airflow.providers.common.compat.openlineage.utils.utils

  • Add moddule airflow.providers.common.compat.security.permissions.RESOURCE_ASSET

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@Lee-W Lee-W force-pushed the rename-dataset-as-asset branch from 9dcc589 to 35ac346 Compare August 9, 2024 10:45
@uranusjr uranusjr mentioned this pull request Aug 13, 2024
Closed
@uranusjr
Copy link
Member

uranusjr commented Aug 13, 2024

I have created #41424 to add the name attribute. These two will conflict, so remember to rebase.

@Lee-W
Copy link
Member Author

Lee-W commented Aug 13, 2024

sure, will do

@Lee-W Lee-W force-pushed the rename-dataset-as-asset branch 4 times, most recently from c6538fa to 875d372 Compare August 13, 2024 13:35
uranusjr
uranusjr reviewed Aug 13, 2024
View reviewed changes
@Lee-W Lee-W force-pushed the rename-dataset-as-asset branch 13 times, most recently from 35d6d6e to 870ba33 Compare August 19, 2024 01:27
@Lee-W Lee-W force-pushed the rename-dataset-as-asset branch from ee2b1b8 to 1c137e5 Compare September 30, 2024 00:23
@Lee-W
Copy link
Member Author

Lee-W commented Sep 30, 2024

Has the UI been tested if all is working? Some changes in UI very probably are not automatically be tested. Did not make a test and opened the UI. Might be a good point to double check not to break any link/HTML.

I've done some testing, and it worked fine. I tried to keep UI changes unrelated to the backend change in another separate PR (not yet there). As there are many conflicts today (sometime missing something during rebasing 🤦‍♂️), I'll try to resolve them and test them again. Thanks!

Just a quick update. I've done the test on my local end and it work fine. And the CI is finally green again! 🎉

uranusjr
uranusjr approved these changes Sep 30, 2024
View reviewed changes
Copy link
Member

@uranusjr uranusjr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Admittedly I did not read every line, but I think I read enough to be confident this is good…

@uranusjr uranusjr merged commit ede7cb2 into apache:main Sep 30, 2024
@uranusjr uranusjr deleted the rename-dataset-as-asset branch September 30, 2024 05:30
@shahar1 shahar1 mentioned this pull request Oct 2, 2024
Merged
@kaxil kaxil mentioned this pull request Oct 9, 2024
Merged
kaxil added a commit to astronomer/airflow that referenced this pull request Oct 9, 2024
@kaxil
Revert Asset to Dataset for Core Extension Doc
030a7bc 
This PR reverts Core Extension doc change (https://airflow.apache.org/docs/apache-airflow-providers/core-extensions/index.html) from apache#41348

Since those docs only work on Airflow stable, we can only change this after 3.0
kaxil added a commit that referenced this pull request Oct 9, 2024
@kaxil
Revert Asset to Dataset for Core Extension Doc (#42867)
bf00235 
This PR reverts Core Extension doc change (https://airflow.apache.org/docs/apache-airflow-providers/core-extensions/index.html) from #41348

Since those docs only work on Airflow stable, we can only change this after 3.0
@eladkal eladkal mentioned this pull request Oct 10, 2024
Closed
96 tasks
kunaljubce pushed a commit to kunaljubce/airflow that referenced this pull request Oct 13, 2024
@kaxil @kunaljubce
Revert Asset to Dataset for Core Extension Doc (apache#42867)
45c977f 
This PR reverts Core Extension doc change (https://airflow.apache.org/docs/apache-airflow-providers/core-extensions/index.html) from apache#41348

Since those docs only work on Airflow stable, we can only change this after 3.0
pavansharma36 pushed a commit to pavansharma36/airflow that referenced this pull request Oct 14, 2024
@kaxil @pavansharma36
Revert Asset to Dataset for Core Extension Doc (apache#42867)
91bcedc 
This PR reverts Core Extension doc change (https://airflow.apache.org/docs/apache-airflow-providers/core-extensions/index.html) from apache#41348

Since those docs only work on Airflow stable, we can only change this after 3.0
@mobuchowski mobuchowski mentioned this pull request Oct 15, 2024
Merged
R7L208 pushed a commit to R7L208/airflow that referenced this pull request Oct 17, 2024
@kaxil
Revert Asset to Dataset for Core Extension Doc (apache#42867)
eb0a028 
This PR reverts Core Extension doc change (https://airflow.apache.org/docs/apache-airflow-providers/core-extensions/index.html) from apache#41348

Since those docs only work on Airflow stable, we can only change this after 3.0
joaopamaral pushed a commit to joaopamaral/airflow that referenced this pull request Oct 21, 2024
@Lee-W @joaopamaral
Rename dataset related python variable names to asset (apache#41348)
673da92
harjeevanmaan pushed a commit to harjeevanmaan/airflow that referenced this pull request Oct 23, 2024
@kaxil @harjeevanmaan
Revert Asset to Dataset for Core Extension Doc (apache#42867)
ed61247 
This PR reverts Core Extension doc change (https://airflow.apache.org/docs/apache-airflow-providers/core-extensions/index.html) from apache#41348

Since those docs only work on Airflow stable, we can only change this after 3.0
PaulKobow7536 pushed a commit to PaulKobow7536/airflow that referenced this pull request Oct 24, 2024
@kaxil @PaulKobow7536
Revert Asset to Dataset for Core Extension Doc (apache#42867)
65fa10c 
This PR reverts Core Extension doc change (https://airflow.apache.org/docs/apache-airflow-providers/core-extensions/index.html) from apache#41348

Since those docs only work on Airflow stable, we can only change this after 3.0
@Lee-W Lee-W self-assigned this Oct 28, 2024
ellisms pushed a commit to ellisms/airflow that referenced this pull request Nov 13, 2024
@Lee-W @ellisms
Rename dataset related python variable names to asset (apache#41348)
9a38320
ellisms pushed a commit to ellisms/airflow that referenced this pull request Nov 13, 2024
@kaxil @ellisms
Revert Asset to Dataset for Core Extension Doc (apache#42867)
3fd5cf3 
This PR reverts Core Extension doc change (https://airflow.apache.org/docs/apache-airflow-providers/core-extensions/index.html) from apache#41348

Since those docs only work on Airflow stable, we can only change this after 3.0
@Lee-W Lee-W mentioned this pull request Nov 18, 2024
Closed
2 tasks
@Lee-W Lee-W mentioned this pull request Dec 2, 2024
Closed
@potiuk potiuk mentioned this pull request Jan 16, 2025
Merged
kosteev pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request May 27, 2025
@kaxil
Revert Asset to Dataset for Core Extension Doc (#42867)
2c5fb74 
This PR reverts Core Extension doc change (https://airflow.apache.org/docs/apache-airflow-providers/core-extensions/index.html) from apache/airflow#41348

Since those docs only work on Airflow stable, we can only change this after 3.0

GitOrigin-RevId: bf002356ba0a38d317adeb651f066588908a79a3
@kyungjunleeme kyungjunleeme mentioned this pull request Jul 14, 2025
Closed
2 tasks
kosteev pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Sep 22, 2025
@kaxil
Revert Asset to Dataset for Core Extension Doc (#42867)
25f0556 
This PR reverts Core Extension doc change (https://airflow.apache.org/docs/apache-airflow-providers/core-extensions/index.html) from apache/airflow#41348

Since those docs only work on Airflow stable, we can only change this after 3.0

GitOrigin-RevId: bf002356ba0a38d317adeb651f066588908a79a3
kosteev pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Oct 20, 2025
@kaxil
Revert Asset to Dataset for Core Extension Doc (#42867)
489d4fe 
This PR reverts Core Extension doc change (https://airflow.apache.org/docs/apache-airflow-providers/core-extensions/index.html) from apache/airflow#41348

Since those docs only work on Airflow stable, we can only change this after 3.0

GitOrigin-RevId: bf002356ba0a38d317adeb651f066588908a79a3
@sjyangkevin sjyangkevin mentioned this pull request Jan 25, 2026
Merged
ntBre pushed a commit to astral-sh/ruff that referenced this pull request Feb 9, 2026
@sjyangkevin
[airflow] Add ruff rules to catch deprecated attribute access from …
5338823 
…context key for Airflow 3.0 (`AIR301`) (#22850)

## Summary

<!-- What's the purpose of the change? What does it do, and why? -->
Context:
apache/airflow#41641

1. apache/airflow#45961
* <strike>create_dagrun removed from airflow...DAG</strike> (This has
already been implemented in AIR301)
* context key dag_run.external_trigger removed
2. apache/airflow#45960
* context["inlet_events"]["url"] → context["inlet_events"][Asset("url")]
3. apache/airflow#41348
* context key triggering_dataset_events → triggering_asset_events

The existing AIR301 rules can detect when users access a removed key
such as `execution_date` through Airflow's `context`. For example,
`context["execution_date"]`, or `context["dag_run"]`. However, if an
attribute is deprecated from a context key, such as
`context["dag_run"].external_trigger`, the current implement will not
flag it.

This PR adds the logic for such check, and add two rules to flag the
deprecated attribute for the `"dag_run"` and `"inlet_events"` context
key. In addition to this, `"triggering_dataset_events"` is a deprecated
context key which can be handled by the existing rule. However, the
existing rule doesn't raise a diagnostic. Hence, the rule logic is
refactored a little bit, such that we can add this check and suggest a
`Replacement::Rename`.

## Test Plan

<!-- How was it tested? -->
The test cases have been added to `AIR301_context.py`, and all the tests
have been run locally and success. @Lee-W , could you please review it
when you have time, thanks!

## Notes

In #22376, we introduced some
improvements to the AIR301 code. I will re-base this PR when we are all
good on that, so it can pick up those code structure improvements, and
updated rules.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mobuchowski mobuchowski mobuchowski left review comments

@uranusjr uranusjr uranusjr approved these changes

@jscheffl jscheffl jscheffl approved these changes

@bolkedebruin bolkedebruin Awaiting requested review from bolkedebruin

@potiuk potiuk Awaiting requested review from potiuk potiuk is a code owner

@ashb ashb Awaiting requested review from ashb ashb is a code owner

@jedcunningham jedcunningham Awaiting requested review from jedcunningham jedcunningham is a code owner

@kaxil kaxil Awaiting requested review from kaxil kaxil is a code owner

@eladkal eladkal Awaiting requested review from eladkal

@o-nikolas o-nikolas Awaiting requested review from o-nikolas

@jhtimmins jhtimmins Awaiting requested review from jhtimmins

@ryanahamilton ryanahamilton Awaiting requested review from ryanahamilton

@bbovenzi bbovenzi Awaiting requested review from bbovenzi

@pierrejeambrun pierrejeambrun Awaiting requested review from pierrejeambrun

@ephraimbuddy ephraimbuddy Awaiting requested review from ephraimbuddy ephraimbuddy is a code owner

@XD-DENG XD-DENG Awaiting requested review from XD-DENG

@vincbeck vincbeck Awaiting requested review from vincbeck

Assignees

@Lee-W Lee-W

Labels

AIP-74 Dataset -> Asset airflow3.0:breaking Candidates for Airflow 3.0 that contain breaking changes area:API Airflow's REST/HTTP API area:lineage area:providers area:serialization area:webserver Webserver related Issues provider:amazon AWS/Amazon - related issues provider:common-io provider:openlineage AIP-53

Projects

No open projects
AIP-74 Introducing Data Assets
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@Lee-W @uranusjr @mobuchowski @jscheffl