Skip to content

DataprocJobBaseOperator not compatible with TaskGroups #23439

@eladkal

Description

@eladkal

Body

Following Stackoverflow question: https://stackoverflow.com/questions/72091119/airflow-issues-with-calling-taskgroup

The issue is that when defining task in TaskGroup the identifier of the task becomes group_id.task_id
DataprocJobBaseOperator have default of using task_id for job name but Google doesn't allow the . char :

google.api_core.exceptions.InvalidArgument: 400 Job id 'weekday_analytics.avg_speed_20220502_22c11bdf' must conform to '[a-zA-Z0-9]([a-zA-Z0-9\-\_]{0,98}[a-zA-Z0-9])?' pattern

We probably should fix DataprocJobBaseOperator to handle cases where the task defined in task group by replacing the . to another char.

Committer

  • I acknowledge that I am a maintainer/committer of the Apache Airflow project.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions