Skip to content

Allow skipped task state task_instance_schema.py#31421

Merged
potiuk merged 11 commits into
apache:mainfrom
JoranDox:patch-1
Jun 8, 2023
Merged

Allow skipped task state task_instance_schema.py#31421
potiuk merged 11 commits into
apache:mainfrom
JoranDox:patch-1

Conversation

@JoranDox

@JoranDox JoranDox commented May 19, 2023

Copy link
Copy Markdown
Contributor

closes: #31420

this PR is mostly to accompany my issue #31420

@JoranDox
JoranDox requested a review from ephraimbuddy as a code owner May 19, 2023 14:28
@boring-cyborg boring-cyborg Bot added the area:API Airflow's REST/HTTP API label May 19, 2023
@boring-cyborg

boring-cyborg Bot commented May 19, 2023

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our pre-commits will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: [email protected]
    Slack: https://s.apache.org/airflow-slack

@hussein-awala hussein-awala left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Allowing all the TI states is not a good idea, where this can lead to unexpected result.
IMO, the task could be stucked if we change its state manually to deferred, removed, shutdown, ...
You can just add skipped as accepted value in these methods. Don't forget to add a unit test for your change

@JoranDox

Copy link
Copy Markdown
Contributor Author

Fair enough, I'm just a sucker for generic solutions, but you're right, we probably wouldn't want states like those to be set manually.
I've been looking through the tests but can't seem to find the ones for that part of the api, I feel like they should be in here: https://github.com/apache/airflow/blob/main/airflow/api/common/mark_tasks.py but I think I'm missing something.

Comment thread airflow/api_connexion/schemas/task_instance_schema.py Outdated
@pierrejeambrun pierrejeambrun changed the title allow all task states task_instance_schema.py allow skipped task states task_instance_schema.py May 25, 2023
@pierrejeambrun pierrejeambrun changed the title allow skipped task states task_instance_schema.py Allow skipped task states task_instance_schema.py May 25, 2023

@hussein-awala hussein-awala left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a new contributor, you should read and follow this doc.

I'll merge the PR once the CI is passed.

@hussein-awala

Copy link
Copy Markdown
Member

@potiuk, do you have any idea why some of the CI checks fail without being executed? Is it related to the changes made yesterday, or is it just a temporary problem?

@potiuk

potiuk commented May 26, 2023

Copy link
Copy Markdown
Member

Likely intermittent. Looks like temporary Github Issue. Easiest is to rebase it or close/reopen such PR.

@hussein-awala

Copy link
Copy Markdown
Member

It still has the same problem after rebasing and closing/reopening it.
I wonder if it's something related to a maintainer commit in no-maintainer PR 🤔

@hussein-awala

Copy link
Copy Markdown
Member

@potiuk I think it's indeed related, the job has self-hosted label, but it was blocked by the security policy. wdyt?
Here is the debugging log for one of the failed jobs:

Requested labels: self-hosted
Job defined at: apache/airflow/.github/workflows/ci.yml@refs/pull/31421/merge
Waiting for a runner to pick up this job...
Job is about to start running on the runner: Airflow Runner 62 (organization)
Error: Running job on worker Airflow Runner 62 disallowed by security policy
##[debug]Finishing: Push Early Image Cache (linux/amd64)

@uranusjr uranusjr added the use public runners Makes sure that Public runners are used even if commiters creates the PR (useful for testing) label May 26, 2023
@uranusjr uranusjr closed this May 26, 2023
@uranusjr uranusjr reopened this May 26, 2023
Comment thread airflow/api_connexion/schemas/task_instance_schema.py Outdated
@pierrejeambrun

Copy link
Copy Markdown
Member

@JoranDox The PR is almost ready, can you fix the CI so we can merge it ?

@pierrejeambrun pierrejeambrun changed the title Allow skipped task states task_instance_schema.py Allow skipped task state task_instance_schema.py Jun 7, 2023
@pierrejeambrun pierrejeambrun changed the title Allow skipped task state task_instance_schema.py Allow skipped task state task_instance_schema.py Jun 7, 2023
@JoranDox

JoranDox commented Jun 7, 2023

Copy link
Copy Markdown
Contributor Author

@JoranDox The PR is almost ready, can you fix the CI so we can merge it ?

Apologies, I thought the CI had issues unrelated to my code and was kind of ignoring the mailthread, thanks for pinging me! I forgot to import TaskInstanceState, but that should be all I think.

@ephraimbuddy ephraimbuddy added this to the Airflow 2.7.0 milestone Jun 7, 2023
@JoranDox

JoranDox commented Jun 7, 2023

Copy link
Copy Markdown
Contributor Author

Why does black complain about files I didn't even edit? I can't even find them...
image

@JoranDox

JoranDox commented Jun 8, 2023

Copy link
Copy Markdown
Contributor Author

No idea what those sql failing tests have to do with my PR..

e.g. the fact that these two strings are not the same doesn't seem quite relevant to allowing the "skipped" state?

E - http://localhost:8080/ javascript:alert(1)
E + /home

@pierrejeambrun

Copy link
Copy Markdown
Member

Make sure to rebase your branch, you are 25 commits behind, there's been work on those, and fix might already be in main.

@potiuk
potiuk merged commit fba6f86 into apache:main Jun 8, 2023
@boring-cyborg

boring-cyborg Bot commented Jun 8, 2023

Copy link
Copy Markdown

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

@JoranDox
JoranDox deleted the patch-1 branch June 14, 2023 10:58
@ephraimbuddy ephraimbuddy added the type:new-feature Changelog: New Features label Jul 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:API Airflow's REST/HTTP API type:new-feature Changelog: New Features use public runners Makes sure that Public runners are used even if commiters creates the PR (useful for testing)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

allow other states than success/failed in tasks by REST API

6 participants