Skip to content

[KubernetesExecutor] Don't skip succeeded status#42624

Merged
RNHTTR merged 2 commits intoapache:mainfrom
vlieven:kubernetes_dont_skip_succeeded_events
Oct 11, 2024
Merged

[KubernetesExecutor] Don't skip succeeded status#42624
RNHTTR merged 2 commits intoapache:mainfrom
vlieven:kubernetes_dont_skip_succeeded_events

Conversation

@vlieven
Copy link
Contributor

@vlieven vlieven commented Oct 1, 2024 •
edited
Loading

This PR implements the fix proposed in #41436 and guards against duplicated success messages by having _change_state() in the KubernetesExecutor only emit to the event_buffer if the task is in self.running,
as proposed by @jedcunningham in #40516.

closes: #40516, closes #41436

@boring-cyborg boring-cyborg bot added area:providers provider:cncf-kubernetes Kubernetes (k8s) provider related issues labels Oct 1, 2024
@boring-cyborg
Copy link

boring-cyborg bot commented Oct 1, 2024

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our pre-commits will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: [email protected]
    Slack: https://s.apache.org/airflow-slack

@vlieven vlieven mentioned this pull request Oct 1, 2024
Closed
2 tasks
@vlieven vlieven force-pushed the kubernetes_dont_skip_succeeded_events branch from 074e73b to c569499 Compare October 1, 2024 15:52
@eladkal eladkal force-pushed the kubernetes_dont_skip_succeeded_events branch from 1ccc083 to 3fad578 Compare October 4, 2024 02:23
jedcunningham
jedcunningham reviewed Oct 4, 2024
View reviewed changes
Copy link
Member

@jedcunningham jedcunningham left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @vlieven.

Can you fix (or probably remove) the broken test, and add coverage for the early return?

@vlieven
Copy link
Contributor Author

vlieven commented Oct 7, 2024 •
edited
Loading

Sorry about that! I removed the broken test, and added a new one to check that no event is emitted when the task key is not present in running.

@vlieven vlieven force-pushed the kubernetes_dont_skip_succeeded_events branch from 9b0f1f6 to 8339243 Compare October 8, 2024 11:23
@RNHTTR
Copy link
Contributor

RNHTTR commented Oct 9, 2024 •
edited
Loading

@vlieven can you please resolve the merge conflicts? I believe this is caused by providers being moved to the root of the project. We'll probably also want a backport PR.

@ashb
Split providers out of the main "airflow/" tree into a UV workspace p…
d5b6d44 
…roject (#42505)

This is only a partial split so far. It moves all the code and tests, but
leaves the creation of `core/` to a separate PR as this is already large
enough.

In addition to the straight file rename the other changes I had to make here
are:

- Some mypy/typing fixes.

  Mypy can be fragile about what it picks up when, so maybe some of those
  changes were caused by that. But the typing changes aren't large.

- Improve typing in common.sql type stub

  Again, likely a mypy file oddity, but the types should be safe

- Removed the `check-providers-init-file-missing` check

  This isn't needed now that airflow/providers shouldn't exist at all in the
  main tree.

- Create a "dev.tests_common" package that contains helper files and common
  pytest fixtures

  Since the provider tests are no longer under tests/ they don't automatically
  share the fixtures from the parent `tests/conftest.py` so they needed
  extracted.

  Ditto for `tests.test_utils` -- they can't be easily imported in provider
  tests anymore, so they are moved to a more explicit shared location.

In future we should switch how the CI image is built to make better use of UV
caching than our own approach as that would remvoe a lot of custom code.
@vlieven vlieven force-pushed the kubernetes_dont_skip_succeeded_events branch from 8339243 to d5b6d44 Compare October 10, 2024 09:34
@vlieven
Copy link
Contributor Author

vlieven commented Oct 10, 2024

Hi @RNHTTR, I rebased the changes, should be good again now.
Could you point me to what is expected as backport PR?

@RNHTTR
Copy link
Contributor

RNHTTR commented Oct 11, 2024

Hi @RNHTTR, I rebased the changes, should be good again now. Could you point me to what is expected as backport PR?

I believe you just need to create a new branch against v2-10-test (not main), run git cherry-pick <commit> (you can find the commit ID in the PR where it says “merge commit into main”), and creating a new PR with thie new branch.

@eladkal
Copy link
Contributor

eladkal commented Oct 11, 2024

Hi @RNHTTR, I rebased the changes, should be good again now. Could you point me to what is expected as backport PR?

I believe you just need to create a new branch against v2-10-test (not main), run git cherry-pick <commit> (you can find the commit ID in the PR where it says “merge commit into main”), and creating a new PR with thie new branch.

Why? This is provider only change. PR is against main branch and that is how it should be.

@RNHTTR
Copy link
Contributor

RNHTTR commented Oct 11, 2024

Hi @RNHTTR, I rebased the changes, should be good again now. Could you point me to what is expected as backport PR?

I believe you just need to create a new branch against v2-10-test (not main), run git cherry-pick <commit> (you can find the commit ID in the PR where it says “merge commit into main”), and creating a new PR with thie new branch.

Why? This is provider only change. PR is against main branch and that is how it should be.

Oh, duh, sorry about that.

@RNHTTR RNHTTR merged commit a052d9e into apache:main Oct 11, 2024
@boring-cyborg
Copy link

boring-cyborg bot commented Oct 11, 2024

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

kunaljubce pushed a commit to kunaljubce/airflow that referenced this pull request Oct 13, 2024
@vlieven @ashb
@RNHTTR @kunaljubce
Split providers out of the main "airflow/" tree into a UV workspace p…
97e14d1 
…roject (apache#42505) (apache#42624)

This is only a partial split so far. It moves all the code and tests, but
leaves the creation of `core/` to a separate PR as this is already large
enough.

In addition to the straight file rename the other changes I had to make here
are:

- Some mypy/typing fixes.

  Mypy can be fragile about what it picks up when, so maybe some of those
  changes were caused by that. But the typing changes aren't large.

- Improve typing in common.sql type stub

  Again, likely a mypy file oddity, but the types should be safe

- Removed the `check-providers-init-file-missing` check

  This isn't needed now that airflow/providers shouldn't exist at all in the
  main tree.

- Create a "dev.tests_common" package that contains helper files and common
  pytest fixtures

  Since the provider tests are no longer under tests/ they don't automatically
  share the fixtures from the parent `tests/conftest.py` so they needed
  extracted.

  Ditto for `tests.test_utils` -- they can't be easily imported in provider
  tests anymore, so they are moved to a more explicit shared location.

In future we should switch how the CI image is built to make better use of UV
caching than our own approach as that would remvoe a lot of custom code.

Co-authored-by: Ash Berlin-Taylor <[email protected]>
Co-authored-by: Ryan Hatter <[email protected]>
pavansharma36 pushed a commit to pavansharma36/airflow that referenced this pull request Oct 14, 2024
@vlieven @ashb
@RNHTTR @pavansharma36
Split providers out of the main "airflow/" tree into a UV workspace p…
3bbfaba 
…roject (apache#42505) (apache#42624)

This is only a partial split so far. It moves all the code and tests, but
leaves the creation of `core/` to a separate PR as this is already large
enough.

In addition to the straight file rename the other changes I had to make here
are:

- Some mypy/typing fixes.

  Mypy can be fragile about what it picks up when, so maybe some of those
  changes were caused by that. But the typing changes aren't large.

- Improve typing in common.sql type stub

  Again, likely a mypy file oddity, but the types should be safe

- Removed the `check-providers-init-file-missing` check

  This isn't needed now that airflow/providers shouldn't exist at all in the
  main tree.

- Create a "dev.tests_common" package that contains helper files and common
  pytest fixtures

  Since the provider tests are no longer under tests/ they don't automatically
  share the fixtures from the parent `tests/conftest.py` so they needed
  extracted.

  Ditto for `tests.test_utils` -- they can't be easily imported in provider
  tests anymore, so they are moved to a more explicit shared location.

In future we should switch how the CI image is built to make better use of UV
caching than our own approach as that would remvoe a lot of custom code.

Co-authored-by: Ash Berlin-Taylor <[email protected]>
Co-authored-by: Ryan Hatter <[email protected]>
R7L208 pushed a commit to R7L208/airflow that referenced this pull request Oct 17, 2024
@vlieven @ashb @RNHTTR
Split providers out of the main "airflow/" tree into a UV workspace p…
a9ac157 
…roject (apache#42505) (apache#42624)

This is only a partial split so far. It moves all the code and tests, but
leaves the creation of `core/` to a separate PR as this is already large
enough.

In addition to the straight file rename the other changes I had to make here
are:

- Some mypy/typing fixes.

  Mypy can be fragile about what it picks up when, so maybe some of those
  changes were caused by that. But the typing changes aren't large.

- Improve typing in common.sql type stub

  Again, likely a mypy file oddity, but the types should be safe

- Removed the `check-providers-init-file-missing` check

  This isn't needed now that airflow/providers shouldn't exist at all in the
  main tree.

- Create a "dev.tests_common" package that contains helper files and common
  pytest fixtures

  Since the provider tests are no longer under tests/ they don't automatically
  share the fixtures from the parent `tests/conftest.py` so they needed
  extracted.

  Ditto for `tests.test_utils` -- they can't be easily imported in provider
  tests anymore, so they are moved to a more explicit shared location.

In future we should switch how the CI image is built to make better use of UV
caching than our own approach as that would remvoe a lot of custom code.

Co-authored-by: Ash Berlin-Taylor <[email protected]>
Co-authored-by: Ryan Hatter <[email protected]>
harjeevanmaan pushed a commit to harjeevanmaan/airflow that referenced this pull request Oct 23, 2024
@vlieven @ashb
@RNHTTR @harjeevanmaan
Split providers out of the main "airflow/" tree into a UV workspace p…
01553fd 
…roject (apache#42505) (apache#42624)

This is only a partial split so far. It moves all the code and tests, but
leaves the creation of `core/` to a separate PR as this is already large
enough.

In addition to the straight file rename the other changes I had to make here
are:

- Some mypy/typing fixes.

  Mypy can be fragile about what it picks up when, so maybe some of those
  changes were caused by that. But the typing changes aren't large.

- Improve typing in common.sql type stub

  Again, likely a mypy file oddity, but the types should be safe

- Removed the `check-providers-init-file-missing` check

  This isn't needed now that airflow/providers shouldn't exist at all in the
  main tree.

- Create a "dev.tests_common" package that contains helper files and common
  pytest fixtures

  Since the provider tests are no longer under tests/ they don't automatically
  share the fixtures from the parent `tests/conftest.py` so they needed
  extracted.

  Ditto for `tests.test_utils` -- they can't be easily imported in provider
  tests anymore, so they are moved to a more explicit shared location.

In future we should switch how the CI image is built to make better use of UV
caching than our own approach as that would remvoe a lot of custom code.

Co-authored-by: Ash Berlin-Taylor <[email protected]>
Co-authored-by: Ryan Hatter <[email protected]>
PaulKobow7536 pushed a commit to PaulKobow7536/airflow that referenced this pull request Oct 24, 2024
@vlieven @ashb
@RNHTTR @PaulKobow7536
Split providers out of the main "airflow/" tree into a UV workspace p…
df3d685 
…roject (apache#42505) (apache#42624)

This is only a partial split so far. It moves all the code and tests, but
leaves the creation of `core/` to a separate PR as this is already large
enough.

In addition to the straight file rename the other changes I had to make here
are:

- Some mypy/typing fixes.

  Mypy can be fragile about what it picks up when, so maybe some of those
  changes were caused by that. But the typing changes aren't large.

- Improve typing in common.sql type stub

  Again, likely a mypy file oddity, but the types should be safe

- Removed the `check-providers-init-file-missing` check

  This isn't needed now that airflow/providers shouldn't exist at all in the
  main tree.

- Create a "dev.tests_common" package that contains helper files and common
  pytest fixtures

  Since the provider tests are no longer under tests/ they don't automatically
  share the fixtures from the parent `tests/conftest.py` so they needed
  extracted.

  Ditto for `tests.test_utils` -- they can't be easily imported in provider
  tests anymore, so they are moved to a more explicit shared location.

In future we should switch how the CI image is built to make better use of UV
caching than our own approach as that would remvoe a lot of custom code.

Co-authored-by: Ash Berlin-Taylor <[email protected]>
Co-authored-by: Ryan Hatter <[email protected]>
ellisms pushed a commit to ellisms/airflow that referenced this pull request Nov 13, 2024
@vlieven @ashb
@RNHTTR @ellisms
Split providers out of the main "airflow/" tree into a UV workspace p…
c59b6f1 
…roject (apache#42505) (apache#42624)

This is only a partial split so far. It moves all the code and tests, but
leaves the creation of `core/` to a separate PR as this is already large
enough.

In addition to the straight file rename the other changes I had to make here
are:

- Some mypy/typing fixes.

  Mypy can be fragile about what it picks up when, so maybe some of those
  changes were caused by that. But the typing changes aren't large.

- Improve typing in common.sql type stub

  Again, likely a mypy file oddity, but the types should be safe

- Removed the `check-providers-init-file-missing` check

  This isn't needed now that airflow/providers shouldn't exist at all in the
  main tree.

- Create a "dev.tests_common" package that contains helper files and common
  pytest fixtures

  Since the provider tests are no longer under tests/ they don't automatically
  share the fixtures from the parent `tests/conftest.py` so they needed
  extracted.

  Ditto for `tests.test_utils` -- they can't be easily imported in provider
  tests anymore, so they are moved to a more explicit shared location.

In future we should switch how the CI image is built to make better use of UV
caching than our own approach as that would remvoe a lot of custom code.

Co-authored-by: Ash Berlin-Taylor <[email protected]>
Co-authored-by: Ryan Hatter <[email protected]>
@vlieven vlieven deleted the kubernetes_dont_skip_succeeded_events branch November 3, 2025 08:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@RNHTTR RNHTTR RNHTTR approved these changes

@jedcunningham jedcunningham jedcunningham approved these changes

@hussein-awala hussein-awala Awaiting requested review from hussein-awala

Assignees

No one assigned

Labels

area:providers provider:cncf-kubernetes Kubernetes (k8s) provider related issues

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Airflow scheduler misses success events of worker pods on kubernetes Sensors stuck in queued after being spot interrupted

5 participants

@vlieven @RNHTTR @eladkal @jedcunningham @ashb