Skip to content

Conversation

@akifcakir
Copy link
Contributor

@akifcakir akifcakir commented Feb 21, 2022


As its known in Airflow when we have a failed step (submitted a run for a Databricks notebook), we are not able to see the error in Airflow logs and need to visit the run page via the given link.

In order to see the notebook error directly in Airflow step logs as in below
image
, in airflow.providers.databricks.operators.databricks - DatabricksSubmitRunOperator, I enriched the AirflowException error message with the error key value from run output by using the Databricks api/2.0/jobs/runs/get-output API and additional method in databricks hook (airflow.providers.databricks.hooks.databricks - DatabricksHook). The operator and hook has been already tested with an example DAG in local setup by running test on both with succeed and fail cases.

The new operator has all the functionality of prev version DatabricksSubmitRunOperator. Besides it has the functionality that with the API endpoint fetches the notebook error and shows the notebook error directly in step logs in Airflow.

@boring-cyborg
Copy link

boring-cyborg bot commented Feb 21, 2022

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

  • Pay attention to the quality of your code (flake8, mypy and type annotations). Our pre-commits will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
  • Consider using Breeze environment for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: [email protected]
    Slack: https://s.apache.org/airflow-slack

@akifcakir akifcakir changed the title Add-raising-ValueError-feature-to-DatabricksSubmitRunOperator Add-showing-runtime-error-feature-to-DatabricksSubmitRunOperator Feb 21, 2022
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It'll make sense to add a test for this

Copy link
Contributor Author

@akifcakir akifcakir Feb 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ephraimbuddy added here, can you please check ? :c5846a8

@github-actions github-actions bot added the okay to merge It's ok to merge this PR as it does not require more tests label Feb 26, 2022
@github-actions
Copy link

The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.

@potiuk
Copy link
Member

potiuk commented Feb 27, 2022

Static checks are failing

@potiuk
Copy link
Member

potiuk commented Feb 28, 2022

needs rebase

@akifcakir akifcakir force-pushed the Add-raising-ValueError-feature-to-DatabricksSubmitRunOperator branch from c5846a8 to 7ce3499 Compare February 28, 2022 14:48
@akifcakir
Copy link
Contributor Author

@potiuk I rebased the pr, can you please run workflows ?

@akifcakir
Copy link
Contributor Author

Hi @potiuk, I could not see anything wrong which causes the static checks failure. If you could, can you point out that please ?

@potiuk
Copy link
Member

potiuk commented Mar 1, 2022

Please take a look at the failed check and read both explanation of the error and instuctions how to fix it (via pre-commit): It is all there if you follow details:

Screenshot from 2022-03-01 10-10-18

Look at explains it all. If you install pre-commit as:

  • strongly recommended in CONTRIBUTING.rst
  • described in detail in STATIC.txt
  • explained in detail in the error message you see in the static checks (including the exact command to run)

image

You won't even have to fix it, because pre-commits will fix it for you.

@akifcakir
Copy link
Contributor Author

Hi @potiuk I run with pre-commit and pushed already , can you please run workflows?

@akifcakir
Copy link
Contributor Author

Hi @potiuk I run with pre-commit run --all-files again since in first it did not catch the one error , can you please run workflows?

@akifcakir
Copy link
Contributor Author

Hi @potiuk do you think that pr can be merged ? It looks like passed all the checks.

@potiuk potiuk merged commit 62bf127 into apache:main Mar 1, 2022
@boring-cyborg
Copy link

boring-cyborg bot commented Mar 1, 2022

Awesome work, congrats on your first merged pull request!

@potiuk
Copy link
Member

potiuk commented Mar 1, 2022

Merged !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers okay to merge It's ok to merge this PR as it does not require more tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants