Skip to content

[AIRFLOW-4163] IntervalCheckOperator supports relative diff#4983

Merged
feng-tao merged 1 commit intoapache:masterfrom
milton0825:interval-check-operator-fail-zeroes
Apr 4, 2019
Merged

[AIRFLOW-4163] IntervalCheckOperator supports relative diff#4983
feng-tao merged 1 commit intoapache:masterfrom
milton0825:interval-check-operator-fail-zeroes

Conversation

@milton0825
Copy link
Copy Markdown
Contributor

@milton0825 milton0825 commented Mar 27, 2019 •
edited
Loading

Make sure you have checked all steps below.

Jira

Description

  • Here are some details about my PR, including screenshots of any UI changes:
    Make IntervalCheckOperator supports calculating ratio with relative difference. And also add an option to not ignore zero values in metric.

Tests

  • My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

  • My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
    1. Subject is separated from body by a blank line
    2. Subject is limited to 50 characters (not including Jira issue reference)
    3. Subject does not end with a period
    4. Subject uses the imperative mood ("add", not "adding")
    5. Body wraps at 72 characters
    6. Body explains "what" and "why", not "how"

Documentation

  • In case of new functionality, my PR adds documentation that describes how to use it.
    • When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.
    • All the public functions and the classes in the PR contain docstrings that explain what it does
    • If you implement backwards incompatible changes, please leave a note in the Updating.md so we can assign it to a appropriate release

Code Quality

  • Passes flake8

@milton0825 milton0825 changed the title [AIRFLOW-4163] IntervalCheckOperator supports relative diff and not i… [AIRFLOW-4163] IntervalCheckOperator supports relative diff. Mar 27, 2019
@milton0825 milton0825 force-pushed the interval-check-operator-fail-zeroes branch 3 times, most recently from ca4f6a9 to 69daf41 Compare March 27, 2019 04:51
@milton0825
Copy link
Copy Markdown
Contributor Author

milton0825 commented Mar 27, 2019

Tests passed on: https://travis-ci.org/milton0825/airflow/builds/511837166

@milton0825
Copy link
Copy Markdown
Contributor Author

milton0825 commented Mar 27, 2019

PTAL @feng-tao @ashb

@milton0825 milton0825 force-pushed the interval-check-operator-fail-zeroes branch 3 times, most recently from 8f91e08 to 01be1aa Compare March 29, 2019 03:55
@Fokko
Copy link
Copy Markdown
Contributor

Fokko commented Mar 29, 2019

@milton0825 can you rebase, this issue has been fixed on master.

@milton0825 milton0825 force-pushed the interval-check-operator-fail-zeroes branch from 01be1aa to a7cb809 Compare March 29, 2019 13:04
@feng-tao
Copy link
Copy Markdown
Member

feng-tao commented Mar 29, 2019

will take a look.

@codecov-io
Copy link
Copy Markdown

codecov-io commented Mar 29, 2019 •
edited
Loading

Codecov Report

Merging #4983 into master will increase coverage by 65.28%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #4983       +/-   ##
==========================================
+ Coverage   10.61%   75.9%   +65.28%     
==========================================
  Files         461     461               
  Lines       29955   29968       +13     
==========================================
+ Hits         3179   22746    +19567     
+ Misses      26776    7222    -19554
Impacted Files Coverage Δ
airflow/operators/check_operator.py 91.97% <100%> (+64.55%) ⬆️
airflow/plugins_manager.py 86.91% <0%> (+0.93%) ⬆️
airflow/executors/dask_executor.py 2% <0%> (+2%) ⬆️
airflow/exceptions.py 100% <0%> (+2.56%) ⬆️
airflow/config_templates/airflow_local_settings.py 76.47% <0%> (+2.94%) ⬆️
airflow/utils/operator_resources.py 86.95% <0%> (+4.34%) ⬆️
...etes_request_factory/kubernetes_request_factory.py 72.11% <0%> (+4.8%) ⬆️
airflow/executors/__init__.py 63.46% <0%> (+5.76%) ⬆️
airflow/settings.py 84.25% <0%> (+11.81%) ⬆️
airflow/_vendor/nvd3/ipynb.py 14.28% <0%> (+14.28%) ⬆️
... and 360 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 157f7bf...7fffd65. Read the comment docs.

@milton0825
Copy link
Copy Markdown
Contributor Author

milton0825 commented Apr 2, 2019 •
edited
Loading

@Fokko tests passed~

feng-tao
feng-tao approved these changes Apr 3, 2019
View reviewed changes
Copy link
Copy Markdown
Member

@feng-tao feng-tao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one nit, lgtm

Comment thread airflow/operators/check_operator.py Outdated
Copy link
Copy Markdown
Member

@feng-tao feng-tao Apr 3, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need that many logging info lines? Let's change it to debug. WDYT?

Copy link
Copy Markdown
Contributor Author

@milton0825 milton0825 Apr 3, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct me if I am wrong. I feel like we only turn on debug mode when there is an Airflow internal issue and we are trying to debug the Airflow internal issue.

For this case, task failure is usually caused by incorrect user behavior, e.g. wrong SQL (not related to Airflow internal), and we should provide the information users need to debug their issues.

Copy link
Copy Markdown
Member

@feng-tao feng-tao Apr 3, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, then could we make it a single log line(we could add \d in between) which is same as previous case?

Copy link
Copy Markdown
Contributor Author

@milton0825 milton0825 Apr 4, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed.

@milton0825 milton0825 force-pushed the interval-check-operator-fail-zeroes branch 2 times, most recently from 61013b7 to 42733f5 Compare April 4, 2019 01:05
@milton0825 milton0825 force-pushed the interval-check-operator-fail-zeroes branch from 42733f5 to 22c504b Compare April 4, 2019 01:09
@milton0825 milton0825 changed the title [AIRFLOW-4163] IntervalCheckOperator supports relative diff. [AIRFLOW-4163] IntervalCheckOperator supports relative diff Apr 4, 2019
@milton0825
Copy link
Copy Markdown
Contributor Author

milton0825 commented Apr 4, 2019

@feng-tao tests passed on https://travis-ci.org/milton0825/airflow/builds/515481016

@feng-tao feng-tao merged commit 6cc0780 into apache:master Apr 4, 2019
@feng-tao
Copy link
Copy Markdown
Member

feng-tao commented Apr 4, 2019

lgtm, thanks @milton0825

@feluelle feluelle mentioned this pull request Apr 9, 2019
Merged
6 tasks
milton0825 added a commit to lyft/airflow that referenced this pull request Apr 9, 2019
@milton0825
[AIRFLOW-4163] IntervalCheckOperator supports relative diff and not i…
8a2ed6f 
…gnore 0 (apache#4983)
@milton0825 milton0825 mentioned this pull request Apr 9, 2019
Merged
milton0825 added a commit to lyft/airflow that referenced this pull request Apr 9, 2019
@milton0825
[AIRFLOW-4163] IntervalCheckOperator supports relative diff and not i… (
0086508 
#125)

Make `IntervalCheckOperator` supports calculating ratio with relative difference. And also add an option to not ignore zero values in metric.

Cherry picked from: apache#4983
@milton0825 milton0825 deleted the interval-check-operator-fail-zeroes branch April 13, 2019 19:03
ashb pushed a commit that referenced this pull request Apr 15, 2019
@milton0825 @ashb
[AIRFLOW-4163] IntervalCheckOperator supports relative diff and not i…
a3265cc 
…gnore 0 (#4983)
cthenderson pushed a commit to cthenderson/apache-airflow that referenced this pull request Apr 16, 2019
@milton0825
[AIRFLOW-4163] IntervalCheckOperator supports relative diff and not i…
e34d531 
…gnore 0 (apache#4983)
andriisoldatenko pushed a commit to andriisoldatenko/airflow that referenced this pull request Jul 26, 2019
@milton0825 @andriisoldatenko
[AIRFLOW-4163] IntervalCheckOperator supports relative diff and not i…
d715873 
…gnore 0 (apache#4983)
wmorris75 pushed a commit to modmed-external/incubator-airflow that referenced this pull request Jul 29, 2019
@milton0825
[AIRFLOW-4163] IntervalCheckOperator supports relative diff and not i…
da90912 
…gnore 0 (apache#4983)
dharamsk pushed a commit to postmates/airflow that referenced this pull request Aug 8, 2019
@milton0825 @dharamsk
[AIRFLOW-4163] IntervalCheckOperator supports relative diff and not i…
064474a 
…gnore 0 (apache#4983)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@feng-tao feng-tao feng-tao approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@milton0825 @Fokko @feng-tao @codecov-io