Closes #3614 from verdan/double-dag-parsing

Closes #3576 from tfpk/master

Switched to using sshtunnel package instead of
popen approach

Closes #3473 from NielsZeilemaker/ssh_hook

Closes #3634 from verdan/AIRFLOW-2782-upgrade-
dagre-d3

Closes #3633 from happyjulie/AIRFLOW-2792

Closes #3635 from verdan/AIRFLOW-2782-dagred3-fix

If `email_on_failure` or `email_on_failure` set to
TRUE and email available, there will be email sent
out on event of retry or failure.

In the implementation, there is a argument
`is_retry` passed to method `self.email_alert`.
However, inside this method, this argument is not
used at all. I believer the initial author of this
method was planning to differentiate the email to
be sent out, but for whatever reason this was not
implemented.

Given in the email to be sent out, there will be a
line "Try {try_number} out of {max_tries}<br>", it
would be fine not to differentiate **retry email**
and **failure email**.

Closes #3640 from XD-DENG/patch-1

Closes #3631 from BagusThanatos/add-kmk-bbm

Closes #3632 from arihantsurana/AIRFLOW-2791

See mailing list thread "Flaky test case:
test_mark_success_no_kill".

Closes #3642 from
tedmiston/test_mark_success_no_kill-postgresql

Closes #3641 from verdan/AIRFLOW-2783-eslint

Closes #3627 from jeffkpayne/AIRFLOW-2765

previous version created the subdag by copying
over all the tasks, and
then filtering them down. it's a lot faster if we
only copy over the
tasks we need

Closes #3621 from abdul-stripe/faster-subdag

The tree view generates JSON that can be massive
for bigger DAGs,
up to 10s of MBs. The JSON is currently
prettified, which both
takes up more CPU time during serialization, and
slows down
everything else that uses it. Considering the JSON
is only
meant to be used programmatically, this is an easy
win

Closes #3620 from abdul-stripe/smaller-tree-view-
json

Closes #3615 from verdan/AIRFLOW-2766-shared-
datetime

Document dag_dir_list_interval in default_airflow.cfg

https://github.com/apache/incubator-airflow/blob/m
aster/UPDATING.md#dag_dir_list_interval

Closes #3645 from elgalu/patch-2

Fix typo in Xcom model timestamp field

No new testing - the field is already represented
in migrations

Closes #3652 from andywilco/fix_datetime_typo

Closes #3655 from tomchapin/master

Unit tests added for PigCliHook as well to prevent
future issues.

Closes #3594 from jakahn/master

"onwards" is not specific enough

* ssh_hook is not a port anymore
* auth backends should never had a copyright clause
* minihivecluster isnt used

Currently the role assumption method works only if
the granting account
does not specify an External ID. The external ID
is used to solved the
confused deputy problem. When using the AWS hook
to export data to
multiple customers, it's good security practice to
use the external ID.

There is no backwards compatibility break, the ID
will be `None` in
existing cases. Moto doesn't provide any
convenient way to verify the
value was passed in the credential response in
tests, so existing
test cases are kept.

Documentation: https://docs.aws.amazon.com/IAM/lat
est/UserGuide/id_roles_create_for-
user_externalid.html

Closes #3647 from vvondra/support_sts_external_id

value of min_file_process_interval in config
template is 0
However it's supposed to be 180 according to
airflow/jobs.py line 592

Closes #3659 from XD-DENG/patch-3

Closes #3578 from JacobHayes/py37-keywords

It's recommended by Falsk community to use random
SECRET_KEY for security reason.

However, in Airflow there is a default value for
secret_key and most users will ignore to change
it.

This may cause security concern.

Closes #3651 from XD-DENG/patch-2

…ver (#3909)

Adhoc queries and Charts features are no longer supported in new
FAB-based webserver and UI. But this is not mentioned at all in the doc
"Data Profiling" (https://airflow.incubator.apache.org/profiling.html)

This commit adds a note to remind users for this.

The docs refer to `conn_id` while the actual argument is `http_conn_id`.

#3913)

* [AIRFLOW-2407] Use feature detection for reload()

[Use feature detection instead of version detection](https://docs.python.org/3/howto/pyporting.html#use-feature-detection-instead-of-version-detection) is a Python porting best practice that avoids a flake8 undefined name error...

flake8 testing of https://github.com/apache/incubator-airflow on Python 3.6.3

* [AIRFLOW-2747] Explicit re-schedule of sensors

Add `mode` property to sensors. If set to `reschedule` an
AirflowRescheduleException is raised instead of sleeping which sets
the task back to state `NONE`. Reschedules are recorded in new
`task_schedule` table and visualized in the Gantt view. New TI
dependency checks if a sensor task is ready to be re-scheduled.

* Reformat sqlalchemy imports

* Make `_handle_reschedule` private

* Remove print

* Add comment

* Add comment

* Don't record reschule request in test mode

…w typos (#3924)

During normal operation, it is not necessary to see the message.  This
can only be useful when debugging an issue.

The current dockerised CI pipeline doesn't run minikube and the
Kubernetes integration tests. This starts a Kubernetes cluster 
using minikube and runs k8s integration tests using docker-compose.

One of the things for tests is being self contained. This means that
it should not depend on anything external, such as loading data.

This PR will use the setUp and tearDown to load the data into MySQL
and remove it afterwards. This removes the actual bash mysql commands
and will make it easier to dockerize the whole testsuite in the future

…ty_dataset to bigquery_hook (#3876)

The Flask flash messages are not displayed properly.

When we don't give a category for a flash message, defautl
value will be 'message'. In some cases, we specify 'error'
category.

Using Flask-AppBuilder, the flash message will be given
a CSS class 'alert-[category]'. But We don't have
'alert-message' or 'alert-error' in the current
'bootstrap-theme.css' file.

This makes the the flash messages in www_rbac UI come with
no background color.

This commit addresses this issue by adding 'alert-message'
(using specs of existing CSS class 'alert-info') and
'alert-error' (using specs of existing CSS class 'alert-danger')
into 'bootstrap-theme.css'.

…a UI by default

The ability to manipulate users from the  command line is a bit clunky.  Currently 'airflow create_user' and 'airflow delete_user' and 'airflow list_users'.  It seems that these ought to be made more like connections, so that it becomes 'airflow users list ...', 'airflow users delete ...' and 'airflow users create ...'

…deprecation warning (#3849)

)

artifacts in default_airflow.cfg

- fixed incorrect instructions in UPDATING.md regarding core.log_filename_template and elasticsearch.elasticsearch_log_id_template
- removed comments referencing "additional curly braces" from
default_airflow.cfg since they're irrelevant to the rendered airflow.cfg

The installation instructions failed to mention how to proceed with the GPL dependency. For those who are not concerned by GPL, it is useful to know how to proceed with GPL dependency.