Ensure Kerberos token is valid in SparkSubmitOperator before running `yarn kill` #9044

aneesh-joseph · 2020-05-28T04:38:04Z

When the spark submit hook's on_kill is called, it tries to kill the yarn app, but doesn't do a kinit even if the keytab and principal is supplied. This would mean that the yarn applicaition -kill <app id> will fail when yarn needs kerberos authentication, and the app will continue to run.

This change tries to do an explicit kinit before killing the YARN app(if keytab and principal are provided).

Make sure to mark the boxes below before creating PR: [x]

Description above provides context of the change
Unit tests coverage for changes (not needed for documentation changes)
Target Github ISSUE in description if exists
Commits follow "How to write a good git commit message"
Relevant documentation is updated including usage instructions.
I will engage committers as explained in Contribution Workflow Example.

In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.

ashb · 2020-05-28T10:44:46Z

airflow/providers/apache/spark/hooks/spark_submit.py

I think this should be

Suggested change

kinit_cmd = "kinit -kt {} {}" \

.format(self._keytab, self._principal).split()

kinit = subprocess.Popen(kinit_cmd,

stdout=subprocess.PIPE,

stderr=subprocess.PIPE)

from airflow.security.kerberos import renew_from_kt

renew_from_kt(self._principal, self._keytab)

(But I'm not familiar with Kerberos, so I could be wrong)

cool, that would renew the kerberos token and store it at /tmp/airflow_krb5_ccache(configured in airflow.cfg) but yarn expects it at /tmp/krb5cc_<uid> the default token location , It would work if the env variable KRB5CCNAME is set to /tmp/<airflow.cfg.kerberos.ccache > in the subprocess used for yarn kill. will update it

@ashb - tried using the renew_from_kt and it seems to do the following

kinit -r 3600m -k -t my.keytab -c /tmp/airflow_krb5_ccache principal

and then does a workaround to renew the ticket

kinit -c /tmp/airflow_krb5_ccache -R

the latter seems to fail for non-renewable tickets and does a sys.exit, any suggestions on how this can be handled? I think the renewal work around won't be needed for a yarn kill

This change looks good, but can you add a comment to the nrew_from_kt call in SparkOperator to say why we aren't treating the errors as fatal.

done, can you have another look please

ashb

Use existing renew function please

aneesh-joseph · 2020-06-06T02:57:18Z

@ashb - squashed and rebased this, can you have another look please

airflow/providers/apache/spark/hooks/spark_submit.py

airflow/security/kerberos.py

aneesh-joseph · 2020-06-17T12:29:24Z

@ashb thank you for all the help with my first PR, have made the suggested changes, can you have a look please :)

ashb

One minor change, and please rebase on to latest master (see https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#id8 if you aren't sure how to do this) to fix the test failures.

airflow/providers/apache/spark/hooks/spark_submit.py

aneesh-joseph · 2020-06-17T13:48:56Z

One minor change, and please rebase on to latest master (see https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#id8 if you aren't sure how to do this) to fix the test failures.

done, rebased on top of master. ~~should I squash the commits as well?~~

found the answer on the same doc, squashed them, can you have a look please

ashb

LGTM. Do you think the title edit I just made is accurate?

ashb

Please add unit tests covering this behaviour (you can use mocking to have renew_from_kt be a no-op, but please exercise this code path in tests)

ashb · 2020-06-17T14:50:11Z

Current tests need updating too:

E       AssertionError: call(['yarn', 'application', '-kill', 'application_1486558679801_1820'], stderr=-1, stdout=-1) not found in [call(['spark-submit', '--master', 'yarn://yarn-master', '--name', 'default-name', '--queue', 'root.etl', '--deploy-mode', 'cluster', ''], bufsize=-1, stderr=-2, stdout=-1, universal_newlines=True),
E        call().wait(),
E        call().__bool__(),
E        call().poll(),
E        call().kill(),
E        call(['yarn', 'application', '-kill', 'application_1486558679801_1820'], env=None, stderr=-1, stdout=-1),
E        call().wait()]

tests/providers/apache/spark/hooks/test_spark_submit.py:683: AssertionError

aneesh-joseph · 2020-06-17T14:52:12Z

sure, will try adding tests and fixing this 👍

aneesh-joseph · 2020-06-29T16:35:49Z

hi @ashb, have added unit tests to cover the new path, can you have look please

ashb

Looks good

ashb · 2020-07-02T18:50:16Z

Could you rebase to latest master, hopefully that should fix the failing Kube tests

aneesh-joseph · 2020-07-02T19:47:53Z

Could you rebase to latest master, hopefully that should fix the failing Kube tests

done, but failed again, is there a way to re-run them?

aneesh-joseph · 2020-07-04T09:56:00Z

@ashb passed after another rebase

* Tests should also be triggered when there is just setup.py change (apache#9690) So far tests were not triggered when only requirements changed, but this is quite needed in fact. * Update FlaskAppBuilder to v3 (apache#9648) * Some Pylint fixes in airflow/models/taskinstance.py (apache#9674) * Update migrations to ensure compatibility with Airflow 1.10.* (apache#9660) closes apache#9640 * Fix _process_executor_events method to use in-memory try_number (apache#9692) * use the correct claim name in the webserver (apache#9688) * Update Thumbtack points of contact in Airflow Users list (apache#9701) The previously-listed person is no longer at the company * generate go client from openapi spec (apache#9502) * generate go client from openapi spec * move openapi codegen to seperate workflow * [AIRFLOW-XXXX] Remove unnecessary docstring in AWSAthenaOperator * Add health API endpoint (apache#8144) (apache#9277) * Add AWS StepFunctions integrations to the aws provider (apache#8749) * Move gcs & wasb task handlers to their respective provider packages (apache#9714) * Allow AWSAthenaHook to get more than 1000/first page of results (apache#6075) Co-authored-by: Dylan Joss <[email protected]> * Add Dag Runs CRUD endpoints (apache#9473) * Make airflow/migrations/env.py Pylint Compatible (apache#9670) * Get Airflow configs with sensitive data from Secret Backends (apache#9645) * YAML file supports extra json parameters (apache#9549) Co-authored-by: Kamil Breguła <[email protected]> Co-authored-by: Vinay <[email protected]> Co-authored-by: Kamil Breguła <[email protected]> * fix grammar in prereq tasks gcp operator docs (apache#9728) * Add The Climate Corporation to user list (apache#9726) * Add Qingping Hou to committers list (apache#9725) * Add new fantastic team member of Polidea. (apache#9724) * Error in description after deployment (apache#9723) * Error in description after deployment Co-authored-by: Daniel Debny <[email protected]> * Skip one version of Python for each test. Skip one version of Python for each test. * Add read-only endpoints for DAG Model (apache#9045) Co-authored-by: Tomek Urbaszek <[email protected]> Co-authored-by: Tomek Urbaszek <[email protected]> * Ensure Kerberos token is valid in SparkSubmitOperator before running `yarn kill` (apache#9044) do a kinit before yarn kill if keytab and principal is provided * Update example DAG for AI Platform operators (apache#9727) * Fix warning about incompatible plugins (apache#9704) One condition was bad and warns when the plugin is for admin and FAB flask. * Update local_task_job.py (apache#9746) Removing the suicide joke. * Tests are working for newly added backport providers (apache#9739) * Tests are working for newly added backport providers * Pre-create Celery db result tables before running Celery worker (apache#9719) Otherwise at large scale this can end up with some tasks failing as they try to create the result table at the same time. This was always possible before, just exceedingly rare, but in large scale performance testing where I create a lot of tasks quickly (especially in my HA testing) I hit this a few times. This is also only a problem for fresh installs/clean DBs, as once these tables exist the possible race goes away. This is the same fix from apache#8909, just for runtime, not test time. * Support extra config options for Sentry (apache#8911) For now only dsn can be configured through the airflow.cfg. Need support 'http_proxy' option for example (it can't be configured through the environment variables). This change implements solution for supporting all existed (and maybe future) options for sentry configuration. * Use namedtuple for TaskInstanceKeyType (apache#9712) * Use namedtuple for TaskInstanceKeyType * Add TargetQueryValue to KEDA Autoscaler (apache#9748) Co-authored-by: Daniel Imberman <[email protected]> * Add unit tests for mlengine_operator_utils (apache#9702) * Mask other forms of password arguments in SparkSubmitOperator (apache#9615) This is a follow-up to apache#6917 before modifying the masking code. Related: apache#9595. * Use absolute paths in howto guides (apache#9758) * Fix StackdriverTaskHandler + add system tests (apache#9761) Co-authored-by: Tomek Urbaszek <[email protected]> Co-authored-by: Tomek Urbaszek <[email protected]> * Check project structure in sensors/transfers directories (apache#9764) * Add tests for yandex hook (apache#9665) * improve type hinting for celery provider (apache#9762) * Add ME-Br to who uses Airflow list (apache#9770) * Add 1.10.11 Changelog & Update UPDATING.md (apache#9757) * Links Breeze documentation to new Breeze video (apache#9768) * Fix is_terminal_support_colors functtion (apache#9734) * Add type hinting for discord provider (apache#9773) * Fix typo in the word "Airflow" (apache#9772) * Add Google Stackdriver link (apache#9765) * Improve type hinting to provider microsoft (apache#9774) * Unit tests jenkins hook (apache#9767) * Fixes failing formatting of DAG file containing {} in docstring (apache#9779) * Upgrade to latest isort (5.0.8) (apache#9782) * Add API Endpoint - DagRuns Batch (apache#9556) Co-authored-by: Ephraim Anierobi <[email protected]> * Improve typing coverage in scheduler_job.py (apache#9783) * Enable pretty output in mypy (apache#9785) * provide_session keep return type (apache#9787) * Refactor Google operators guides (apache#9766) * Refactor Google guides * fixup! Refactor Google guides * fixup! fixup! Refactor Google guides * Fix small errors in image building documentation (apache#9792) * Backfill reset_dagruns set DagRun to NONE state (apache#9756) * Add DAG Source endpoint (apache#9322) * The group of embedded DAGs should be root to be OpenShift compatible (apache#9794) * Add docs for replace_microseconds parameters in trigger DAG endpoint (apache#9793) * Add multiple file upload functionality to GCS hook (apache#8849) Co-authored-by: Timothy Healy <[email protected]> * Keep functions signatures in decorators (apache#9786) * Use paths relative to root docs dir in *include directives (apache#9797) * Add Migration guide from the experimental API to the REST API (apache#9771) Co-authored-by: Kaxil Naik <[email protected]> Co-authored-by: Kamil Breguła <[email protected]> * Update paths in .github/boring-cyborg.yml (apache#9799) * Update paths in .github/boring-cyborg.yml * fixup! Update paths in .github/boring-cyborg.yml * Minor typo fix in OpenAPI specification (apache#9809) * Enable annotations to be added to the webserver service (apache#9776) * Make airflow package type check compatible (apache#9791) * Update README to add Py 3.8 in supported versions (apache#9804) * Remove unnecessary comprehension (apache#9805) * Add type annotations for redis provider (apache#9815) * Remove package.json and yarn.lock from the prod image (apache#9814) Closes apache#9810 * For now cloud tools are not needed in CI (apache#9818) Currently there is "unbound" variable error printed in CI logs because of that. * Python 3.8.4 release breaks our builds (apache#9820) * Allow `replace` flag in gcs_to_gcs operator. (apache#9667) * Allow `replace` flag in gcs_to_gcs operator. If we are not replacing, list all files in the Destination GCS bucket and only keep those files which are present in Source GCS bucket and not in Destination GCS bucket * Add kylin operator (apache#9149) Co-authored-by: yongheng.liu <[email protected]> * Fix SqlAlchemy-Flask failure with python 3.8.4 (apache#9821) * Add API Reference docs (redoc) to sphinx (apache#9806) * Add Google Deployment Manager Hook (apache#9159) Co-authored-by: Ephraim Anierobi <[email protected]> * Remove HTTP guide index in docs (apache#9796) * Improve type hinting to provider cloudant (apache#9825) Co-authored-by: Refael Y <[email protected]> * Add option to delete by prefix to S3DeleteObjectsOperator (apache#9350) Co-authored-by: Felix Uellendall <[email protected]> * Add CloudVisionDeleteReferenceImageOperator (apache#9698) * Add note in Updating.md about the change in `run_as_user` default (apache#9822) Until Airflow 1.10.10 the default run_as_user config (https://airflow.readthedocs.io/en/1.10.10/configurations-ref.html#run-as-user) which defaulted it to root user `0` (https://github.com/apache/airflow/blob/96697180d79bfc90f6964a8e99f9dd441789177c/airflow/contrib/executors/kubernetes_executor.py#L295-L301) In Airflow 1.10.11 we changed it to `50000` * Improve typing in airflow/models/pool.py (apache#9835) * Remove global variable with API auth backend (apache#9833) * Fix Writing Serialized Dags to DB (apache#9836) * Update gcp to google in docs (apache#9839) Co-authored-by: Ashwin Shankar <[email protected]> * BigQueryTableExistenceSensor needs to specify keyword arguments (apache#9832) * Add guide for AI Platform (previously Machine Learning Engine) Operators (apache#9798) * Change DAG.clear to take dag_run_state (apache#9824) * Change DAG.clear to take dag_run_state * fix lint * fix tests * assign var * extend original clause * Rename DagBag.store_serialized_dags to Dagbag.read_dags_from_db (apache#9838) * Update more occurrences of gcp to google (apache#9842) * Add Dynata to the Airflow users list (apache#9846) * Fix S3FileTransformOperator to support S3 Select transformation only (apache#8936) Documentation for S3FileTransformOperator states that users can skip transformation script if S3 Select experession is specified, but in this case the created file is always zero bytes long. This fix changes the behaviour, so in case of no transformation given, the source file (a result of S3Select) is uploaded. * Fix DagRun.conf when using trigger_dag API (apache#9853) fixes apache#9852 * Helm chart can now place arbitrary config settings in to airflow.cfg (apache#9816) Rather than only allowing specific pre-determined config settings, this change allows the user to place _any_ config setting they like in the generated airflow.cfg, including overwriting the "generated defaults". This providers a nicer interface for the users of the chart (even if the could already set these via the env vars). * Fix typo in datafusion operator (apache#9859) Co-authored-by: michalslowikowski00 <[email protected]> * Fix Experimental API Client (apache#9849) * Add imagePullSecrets to the create user job (apache#9802) So that it can pull the specified image from a private registry. * Group CI scripts in subdirectories (apache#9653) Reviewed the scripts and removed some of the old unused ones. Co-authored-by: Jarek Potiuk <[email protected]> Co-authored-by: Ephraim Anierobi <[email protected]> Co-authored-by: Kaxil Naik <[email protected]> Co-authored-by: Tomek Urbaszek <[email protected]> Co-authored-by: Aneesh Joseph <[email protected]> Co-authored-by: Dylan Joss <[email protected]> Co-authored-by: QP Hou <[email protected]> Co-authored-by: Cooper Gillan <[email protected]> Co-authored-by: Omair Khan <[email protected]> Co-authored-by: chamcca <[email protected]> Co-authored-by: lindsable <[email protected]> Co-authored-by: Vinay G B <[email protected]> Co-authored-by: Kamil Breguła <[email protected]> Co-authored-by: Vinay <[email protected]> Co-authored-by: Vismita Uppalli <[email protected]> Co-authored-by: Jeff Melching <[email protected]> Co-authored-by: Daniel Debny <[email protected]> Co-authored-by: James Timmins <[email protected]> Co-authored-by: Tomek Urbaszek <[email protected]> Co-authored-by: Morgan Racine <[email protected]> Co-authored-by: Ash Berlin-Taylor <[email protected]> Co-authored-by: Bolgov Andrey <[email protected]> Co-authored-by: Daniel Imberman <[email protected]> Co-authored-by: Daniel Imberman <[email protected]> Co-authored-by: chipmyersjr <[email protected]> Co-authored-by: Jacek Kołodziej <[email protected]> Co-authored-by: Kanthi <[email protected]> Co-authored-by: morrme <[email protected]> Co-authored-by: Nitai Bezerra da Silva <[email protected]> Co-authored-by: Rafferty Chen <[email protected]> Co-authored-by: Mauricio De Diana <[email protected]> Co-authored-by: Guilherme Da Silva Gonçalves <[email protected]> Co-authored-by: takunnithan <[email protected]> Co-authored-by: Chao-Han Tsai <[email protected]> Co-authored-by: Tobiasz Kędzierski <[email protected]> Co-authored-by: Tim Healy <[email protected]> Co-authored-by: Timothy Healy <[email protected]> Co-authored-by: Adam Dobrawy <[email protected]> Co-authored-by: Vicken Simonian <[email protected]> Co-authored-by: Alexander Sutcliffe <[email protected]> Co-authored-by: royberkoweee <[email protected]> Co-authored-by: yongheng.liu <[email protected]> Co-authored-by: yongheng.liu <[email protected]> Co-authored-by: Sam Wheating <[email protected]> Co-authored-by: Ephraim Anierobi <[email protected]> Co-authored-by: rafyzg <[email protected]> Co-authored-by: Refael Y <[email protected]> Co-authored-by: Shoichi Kagawa <[email protected]> Co-authored-by: Felix Uellendall <[email protected]> Co-authored-by: ashwinshankar77 <[email protected]> Co-authored-by: Ashwin Shankar <[email protected]> Co-authored-by: Nathan Hadfield <[email protected]> Co-authored-by: Neil Bhandari <[email protected]> Co-authored-by: Mariusz Strzelecki <[email protected]> Co-authored-by: Michał Słowikowski <[email protected]> Co-authored-by: michalslowikowski00 <[email protected]>

boring-cyborg bot added the provider:Apache label May 28, 2020

aneesh-joseph changed the title ~~do a kinit before yarn kill if keytab and principal is provided~~ do a kinit before yarn kill if keytab and principal are provided May 28, 2020

ashb reviewed May 28, 2020

View reviewed changes

ashb requested changes May 28, 2020

View reviewed changes

aneesh-joseph force-pushed the patch-1 branch from d034acb to 0762dd5 Compare May 28, 2020 13:31

aneesh-joseph force-pushed the patch-1 branch from 0762dd5 to bb6a27e Compare June 5, 2020 18:36

aneesh-joseph requested a review from ashb June 10, 2020 09:49

aneesh-joseph force-pushed the patch-1 branch 2 times, most recently from b708275 to f351211 Compare June 11, 2020 10:43

ashb requested changes Jun 11, 2020

View reviewed changes

airflow/providers/apache/spark/hooks/spark_submit.py Outdated Show resolved Hide resolved

airflow/security/kerberos.py Outdated Show resolved Hide resolved

airflow/security/kerberos.py Outdated Show resolved Hide resolved

aneesh-joseph requested a review from ashb June 15, 2020 15:32

ashb reviewed Jun 17, 2020

View reviewed changes

airflow/providers/apache/spark/hooks/spark_submit.py Outdated Show resolved Hide resolved

aneesh-joseph force-pushed the patch-1 branch from 7aa27db to da34645 Compare June 17, 2020 13:48

aneesh-joseph force-pushed the patch-1 branch from da34645 to bfb5569 Compare June 17, 2020 13:58

ashb changed the title ~~do a kinit before yarn kill if keytab and principal are provided~~ Ensure Kerberos token is valid in SparkSubmitOperator before running yarn kill Jun 17, 2020

ashb approved these changes Jun 17, 2020

View reviewed changes

ashb requested changes Jun 17, 2020

View reviewed changes

aneesh-joseph requested a review from ashb June 18, 2020 10:43

aneesh-joseph force-pushed the patch-1 branch 2 times, most recently from b4d2d13 to a815768 Compare June 25, 2020 18:56

aneesh-joseph force-pushed the patch-1 branch 2 times, most recently from edff08d to 9a2994e Compare July 2, 2020 14:20

ashb approved these changes Jul 2, 2020

View reviewed changes

aneesh-joseph force-pushed the patch-1 branch from 9a2994e to d237e0b Compare July 2, 2020 18:53

aneesh-joseph added 3 commits July 4, 2020 07:42

do a kinit before yarn kill if keytab and principal is provided

e7e6e80

add a test to check if KRB5CCNAME is set

abfeb69

move the new test path into the existing test

8d41a61

aneesh-joseph force-pushed the patch-1 branch 3 times, most recently from 43d8e86 to 5dbfd90 Compare July 4, 2020 07:17

rename mock

2df3b1b

aneesh-joseph force-pushed the patch-1 branch from 5dbfd90 to 2df3b1b Compare July 4, 2020 07:43

ashb merged commit 13a827d into apache:master Jul 9, 2020

aneesh-joseph deleted the patch-1 branch July 9, 2020 09:39

potiuk mentioned this pull request Jul 27, 2020

Release a 2nd wave of Backport packages #10014

Closed

13 tasks

eladkal mentioned this pull request Feb 19, 2022

Infinite wait in SparkSubmitOperator kill #19752

Closed

2 tasks

-                    kinit_cmd = "kinit -kt {} {}" \
-                        .format(self._keytab, self._principal).split()
-                    kinit = subprocess.Popen(kinit_cmd,
-                                             stdout=subprocess.PIPE,
-                                             stderr=subprocess.PIPE)
+                    from airflow.security.kerberos import renew_from_kt
+                    renew_from_kt(self._principal, self._keytab)

Ensure Kerberos token is valid in SparkSubmitOperator before running yarn kill #9044

Ensure Kerberos token is valid in SparkSubmitOperator before running yarn kill #9044

Uh oh!

Conversation

aneesh-joseph commented May 28, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ashb May 28, 2020

Choose a reason for hiding this comment

Uh oh!

ashb May 28, 2020

Choose a reason for hiding this comment

Uh oh!

aneesh-joseph May 28, 2020

Choose a reason for hiding this comment

Uh oh!

aneesh-joseph May 28, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ashb Jun 11, 2020

Choose a reason for hiding this comment

Uh oh!

aneesh-joseph Jun 15, 2020

Choose a reason for hiding this comment

Uh oh!

ashb left a comment

Choose a reason for hiding this comment

Uh oh!

aneesh-joseph commented Jun 6, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aneesh-joseph commented Jun 17, 2020

Uh oh!

ashb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aneesh-joseph commented Jun 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ashb left a comment

Choose a reason for hiding this comment

Uh oh!

ashb left a comment

Choose a reason for hiding this comment

Uh oh!

ashb commented Jun 17, 2020

Uh oh!

aneesh-joseph commented Jun 17, 2020

Uh oh!

aneesh-joseph commented Jun 29, 2020

Uh oh!

ashb left a comment

Choose a reason for hiding this comment

Uh oh!

ashb commented Jul 2, 2020

Uh oh!

aneesh-joseph commented Jul 2, 2020

Uh oh!

aneesh-joseph commented Jul 4, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Ensure Kerberos token is valid in SparkSubmitOperator before running `yarn kill` #9044

Ensure Kerberos token is valid in SparkSubmitOperator before running `yarn kill` #9044

aneesh-joseph commented May 28, 2020 •

edited

Loading

aneesh-joseph May 28, 2020 •

edited

Loading

aneesh-joseph commented Jun 6, 2020 •

edited

Loading

aneesh-joseph commented Jun 17, 2020 •

edited

Loading