Skip to content

Making webserver config customisable #29926

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
potiuk merged 30 commits into from
May 18, 2023
Merged

Making webserver config customisable #29926

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
a6c5ee9
Making webserver config customizable
Mar 5, 2023
77802f7
Merge branch 'main' into webServerConfigurable
Mar 5, 2023
26fc78e
Adding unit tests
Mar 5, 2023
b85cd0c
Fixing staticcheck issues
Mar 5, 2023
4ce74b9
Adding deprecation warning
Mar 5, 2023
5b73b18
Fixing failing tests
Mar 5, 2023
2152dea
Making webserver config customizable
Mar 5, 2023
5edfd1d
Adding unit tests
Mar 5, 2023
68e422b
Fixing staticcheck issues
Mar 5, 2023
ba81ae6
Adding deprecation warning
Mar 5, 2023
0f6b784
Fixing failing tests
Mar 5, 2023
cddd581
Merge branch 'webServerConfigurable' of github.com:amoghrajesh/airflo…
Mar 12, 2023
acd291e
Adding documentation changes
Mar 12, 2023
13f1e9c
Correcting bug in the code
Mar 13, 2023
62bbcce
Using os.path.join
Mar 13, 2023
26d07ba
Adding notes as newsfragments
Mar 14, 2023
131d2c2
Fixing static checks
Mar 14, 2023
6d3f701
Merge branch 'main' into webServerConfigurable
uranusjr Mar 14, 2023
5882ee9
Reworking the PR as per Jed's comments
Mar 20, 2023
2102599
Merge branch 'webServerConfigurable' of github.com:amoghrajesh/airflo…
Mar 20, 2023
02685c8
Merge branch 'main' into webServerConfigurable
amoghrajesh Mar 20, 2023
98ace93
Handling review comments from Jed
Mar 23, 2023
2b271f6
Merge branch 'webServerConfigurable' of github.com:amoghrajesh/airflo…
Mar 23, 2023
3be8ca0
Handling review comments from Jed
Mar 27, 2023
86048bd
removing unwanted import
Mar 27, 2023
4305a48
Addressing review comments from Jed
May 2, 2023
4b0ca76
Merge branch 'main' into webServerConfigurable
amoghrajesh May 14, 2023
41052e3
Fix test
jedcunningham May 17, 2023
5184664
nits from Jed
May 17, 2023
2993bff
Merge branch 'webServerConfigurable' of github.com:amoghrajesh/airflo…
May 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions airflow/config_templates/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1334,6 +1334,13 @@ hive:
webserver:
description: ~
options:
config_file:
description: |
Path of webserver config file used for configuring the webserver parameters
version_added: 2.7.0
type: string
example: ~
default: "{AIRFLOW_HOME}/webserver_config.py"
base_url:
description: |
The base url of your website as airflow cannot guess what domain or
Expand Down
3 changes: 3 additions & 0 deletions airflow/config_templates/default_airflow.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -705,6 +705,9 @@ default_hive_mapred_queue =
# mapred_job_name_template =

[webserver]
# Path of webserver config file used for configuring the webserver parameters
config_file = {AIRFLOW_HOME}/webserver_config.py

# The base url of your website as airflow cannot guess what domain or
# cname you are using. This is used in automated emails that
# airflow sends to point links to the right web server
Expand Down
9 changes: 2 additions & 7 deletions airflow/configuration.py
View file
Open in desktop
Comment thread
jedcunningham marked this conversation as resolved.
Original file line number Diff line number Diff line change
Expand Up @@ -1483,7 +1483,7 @@ def initialize_config() -> AirflowConfigParser:

Called for you automatically as part of the Airflow boot process.
"""
global FERNET_KEY, AIRFLOW_HOME
global FERNET_KEY, AIRFLOW_HOME, WEBSERVER_CONFIG

default_config = _parameterized_config_from_template("default_airflow.cfg")

Expand Down Expand Up @@ -1548,10 +1548,7 @@ def initialize_config() -> AirflowConfigParser:
if local_conf.getboolean("core", "unit_test_mode"):
local_conf.load_test_config()

# Make it no longer a proxy variable, just set it to an actual string
global WEBSERVER_CONFIG
WEBSERVER_CONFIG = AIRFLOW_HOME + "/webserver_config.py"

WEBSERVER_CONFIG = local_conf.get("webserver", "config_file")

@potiuk potiuk May 17, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs formatting, but I think we need:

Suggested change
WEBSERVER_CONFIG = local_conf.get("webserver", "config_file")
WEBSERVER_CONFIG = local_conf.get("webserver", "config_file",
fallback= AIRFLOW_HOME + "/webserver_config.py")

Otherwise it will fail for those who have the old config file

@potiuk potiuk May 17, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(or is it aautomaticaly handled with the default?

@jedcunningham jedcunningham May 17, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's automatically handled. Most fallbacks can go honestly, much more useful in providers than core.

@potiuk potiuk May 17, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I vaguely recall that - I wanted to clean up the code doing it once and failed miserably :)

@amoghrajesh amoghrajesh May 18, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for clarifying this. Was not so aware about this

if not os.path.isfile(WEBSERVER_CONFIG):
import shutil

Expand Down Expand Up @@ -1786,7 +1783,6 @@ def __getattr__(name):
AIRFLOW_HOME = get_airflow_home()
AIRFLOW_CONFIG = get_airflow_config(AIRFLOW_HOME)


# Set up dags folder for unit tests
# this directory won't exist if users install via pip
_TEST_DAGS_FOLDER = os.path.join(
Expand All @@ -1806,7 +1802,6 @@ def __getattr__(name):
else:
TEST_PLUGINS_FOLDER = os.path.join(AIRFLOW_HOME, "plugins")


TEST_CONFIG_FILE = get_airflow_test_config(AIRFLOW_HOME)

SECRET_KEY = b64encode(os.urandom(16)).decode("utf-8")
Expand Down
12 changes: 12 additions & 0 deletions tests/www/views/test_views.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,18 @@ def test_configuration_expose_config(admin_client):
check_content_in_response(["Airflow Configuration"], resp)


@mock.patch("airflow.configuration.WEBSERVER_CONFIG")
def test_webserver_configuration_config_file(mock_webserver_config_global, admin_client, tmp_path):
import airflow.configuration

config_file = str(tmp_path / "my_custom_webserver_config.py")
with mock.patch.dict(os.environ, {"AIRFLOW__WEBSERVER__CONFIG_FILE": config_file}):
initialize_config()
assert airflow.configuration.WEBSERVER_CONFIG == config_file

assert os.path.isfile(config_file)


def test_redoc_should_render_template(capture_templates, admin_client):
from airflow.utils.docs import get_docs_url

Expand Down