Skip to content

Commit d41ac7b

Browse files
potiukamoghrajesh
andauthored
Hide team scoped values of options registered as sensitive (#70755)
* Hide team scoped values of options registered as sensitive Options are registered as sensitive under their base section, but a team scoped override lives in a `[<team>=<section>]` config file section, or in an `AIRFLOW__<TEAM>___<SECTION>__<KEY>` environment variable. Every sensitivity decision was a direct membership test against the registered base pairs, and the masking pass iterated those pairs and looked each section up verbatim, so a team scoped section was never visited and its value was returned in clear. Resolve the team scoped spelling back to the base option before deciding whether a value is sensitive. `team_section_name` now builds the config file section name at both construction sites so the two representations cannot drift, `base_section_name` recovers the base section from it, and `is_sensitive_option` tests a pair directly, then via the base section, then via the tail an environment variable contributes -- so it can only ever recognise more options as sensitive, never fewer. The team name is not parsed out of an environment variable name, because a team name may contain underscores; the name is matched against the tail each registered option contributes instead. The config file section name is split on the last separator, so the base section is recovered even for a team name that contains the separator itself. Four call sites use the predicate: the masking pass, the environment collection, `write`, and the single option config route, which does not go through `as_dict` and so needed its own change. Team scoped `_cmd` and `_secret` entries are hidden in place rather than resolved into their value, because resolving them is not supported for a team. Generated-by: Claude Opus 5 (1M context) following the guidelines at https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions * Add newsfragment for the team scoped sensitive option masking change * Keep one copy of the team scoped masking rationale, and record the untouched gap The per-key secrets-backend options are matched by literal section name and stay unaware of a team scoped spelling. Nothing leaks while the secrets backend itself is not team aware, so it is recorded rather than fixed here. * Update shared/configuration/tests/configuration/test_parser.py Co-authored-by: Amogh Desai <[email protected]> * Update airflow-core/tests/unit/api_fastapi/core_api/routes/public/test_config.py Co-authored-by: Amogh Desai <[email protected]> --------- Co-authored-by: Amogh Desai <[email protected]>
1 parent f814598 commit d41ac7b

6 files changed

Lines changed: 351 additions & 18 deletions

File tree

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
Team scoped values of options registered as sensitive are now hidden
2+
3+
Configuration options are registered as sensitive under their base section, so until now only the
4+
base spelling of an option was masked. A team scoped override -- set in a ``[<team>=<section>]``
5+
config file section, or through an ``AIRFLOW__<TEAM>___<SECTION>__<KEY>`` environment variable --
6+
was not recognised as the same option and was returned in full.
7+
8+
Sensitivity is now decided after resolving the team scoped spelling back to the base option, so a
9+
team scoped value is masked exactly as the base value already was.
10+
11+
**Behaviour changes:**
12+
13+
- ``AirflowConfigParser.as_dict(display_sensitive=False)``, ``GET /config``,
14+
``GET /config/section/{section}/option/{option}`` and ``airflow config list`` now return
15+
``< hidden >`` for a team scoped value of an option registered as sensitive. Deployments that
16+
read a team's real value through any of these will now receive the mask; use
17+
``display_sensitive=True`` where a real value is required and appropriate.
18+
- Team scoped ``_cmd`` and ``_secret`` entries are replaced with ``< hidden >`` in place, rather
19+
than being resolved into their value and removed as they are in a base section. Resolving them
20+
is not supported for a team, so the command string or secret path is no longer shown either.
21+
- Non team configuration is unaffected, and ``display_sensitive=True`` continues to return real
22+
values.
23+
24+
**New public helpers on the shared configuration parser** (additive; no signatures changed):
25+
26+
- ``team_section_name(team_name, section)`` builds the team scoped config file section name.
27+
- ``base_section_name(section)`` recovers the base section from a possibly team scoped one.
28+
- ``AirflowConfigParser.is_sensitive_option(section, key)`` reports whether an option is
29+
registered as sensitive under any of its spellings.

‎airflow-core/src/airflow/api_fastapi/core_api/routes/public/config.py‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -153,9 +153,7 @@ def get_config_value(
153153
)
154154

155155
section_l, option_l = section.lower(), option.lower()
156-
if (section_l, option_l) in conf.sensitive_config_values or _is_per_key_sensitive_option(
157-
section_l, option_l
158-
):
156+
if conf.is_sensitive_option(section_l, option_l) or _is_per_key_sensitive_option(section_l, option_l):
159157
value = "< hidden >"
160158
else:
161159
value = conf.get(section, option)

‎airflow-core/src/airflow/api_fastapi/core_api/services/public/config.py‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,11 @@
3131
# (e.g. Vault role_id / secret_id) as the registered ``backend_kwargs``
3232
# option, so they need the same redaction treatment when
3333
# ``display_sensitive=False``.
34+
# These match literal section names only, so a team scoped spelling of the same option -- the
35+
# ``[<team>=secrets]`` section, or ``AIRFLOW__<TEAM>___SECRETS__BACKEND_KWARG__*``, which is
36+
# reported under a section named after the team -- is not recognised. Nothing leaks today because
37+
# the secrets backend is not team aware; tracked at
38+
# https://github.com/apache/airflow/issues/71037
3439
_PER_KEY_SENSITIVE_PREFIXES: dict[str, str] = {
3540
"secrets": "backend_kwarg__",
3641
"workers": "secrets_backend_kwarg__",

‎airflow-core/tests/unit/api_fastapi/core_api/routes/public/test_config.py‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,8 @@
2222

2323
import pytest
2424

25+
from airflow.configuration import conf
26+
2527
from tests_common.test_utils.config import conf_vars
2628

2729
pytestmark = pytest.mark.db_test
@@ -559,3 +561,65 @@ def test_get_config_value_masks_per_key_secrets_backend_kwarg(self, test_client)
559561
)
560562
assert response.status_code == 200
561563
assert response.json()["sections"][0]["options"][0]["value"] == OPTION_VALUE_SENSITIVE_HIDDEN
564+
565+
566+
TEAM_NAME = "team_a"
567+
SECTION_TEAM_DATABASE = f"{TEAM_NAME}={SECTION_DATABASE}"
568+
SECTION_TEAM_CORE = f"{TEAM_NAME}={SECTION_CORE}"
569+
OPTION_VALUE_TEAM_SQL_ALCHEMY_CONN = "postgresql://team_a_user:team_a_password@team-a-db/airflow"
570+
OPTION_VALUE_TEAM_PARALLELISM = "512"
571+
572+
573+
class TestTeamScopedOptionMasking(TestConfigEndpoint):
574+
"""A team scoped override of a sensitive option lives in a ``[<team>=<section>]`` section rather
575+
than in the section the option is registered under. The single-option endpoint has to resolve
576+
that spelling back to the registered option, so that it redacts a team scoped value exactly as
577+
it redacts the base one."""
578+
579+
@pytest.fixture(autouse=True)
580+
def setup_team_scoped(self) -> Generator[None, None, None]:
581+
team_config = {
582+
SECTION_TEAM_DATABASE: {OPTION_KEY_SQL_ALCHEMY_CONN: OPTION_VALUE_TEAM_SQL_ALCHEMY_CONN},
583+
SECTION_TEAM_CORE: {OPTION_KEY_PARALLELISM: OPTION_VALUE_TEAM_PARALLELISM},
584+
}
585+
586+
# Anything that is not one of the team scoped options under test keeps being read from the
587+
# real configuration - the endpoint reads [api] expose_config through the same object.
588+
real_get = conf.get
589+
590+
def _mock_has_option(section: str, option: str, **_) -> bool:
591+
return option in team_config.get(section, {})
592+
593+
def _mock_get(section: str, option: str, *args, **kwargs) -> str:
594+
if option in team_config.get(section, {}):
595+
return team_config[section][option]
596+
return real_get(section, option, *args, **kwargs)
597+
598+
with (
599+
conf_vars(AIRFLOW_CONFIG_ENABLE_EXPOSE_CONFIG),
600+
patch(
601+
"airflow.api_fastapi.core_api.routes.public.config.conf.has_option",
602+
new=_mock_has_option,
603+
),
604+
patch(
605+
"airflow.api_fastapi.core_api.routes.public.config.conf.get",
606+
new=_mock_get,
607+
),
608+
):
609+
yield
610+
611+
def test_get_config_value_masks_team_scoped_sensitive_option(self, test_client):
612+
response = test_client.get(
613+
f"/config/section/{SECTION_TEAM_DATABASE}/option/{OPTION_KEY_SQL_ALCHEMY_CONN}",
614+
headers=HEADERS_JSON,
615+
)
616+
assert response.status_code == 200
617+
assert response.json()["sections"][0]["options"][0]["value"] == OPTION_VALUE_SENSITIVE_HIDDEN
618+
619+
def test_get_config_value_keeps_team_scoped_non_sensitive_option(self, test_client):
620+
response = test_client.get(
621+
f"/config/section/{SECTION_TEAM_CORE}/option/{OPTION_KEY_PARALLELISM}",
622+
headers=HEADERS_JSON,
623+
)
624+
assert response.status_code == 200
625+
assert response.json()["sections"][0]["options"][0]["value"] == OPTION_VALUE_TEAM_PARALLELISM

‎shared/configuration/src/airflow_shared/configuration/parser.py‎

Lines changed: 109 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -77,6 +77,34 @@ def _collect_kwarg_env_vars(prefix: str) -> dict[str, str]:
7777
ConfigSectionSourcesType = dict[str, str | tuple[str, str]]
7878
ConfigSourcesType = dict[str, ConfigSectionSourcesType]
7979
ENV_VAR_PREFIX = "AIRFLOW__"
80+
# Separates the team name from the base section name in a team scoped config file section.
81+
TEAM_SECTION_SEPARATOR = "="
82+
83+
84+
def team_section_name(team_name: str, section: str) -> str:
85+
"""
86+
Build the config file section name that holds the team scoped overrides of ``section``.
87+
88+
:param team_name: name of the team the overrides belong to
89+
:param section: base section name that is being overridden
90+
:return: the team scoped section name, e.g. ``team_a=celery``
91+
"""
92+
return f"{team_name}{TEAM_SECTION_SEPARATOR}{section}"
93+
94+
95+
def base_section_name(section: str) -> str:
96+
"""
97+
Return the base section name of a possibly team scoped config file section.
98+
99+
Team scoped sections are built by :func:`team_section_name`. Base section names never contain
100+
the separator, so the name is split on the last one - that way the base section is recovered
101+
even for a team name that contains the separator itself.
102+
103+
:param section: section name, either a base one or a team scoped one
104+
:return: the base section name, which is ``section`` itself when it is not team scoped
105+
"""
106+
_, separator, base_section = section.rpartition(TEAM_SECTION_SEPARATOR)
107+
return base_section if separator else section
80108

81109

82110
if TYPE_CHECKING:
@@ -562,6 +590,68 @@ def sensitive_config_values(self) -> set[tuple[str, str]]:
562590
sensitive.update(depr_section, depr_option)
563591
return sensitive
564592

593+
def _names_sensitive_team_env_var(self, env_var: str) -> bool:
594+
"""
595+
Check whether an environment variable name is a team scoped override of a sensitive option.
596+
597+
Team scoped variables are named ``AIRFLOW__<TEAM>___<SECTION>__<KEY>`` (see
598+
:meth:`_env_var_name`). A team name may contain underscores itself, so the team name is not
599+
parsed out of the variable name; the name is matched against the tail that each option
600+
registered as sensitive contributes instead. The ``_CMD`` / ``_SECRET`` fallbacks are
601+
matched too, because - unlike for a base section - they are never resolved into their value.
602+
603+
:param env_var: environment variable name
604+
:return: True if the variable holds a team scoped value of an option registered as sensitive
605+
"""
606+
env_var = env_var.upper()
607+
if not env_var.startswith(ENV_VAR_PREFIX):
608+
return False
609+
# Every tail matched below starts with the ``___`` separating the team name from the
610+
# section, so a name not containing it cannot be a team scoped one.
611+
if "___" not in env_var:
612+
return False
613+
for section, key in self.sensitive_config_values:
614+
option_tail = self._env_var_name(section, key).removeprefix(ENV_VAR_PREFIX)
615+
for tail in (f"___{option_tail}", f"___{option_tail}_CMD", f"___{option_tail}_SECRET"):
616+
# The team name sits between the prefix and the tail, so it must not be empty.
617+
if env_var.endswith(tail) and len(env_var) > len(ENV_VAR_PREFIX) + len(tail):
618+
return True
619+
return False
620+
621+
def is_sensitive_option(self, section: str, key: str) -> bool:
622+
"""
623+
Check whether the value of ``key`` in ``section`` is registered as sensitive.
624+
625+
Options are registered as sensitive under their base section name, while a team scoped
626+
override of the very same option is held by a ``<team name>=<section>`` config file section
627+
or by an ``AIRFLOW__<TEAM>___<SECTION>__<KEY>`` environment variable. Both of those
628+
spellings are resolved back to the base option here, so that a team scoped value is treated
629+
exactly like the base one. A name that does not resolve to a registered option is not
630+
sensitive - so this only ever recognises more options as sensitive, never fewer.
631+
632+
:param section: section name, either a base one or a team scoped one
633+
:param key: option name
634+
:return: True if the value of the option should be treated as sensitive
635+
"""
636+
section = section.lower()
637+
key = key.lower()
638+
if (section, key) in self.sensitive_config_values:
639+
return True
640+
base_section = base_section_name(section)
641+
if base_section != section:
642+
if (base_section, key) in self.sensitive_config_values:
643+
return True
644+
# A team scoped ``_cmd`` / ``_secret`` fallback is not resolved into its value, so it
645+
# stays in the output as configured and has to be recognised on its own.
646+
for fallback_suffix in ("_cmd", "_secret"):
647+
if not key.endswith(fallback_suffix):
648+
continue
649+
if (base_section, key.removesuffix(fallback_suffix)) in self.sensitive_config_values:
650+
return True
651+
# A team scoped environment variable is reported under the section and key its name splits
652+
# into, which is neither the base nor the team scoped section name.
653+
return self._names_sensitive_team_env_var(self._env_var_name(section, key))
654+
565655
def _update_defaults_from_string(self, config_string: str) -> None:
566656
"""
567657
Update the defaults in _default_values based on values in config_string ("ini" format).
@@ -788,8 +878,11 @@ def _include_envs(
788878
log.warning("Ignoring unknown env var '%s'", env_var)
789879
continue
790880
if not display_sensitive and env_var != self._env_var_name("core", "unit_test_mode"):
881+
if self._names_sensitive_team_env_var(env_var):
882+
# Covers the cmd/secret variants too; see is_sensitive_option.
883+
opt = "< hidden >"
791884
# Don't hide cmd/secret values here
792-
if not env_var.lower().endswith(("cmd", "secret")):
885+
elif not env_var.lower().endswith(("cmd", "secret")):
793886
if (section, key) in self.sensitive_config_values:
794887
opt = "< hidden >"
795888
elif raw:
@@ -1121,7 +1214,7 @@ def _get_option_from_config_file(
11211214
) -> str | ValueNotFound:
11221215
"""Get config option from config file."""
11231216
if team_name := kwargs.get("team_name", None):
1124-
section = f"{team_name}={section}"
1217+
section = team_section_name(team_name, section)
11251218
# since this is the last lookup that supports team_name, pop it
11261219
kwargs.pop("team_name")
11271220
if super().has_option(section, key):
@@ -1792,15 +1885,19 @@ def as_dict(
17921885
if not display_sensitive:
17931886
# This ensures the ones from config file is hidden too
17941887
# if they are not provided through env, cmd and secret
1888+
# The collected options are walked (rather than the registered sensitive ones) so that
1889+
# team scoped sections are covered as well - they are named after the team, not after
1890+
# the base section the option is registered under.
17951891
hidden = "< hidden >"
1796-
for section, key in self.sensitive_config_values:
1797-
if config_sources.get(section):
1798-
if config_sources[section].get(key, None):
1799-
if display_source:
1800-
source = config_sources[section][key][1]
1801-
config_sources[section][key] = (hidden, source)
1802-
else:
1803-
config_sources[section][key] = hidden
1892+
for section, options in config_sources.items():
1893+
for key, value in list(options.items()):
1894+
if not value or not self.is_sensitive_option(section, key):
1895+
continue
1896+
if display_source:
1897+
source = value[1]
1898+
options[key] = (hidden, source)
1899+
else:
1900+
options[key] = hidden
18041901

18051902
return config_sources
18061903

@@ -1877,7 +1974,7 @@ def getsection(self, section: str, team_name: str | None = None) -> ConfigOption
18771974
:param team_name: optional team name for team-specific configuration lookup
18781975
"""
18791976
# Handle team-specific section lookup for config file
1880-
config_section = f"{team_name}={section}" if team_name else section
1977+
config_section = team_section_name(team_name, section) if team_name else section
18811978

18821979
if not self.has_section(config_section) and not self._has_section_in_any_defaults(config_section):
18831980
return None
@@ -2032,10 +2129,7 @@ def write( # type: ignore[override]
20322129
section_to_write=section_to_write,
20332130
sources_dict=sources_dict,
20342131
)
2035-
is_sensitive = (
2036-
section_to_write.lower(),
2037-
option.lower(),
2038-
) in self.sensitive_config_values
2132+
is_sensitive = self.is_sensitive_option(section_to_write, option)
20392133
self._write_value(
20402134
file=file,
20412135
option=option,

0 commit comments

Comments
 (0)