You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Airflow Logging Configs are ignored / not used in Airflow 3.x #53442
Despite all of the documentation on how to configure logging in Airflow 3, none of these configs are used and instead Airflow uses a hard-coded setting.
Specifically
rdjouder3 weeks ago
Hello,
Looking at the code of airflow.sdk.log.configure_logging I see that conf.get("logging","logging_config_class", ... ) is not called.
Also logging.config.dictConfig is called with hard coded defaults thus completly erasing what configure_logging() from logging_config.py is doing.
I have a custom logging config that is working fine as of airflow 2.10.X and older.
Is my analysis wrong ? Is it a bug ?
ashb3 weeks ago
It's intentional I'm afraid, and I am in the process of completely overhauling all of Airflow's logging to use structlog and to be able to produce JSON logs in this PR #52651
ashb3 weeks ago
The logger is configured but Airflow won't use it anymore. The reason is that logs from the task process is sent as JSON to the supervisor process and written to file, and the UI then takes that JSON and renders it for display.
What you think should happen instead?
Airflow log configs should be used.
Anything else?
If I understand correctly, it will be resolved with #52651.
Creating this issue just for tracking 3.x blockers. Please add affected_version_3.0 label.
Adding line number & file name to the format before msg
Moving source to the start of line/before msg instead of at the end of the line/after msg
Eventually further debugging to see if I can figure out why running DBT via bashOperator logs so much differently in v3 than it does in v2. (in v2 it logs once per sql statement whereas in v3 it logs once per line which floods the logs with a header for every line; I suspect this is due to some underlying changes and may not be avoidable but the user complaints I've been subject to dictate I at least try and look).
As for the display of task logs the big feedback item from myself and from my users is that select/copy is much harder in v3 than v2. Also scrolling is much worse as it gets very slow after several hundred lines & page up/down don't work properly anymore. (This is almost certainly outside of what can be controlled with logging configs but is something I've very interested in sharing.
Those are the big items IMO, FWIW I opened a slack thread when I first stumbled on this stuff. Eventually I found the above linked discussion which explained some of my 2 -> 3 logging pain hence me opening this issue. Sorry for the wall of text hopefully that helps.
@r-richmond All of those are about viewing the task logs, correct?
The reason I ask is that in #52651 I'm re-working all the logging and config, but display and fields of Task logs specifically will mostly need to be handled in the UI
Yes this is all directed at task logs. No issues with audit logs and others at the moment.
Edit: actually can you clarify? I'm assuming that once you implement the above fix for the configs the logging displayed in the task logs will respect those configs (i.e things like 1&2 above not 3 or 4). Or is there another layer that needs be done for the format of the logs in the task log viewer to be updated?
No -- my change linked above will not change the display of task logs. It might add back in line number and filename back, but the log_fmt won't change the display of task logs in the UI.
making the Supervisor pluggable via a stable interface or entrypoint
Making the whole thing pluggable? no. Absolutely not.
Making the remote logger IO have some way of flagging "don't write logs on disk", sure, that could be added.
What we'd really like is the ability to swap in our own Supervisor implementation (e.g. a DatadogSupervisor) that consumes the structured task JSON events, enriches them with tags/trace IDs, and forwards them directly to our pipeline — without touching disk.
@Abdel39 You should be able to do all of that with the RemoteLoggerIO interface today -- make sure you define processors on your logging class. It will still create the file on disk, but if you end your processor with a raise DropEvent nothing will get written to that file.
I would second that the ability to completely disable writing task logs to disk would be immensely useful.
Making the remote logger IO have some way of flagging "don't write logs on disk", sure, that could be added.
You should be able to do all of that with the RemoteLoggerIO interface today
@ashb is there up-to-date documentation on this (and more generally docs on configuring the new Airflow structlog setup as of 3.1.0)? I have a rough idea of what you're describing in the linked code snippets you provided, but I don't see any mention of the RemoteLoggerIO class in the current Airflow 3.1.0 docs. Can that interface be considered part of the public Airflow interface?
Is there an issue with the LOGGING_CONFIG override?
Based on the advanced logging documentation I have been trying to change the logging levels for specific loggers using a config.py file, however, it doesn't seem to be working.
I would also very much like to be able to add some custom log processor.
My use case, that was working fine with Airflow 2, by adding a custom handler: I want to collect task log lines and periodically write them in a chunk to a custom log file.
I've done this on Airflow 2, because 1) I want to be able to see these log chunks as multiline log entries in Splunk, and 2) because the otel collector we use, with a modest amount of CPU and memory, chokes when it's configured to monitor all the individual standard task logs under AIRFLOW_HOME/logs/dag_id=WHATEVER/task_id=WHATEVER/*.log .
I don't see any way to do this in Airflow 3. Perhaps there is, I don't know. Can we somehow just inject an additional processor to the structlog processors for all task logs?
Edit: I'd also like to know how to add fields like dag_id, task_id, and run_id to the task log records.
Reacted by arvinderpalahaf1w, Adam Melicher and Zach
hi, sorry for maybe non-exactly related question, but what has been changed in airflow 3+ that prevents "search on this page" functionality (ctrl+f) from finding results on task logs view in a case of pretty large logs (5000+)?
i am asking because that was very common scenario in our team to open logs for failed task run, search for specific line with 'ctrl+f' there and then compare it with the same/similar lines of previous runs.
now to investigate task fails, users have to download logs file for multiple task runs, that not very convenient.
could you, please, clarify was it intended and if there some workaround to search through the logs via airflow ui ?
ps: thanks for all your work and progress that you have been done with airflow 3+
Yes. The question is unrelated, you will be better of asking it as a discussion maybe. loosely related questions and comments make it super difficult for people to focus on things and get you answers
I want to:
(1) set the airflow.task logger to DEBUG level by default (without having to change thelogging_level airflow config from INFO to DEBUG
(2) add some DAG params to the logging format (for example by modifying the dictConfig).
What is the recommended way to do this in Airflow 3.x?
Reacted by Abramovich Alex, Roman Parkhunovskyi, Amir Kedem and mbraun-credibly
As far as I understand it, in Airflow 3.1.6 logging works like that:
in task:
we make logs
logs get passed to supervisor
in most of airflow itself (supervisor, etc):
we make logs OR they get passed from task
they get passed to structlog bound logger
to bound logger's wrapper AirflowBoundLoggerFilteringAt<...>
to wrapper's logger factory's logger: NamedBytesLogger or NamedWriteLogger
to logger's output: stdout or log file
in api-server:
we make logs
logs get passed to ???
So, what we have to do is:
disable output to files (because I wanted to only use stdout and custom handlers a-la Airflow 2)
replace logger_factory with one that returns stdlib logging loggers, when we are not just sending JSONs to another parts of Airflow
modify airflow.sdk.log.configure_logging() to load custom logging config too just for triggerer and worker (most airflow components load it already via airflow.logging_config.configure_logging(), but some do not, for unknown reasons)
(in progress) deal with:
(optionally) allow all logs to be JSON and support for stdlib logging's records' .msg field to be a JSON
custom logging handlers' set_context() method not being called
I'm currently on the last step, but I'm not sure if I would be allowed to post the final code here, despite it being only a crutch honestly.
P.S.:
I'm fine with contributors making changes that break providers' functionality, but, IMO, either providers' docs should be updated (which is kinda hard without the main contributor's involvement), or providers should explicitly declare that they are incompatible with the last "Stable" version of Airflow. As of now it's just confusing.
Most logging providers' handlers currently require for the set_context() method to be called, to pass dag and task information to it. But:
Some contributor or maintainer (can't remember who it was exactly now) has mentioned previously that set_context() method is not supposed to be called by Airflow 3
Some handlers, like ElasticsearchTaskHandler, could also explicitly try to access Airflow database, which would fail, due to direct database access being against Airflow 3 architectural principles
Now, the dag and task information is accessible from a task itself by calling a airflow.sdk.get_current_context() method, and it is possible to create a structlogprocessor that would call it and enrich log JSONs. But:
You can only enrich a structlog log like that, and while your code could probably be modified to use it, you would also need to modify your requirements' logging too, and it does not look easily done to me
Theoretically, you could enrich all logs passed from a task to supervisor, but that would require for supervisor.process_log_messages_from_subprocess() generator method to get an ActivitySubprocess.ti as an optional argument
I'll try the last option listed here, as it seems both viable and consistent with Airflow 3. It would rely on all structlog logs being JSON and logginghandlers' emit() method supporting record.msg being a JSON, though.
UPD:
enriching task logs works (added a subprocess_wrapper: ActivitySubprocess | WatchedSubprocess parameter)
ended up creating a custom descendant of logging.StreamHandler and using it in three cases:
airflow.task logger
flask_appbuilder logger
root logger (you should change it in airflow itself, meaning /airflow/_shared/logging/structlog.py and /airflow/sdk/_shared/logging/structlog.py, because it overrides the value you set in logging config)
Activity
[-]Airflow Logging Configs are not Used[/-][+]Airflow Logging Configs are ignored / not used in Airflow 3.x[/+]ashb commentedon Sep 1, 2025
@r-richmond Are you most interested in the logging format for the daemons/airflow components, or for the display of task logs in the UI?
r-richmond commentedon Sep 2, 2025
@ashb Honestly I'm interested in several things
Those are the big items IMO, FWIW I opened a slack thread when I first stumbled on this stuff. Eventually I found the above linked discussion which explained some of my 2 -> 3 logging pain hence me opening this issue. Sorry for the wall of text hopefully that helps.
ashb commentedon Sep 2, 2025
@r-richmond All of those are about viewing the task logs, correct?
The reason I ask is that in #52651 I'm re-working all the logging and config, but display and fields of Task logs specifically will mostly need to be handled in the UI
r-richmond commentedon Sep 2, 2025
Yes this is all directed at task logs. No issues with audit logs and others at the moment.
Edit: actually can you clarify? I'm assuming that once you implement the above fix for the configs the logging displayed in the task logs will respect those configs (i.e things like 1&2 above not 3 or 4). Or is there another layer that needs be done for the format of the logs in the task log viewer to be updated?
ashb commentedon Sep 2, 2025
No -- my change linked above will not change the display of task logs. It might add back in line number and filename back, but the
log_fmtwon't change the display of task logs in the UI.10 remaining items
ashb commentedon Oct 3, 2025
Making the whole thing pluggable? no. Absolutely not.
Making the remote logger IO have some way of flagging "don't write logs on disk", sure, that could be added.
@Abdel39 You should be able to do all of that with the RemoteLoggerIO interface today -- make sure you define
processorson your logging class. It will still create the file on disk, but if you end your processor with araise DropEventnothing will get written to that file.airflow/airflow-core/src/airflow/logging/remote.py
Lines 33 to 41 in 6583b6d
and an example processor (it doesn't drop, but you could)
airflow/providers/amazon/src/airflow/providers/amazon/aws/log/cloudwatch_task_handler.py
Lines 124 to 146 in 6583b6d
You are a bit late on that front, 3.1.0 was released two weeks ago... 😃
zach-overflow commentedon Oct 3, 2025
I would second that the ability to completely disable writing task logs to disk would be immensely useful.
@ashb is there up-to-date documentation on this (and more generally docs on configuring the new Airflow structlog setup as of 3.1.0)? I have a rough idea of what you're describing in the linked code snippets you provided, but I don't see any mention of the
RemoteLoggerIOclass in the current Airflow 3.1.0 docs. Can that interface be considered part of the public Airflow interface?ashb commentedon Oct 3, 2025
No, nothing outside of the code currently, sorry.
Dhunsheth commentedon Oct 10, 2025
Is there an issue with the LOGGING_CONFIG override?
Based on the advanced logging documentation I have been trying to change the logging levels for specific loggers using a config.py file, however, it doesn't seem to be working.
I have documented the full issue here: #56377
Is this an issue with my setup specifically or a global issue as a result of the logging overhaul?
brki commentedon Nov 4, 2025
I would also very much like to be able to add some custom log processor.
My use case, that was working fine with Airflow 2, by adding a custom handler: I want to collect task log lines and periodically write them in a chunk to a custom log file.
I've done this on Airflow 2, because 1) I want to be able to see these log chunks as multiline log entries in Splunk, and 2) because the otel collector we use, with a modest amount of CPU and memory, chokes when it's configured to monitor all the individual standard task logs under AIRFLOW_HOME/logs/dag_id=WHATEVER/task_id=WHATEVER/*.log .
I don't see any way to do this in Airflow 3. Perhaps there is, I don't know. Can we somehow just inject an additional processor to the structlog processors for all task logs?
Edit: I'd also like to know how to add fields like dag_id, task_id, and run_id to the task log records.
samush commentedon Nov 24, 2025
hi, sorry for maybe non-exactly related question, but what has been changed in airflow 3+ that prevents "search on this page" functionality (ctrl+f) from finding results on task logs view in a case of pretty large logs (5000+)?
i am asking because that was very common scenario in our team to open logs for failed task run, search for specific line with 'ctrl+f' there and then compare it with the same/similar lines of previous runs.
now to investigate task fails, users have to download logs file for multiple task runs, that not very convenient.
could you, please, clarify was it intended and if there some workaround to search through the logs via airflow ui ?
ps: thanks for all your work and progress that you have been done with airflow 3+
potiuk commentedon Nov 24, 2025
Yes. The question is unrelated, you will be better of asking it as a discussion maybe. loosely related questions and comments make it super difficult for people to focus on things and get you answers
samush commentedon Nov 24, 2025
got it, thanks
tonytrill commentedon Dec 10, 2025
Currently running 3.1.3 and running into the same issue as the original issue listed.
pam-emtelligent commentedon Jan 25, 2026
Currently running 3.1.6 and this is still happening - cannot modify log configs using. logging_class_path config
This documentation is stale:
https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/advanced-logging-configuration.html#create-a-custom-logging-class
I want to:
(1) set the
airflow.tasklogger to DEBUG level by default (without having to change thelogging_levelairflow config from INFO to DEBUG(2) add some DAG params to the logging format (for example by modifying the dictConfig).
What is the recommended way to do this in Airflow 3.x?
DartVeDroid commentedon Feb 1, 2026
As far as I understand it, in Airflow 3.1.6 logging works like that:
task:supervisorsupervisor, etc):taskstructlogbound loggerAirflowBoundLoggerFilteringAt<...>NamedBytesLoggerorNamedWriteLoggerSo, what we have to do is:
logger_factorywith one that returns stdlibloggingloggers, when we are not just sending JSONs to another parts of Airflowairflow.sdk.log.configure_logging()to load custom logging config too just fortriggererandworker(most airflow components load it already viaairflow.logging_config.configure_logging(), but some do not, for unknown reasons)logging's records'.msgfield to be a JSONset_context()method not being calledI'm currently on the last step, but I'm not sure if I would be allowed to post the final code here, despite it being only a crutch honestly.
P.S.:
I'm fine with contributors making changes that break
providers' functionality, but, IMO, eitherproviders' docs should be updated (which is kinda hard without the main contributor's involvement), or providers should explicitly declare that they are incompatible with the last "Stable" version of Airflow. As of now it's just confusing.DartVeDroid commentedon Feb 4, 2026
Welp, it's weird:
set_context()method to be called, to passdagandtaskinformation to it. But:set_context()method is not supposed to be called by Airflow 3ElasticsearchTaskHandler, could also explicitly try to access Airflow database, which would fail, due to direct database access being against Airflow 3 architectural principlesdagandtaskinformation is accessible from a task itself by calling aairflow.sdk.get_current_context()method, and it is possible to create astructlogprocessorthat would call it and enrich log JSONs. But:structloglog like that, and while your code could probably be modified to use it, you would also need to modify your requirements' logging too, and it does not look easily done to metasktosupervisor, but that would require forsupervisor.process_log_messages_from_subprocess()generator method to get anActivitySubprocess.tias an optional argumentI'll try the last option listed here, as it seems both viable and consistent with Airflow 3.
It would rely on all
structloglogs being JSON andlogginghandlers'emit()method supportingrecord.msgbeing a JSON, though.UPD:
tasklogs works (added asubprocess_wrapper: ActivitySubprocess | WatchedSubprocessparameter)logging.StreamHandlerand using it in three cases:airflow.taskloggerflask_appbuilderloggerrootlogger (you should change it in airflow itself, meaning/airflow/_shared/logging/structlog.pyand/airflow/sdk/_shared/logging/structlog.py, because it overrides the value you set in logging config)