Skip to content

Airflow Logging Configs are ignored / not used in Airflow 3.x #53442

Closed

Description

@r-richmond

Apache Airflow version

3.0.3

What happened?

Despite all of the documentation on how to configure logging in Airflow 3, none of these configs are used and instead Airflow uses a hard-coded setting.

Specifically

rdjouder 3 weeks ago
Hello,
Looking at the code of airflow.sdk.log.configure_logging I see that conf.get("logging","logging_config_class", ... ) is not called.
Also logging.config.dictConfig is called with hard coded defaults thus completly erasing what configure_logging() from logging_config.py is doing.
I have a custom logging config that is working fine as of airflow 2.10.X and older.
Is my analysis wrong ? Is it a bug ?

ashb 3 weeks ago
It's intentional I'm afraid, and I am in the process of completely overhauling all of Airflow's logging to use structlog and to be able to produce JSON logs in this PR #52651

ashb 3 weeks ago
The logger is configured but Airflow won't use it anymore. The reason is that logs from the task process is sent as JSON to the supervisor process and written to file, and the UI then takes that JSON and renders it for display.

What you think should happen instead?

Airflow log configs should be used.

Anything else?

If I understand correctly, it will be resolved with #52651.

Creating this issue just for tracking 3.x blockers. Please add affected_version_3.0 label.

Code of Conduct

Activity

  1. added and removed
    needs-triagelabel for new issues that we didn't triage yet
    kind:bugThis is a clearly a bug
    on Jul 17, 2025
  2. changed the title [-]Airflow Logging Configs are not Used[/-] [+]Airflow Logging Configs are ignored / not used in Airflow 3.x[/+] on Jul 30, 2025
  3. ashb commented on Sep 1, 2025

    @ashb
    Member

    @r-richmond Are you most interested in the logging format for the daemons/airflow components, or for the display of task logs in the UI?

  4. r-richmond commented on Sep 2, 2025

    @r-richmond
    ContributorAuthor

    @r-richmond Are you most interested in the logging format for the daemons/airflow components, or for the display of task logs in the UI?

    @ashb Honestly I'm interested in several things

    1. Adding line number & file name to the format before msg
    2. Moving source to the start of line/before msg instead of at the end of the line/after msg
    3. Eventually further debugging to see if I can figure out why running DBT via bashOperator logs so much differently in v3 than it does in v2. (in v2 it logs once per sql statement whereas in v3 it logs once per line which floods the logs with a header for every line; I suspect this is due to some underlying changes and may not be avoidable but the user complaints I've been subject to dictate I at least try and look).
    4. As for the display of task logs the big feedback item from myself and from my users is that select/copy is much harder in v3 than v2. Also scrolling is much worse as it gets very slow after several hundred lines & page up/down don't work properly anymore. (This is almost certainly outside of what can be controlled with logging configs but is something I've very interested in sharing.

    Those are the big items IMO, FWIW I opened a slack thread when I first stumbled on this stuff. Eventually I found the above linked discussion which explained some of my 2 -> 3 logging pain hence me opening this issue. Sorry for the wall of text hopefully that helps.

  5. ashb commented on Sep 2, 2025

    @ashb
    Member

    @r-richmond All of those are about viewing the task logs, correct?

    The reason I ask is that in #52651 I'm re-working all the logging and config, but display and fields of Task logs specifically will mostly need to be handled in the UI

  6. r-richmond commented on Sep 2, 2025

    @r-richmond
    ContributorAuthor

    Yes this is all directed at task logs. No issues with audit logs and others at the moment.

    Edit: actually can you clarify? I'm assuming that once you implement the above fix for the configs the logging displayed in the task logs will respect those configs (i.e things like 1&2 above not 3 or 4). Or is there another layer that needs be done for the format of the logs in the task log viewer to be updated?

  7. ashb commented on Sep 2, 2025

    @ashb
    Member

    No -- my change linked above will not change the display of task logs. It might add back in line number and filename back, but the log_fmt won't change the display of task logs in the UI.

  8. 10 remaining items

  9. ashb commented on Oct 3, 2025

    @ashb
    Member

    making the Supervisor pluggable via a stable interface or entrypoint

    Making the whole thing pluggable? no. Absolutely not.

    Making the remote logger IO have some way of flagging "don't write logs on disk", sure, that could be added.

    What we'd really like is the ability to swap in our own Supervisor implementation (e.g. a DatadogSupervisor) that consumes the structured task JSON events, enriches them with tags/trace IDs, and forwards them directly to our pipeline — without touching disk.

    @Abdel39 You should be able to do all of that with the RemoteLoggerIO interface today -- make sure you define processors on your logging class. It will still create the file on disk, but if you end your processor with a raise DropEvent nothing will get written to that file.

    @property
    def processors(self) -> tuple[structlog.typing.Processor, ...]: ...
    """
    List of structlog processors to install in the task write path.
    This is useful if a remote logging provider wants to either transform the structured log messages as they
    are being written to a file, or if you want to upload messages as they are generated.
    """

    and an example processor (it doesn't drop, but you could)

    def proc(logger: structlog.typing.WrappedLogger, method_name: str, event: structlog.typing.EventDict):
    if not logger or not (stream_name := relative_path_from_logger(logger)):
    return event
    # We can't set the log stream name in the above init handler because
    # the log path isn't known at that stage.
    # Instead, we should always rely on the path (log stream name) provided by the logger.
    _handler.log_stream_name = stream_name.as_posix().replace(":", "_")
    name = event.get("logger_name") or event.get("logger", "")
    level = structlog.stdlib.NAME_TO_LEVEL.get(method_name.lower(), logging.INFO)
    msg = copy.copy(event)
    created = None
    if ts := msg.pop("timestamp", None):
    with contextlib.suppress(Exception):
    created = datetime.fromisoformat(ts)
    record = logRecordFactory(
    name, level, pathname="", lineno=0, msg=msg, args=(), exc_info=None, func=None, sinfo=None
    )
    if created is not None:
    ct = created.timestamp()
    record.created = ct
    record.msecs = int((ct - int(ct)) * 1000) + 0.0 # Copied from stdlib logging
    _handler.handle(record)
    return event

    This is not time sensitive and can wait after 3.1

    You are a bit late on that front, 3.1.0 was released two weeks ago... 😃

  10. zach-overflow commented on Oct 3, 2025

    @zach-overflow
    Contributor

    I would second that the ability to completely disable writing task logs to disk would be immensely useful.

    Making the remote logger IO have some way of flagging "don't write logs on disk", sure, that could be added.
    You should be able to do all of that with the RemoteLoggerIO interface today

    @ashb is there up-to-date documentation on this (and more generally docs on configuring the new Airflow structlog setup as of 3.1.0)? I have a rough idea of what you're describing in the linked code snippets you provided, but I don't see any mention of the RemoteLoggerIO class in the current Airflow 3.1.0 docs. Can that interface be considered part of the public Airflow interface?

  11. ashb commented on Oct 3, 2025

    @ashb
    Member

    No, nothing outside of the code currently, sorry.

  12. Dhunsheth commented on Oct 10, 2025

    @Dhunsheth

    Is there an issue with the LOGGING_CONFIG override?

    Based on the advanced logging documentation I have been trying to change the logging levels for specific loggers using a config.py file, however, it doesn't seem to be working.

    I have documented the full issue here: #56377

    Is this an issue with my setup specifically or a global issue as a result of the logging overhaul?

  13. brki commented on Nov 4, 2025

    @brki
    Contributor

    I would also very much like to be able to add some custom log processor.

    My use case, that was working fine with Airflow 2, by adding a custom handler: I want to collect task log lines and periodically write them in a chunk to a custom log file.

    I've done this on Airflow 2, because 1) I want to be able to see these log chunks as multiline log entries in Splunk, and 2) because the otel collector we use, with a modest amount of CPU and memory, chokes when it's configured to monitor all the individual standard task logs under AIRFLOW_HOME/logs/dag_id=WHATEVER/task_id=WHATEVER/*.log .

    I don't see any way to do this in Airflow 3. Perhaps there is, I don't know. Can we somehow just inject an additional processor to the structlog processors for all task logs?

    Edit: I'd also like to know how to add fields like dag_id, task_id, and run_id to the task log records.

  14. samush commented on Nov 24, 2025

    @samush

    hi, sorry for maybe non-exactly related question, but what has been changed in airflow 3+ that prevents "search on this page" functionality (ctrl+f) from finding results on task logs view in a case of pretty large logs (5000+)?
    i am asking because that was very common scenario in our team to open logs for failed task run, search for specific line with 'ctrl+f' there and then compare it with the same/similar lines of previous runs.
    now to investigate task fails, users have to download logs file for multiple task runs, that not very convenient.
    could you, please, clarify was it intended and if there some workaround to search through the logs via airflow ui ?
    ps: thanks for all your work and progress that you have been done with airflow 3+

  15. potiuk commented on Nov 24, 2025

    @potiuk
    Member

    hi, sorry for maybe non-exactly related question

    Yes. The question is unrelated, you will be better of asking it as a discussion maybe. loosely related questions and comments make it super difficult for people to focus on things and get you answers

  16. samush commented on Nov 24, 2025

    @samush

    got it, thanks

  17. tonytrill commented on Dec 10, 2025

    @tonytrill

    Currently running 3.1.3 and running into the same issue as the original issue listed.

  18. pam-emtelligent commented on Jan 25, 2026

    @pam-emtelligent

    Currently running 3.1.6 and this is still happening - cannot modify log configs using. logging_class_path config
    This documentation is stale:
    https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/advanced-logging-configuration.html#create-a-custom-logging-class

    I want to:
    (1) set the airflow.task logger to DEBUG level by default (without having to change thelogging_level airflow config from INFO to DEBUG
    (2) add some DAG params to the logging format (for example by modifying the dictConfig).

    What is the recommended way to do this in Airflow 3.x?

  19. DartVeDroid commented on Feb 1, 2026

    @DartVeDroid

    As far as I understand it, in Airflow 3.1.6 logging works like that:

    • in task:
      • we make logs
      • logs get passed to supervisor
    • in most of airflow itself (supervisor, etc):
      • we make logs OR they get passed from task
        • they get passed to structlog bound logger
          • to bound logger's wrapper AirflowBoundLoggerFilteringAt<...>
            • to wrapper's logger factory's logger: NamedBytesLogger or NamedWriteLogger
              • to logger's output: stdout or log file
    • in api-server:
      • we make logs
        • logs get passed to ???

    So, what we have to do is:

    • disable output to files (because I wanted to only use stdout and custom handlers a-la Airflow 2)
    • replace logger_factory with one that returns stdlib logging loggers, when we are not just sending JSONs to another parts of Airflow
    • modify airflow.sdk.log.configure_logging() to load custom logging config too just for triggerer and worker (most airflow components load it already via airflow.logging_config.configure_logging(), but some do not, for unknown reasons)
    • (in progress) deal with:
      • (optionally) allow all logs to be JSON and support for stdlib logging's records' .msg field to be a JSON
      • custom logging handlers' set_context() method not being called

    I'm currently on the last step, but I'm not sure if I would be allowed to post the final code here, despite it being only a crutch honestly.

    P.S.:
    I'm fine with contributors making changes that break providers' functionality, but, IMO, either providers' docs should be updated (which is kinda hard without the main contributor's involvement), or providers should explicitly declare that they are incompatible with the last "Stable" version of Airflow. As of now it's just confusing.

  20. DartVeDroid commented on Feb 4, 2026

    @DartVeDroid

    Welp, it's weird:

    • Most logging providers' handlers currently require for the set_context() method to be called, to pass dag and task information to it. But:
      • Some contributor or maintainer (can't remember who it was exactly now) has mentioned previously that set_context() method is not supposed to be called by Airflow 3
      • Some handlers, like ElasticsearchTaskHandler, could also explicitly try to access Airflow database, which would fail, due to direct database access being against Airflow 3 architectural principles
    • Now, the dag and task information is accessible from a task itself by calling a airflow.sdk.get_current_context() method, and it is possible to create a structlog processor that would call it and enrich log JSONs. But:
      • You can only enrich a structlog log like that, and while your code could probably be modified to use it, you would also need to modify your requirements' logging too, and it does not look easily done to me
    • Theoretically, you could enrich all logs passed from a task to supervisor, but that would require for supervisor.process_log_messages_from_subprocess() generator method to get an ActivitySubprocess.ti as an optional argument

    I'll try the last option listed here, as it seems both viable and consistent with Airflow 3.
    It would rely on all structlog logs being JSON and logging handlers' emit() method supporting record.msg being a JSON, though.

    UPD:

    • enriching task logs works (added a subprocess_wrapper: ActivitySubprocess | WatchedSubprocess parameter)
    • ended up creating a custom descendant of logging.StreamHandler and using it in three cases:
      • airflow.task logger
      • flask_appbuilder logger
      • root logger (you should change it in airflow itself, meaning /airflow/_shared/logging/structlog.py and /airflow/sdk/_shared/logging/structlog.py, because it overrides the value you set in logging config)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions