Skip to content

GCS gzip log upload #12609

New issue
New issue
Open
Open
GCS gzip log upload#12609
Labels
area:loggingkind:featureFeature Requestsprovider:googleGoogle (including GCP) related issues

Description

@RyanSiu1995
RyanSiu1995
opened on Nov 25, 2020

Description
For now, GCS Log Handler does not provide an option for user to upload the logs with gzip. It is good if we could configure it directly.

Use case / motivation
The gzip log files will help the user to save a ton of money in storing the log files.

Related Issues

Activity

  1. RyanSiu1995
    added
    kind:featureFeature Requests
    on Nov 25, 2020
  2. eladkal
    added
    area:logging
    on Nov 25, 2020
  3. eladkal
    added
    provider:googleGoogle (including GCP) related issues
    on Oct 10, 2021

  4. dominikhei commented on Dec 9, 2024

    @dominikhei
    dominikhei
    on Dec 9, 2024
    Contributor

    @eladkal I just checked the GCSTaskHandler class under: providers/src/airflow/providers/google/cloud/log/gcs_task_handler.py and there seems to be no implementation of this.

    Is this functionality still desired? If so, using Python's standard gzip module is probably a good option, as it does not require additional dependencies. To upload the logs we could simply add a boolean param like gzip_compress_logs to the class.

    Do you think the _read_remote_logs method should be updated accordingly, to be able to read and decompress, gzip compressed logs?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:loggingkind:featureFeature Requestsprovider:googleGoogle (including GCP) related issues

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions