Skip to content

numpy.bool_ in Numpy 2 cannot be serialized/deserialized when being passed into XComs #52753

@sjyangkevin

Description

@sjyangkevin

Apache Airflow version

3.0.2

If "Other Airflow 2 version" selected, which one?

No response

What happened?

When numpy is upgraded to 2.2.6. The numpy.bool_ objects cannot be serialized/deserialized. It is because the qualified name is resolved to numpy.bool instead of numpy.bool_ while the serializers scan for numpy.bool_. As shown below:

Therefore, this object is not matched to any of the serializer and return TypeError: cannot serialize object of type <class 'numpy.bool'>

When numpy version is 1.26.4, the qualified name is resolved to numpy.bool_
Image
When numpy version is 2.2.6, the qualified name is resolved to numpy.bool
Image

What you think should happen instead?

numpy.bool_ objects should be serialized/deserialized when it is passed through XComs.

How to reproduce

The DAG code to reproduce the error.

from airflow.decorators import dag, task
from pendulum import datetime


@dag(
    start_date=datetime(2025, 5, 23),
    schedule=None,
    catchup=False,
    tags=["serialization", "airflow"],
)
def pydantic_serde():

    # 5. NumPy scalar types - misc
    @task
    def get_numpy_misc():
        import numpy as np
        print(np.__version__)
        print(type(np.bool_(False)))
        print(np.bool_.__name__)
        return {
            "bool_": np.bool_(False),
            "float16": np.float16(0.125),
            "float64": np.float64(3.14159),
            "complex64": np.complex64(1 + 2j),
            "complex128": np.complex128(3 + 4j),
        }

    @task
    def print_numpy_misc(obj):
        print("NumPy Misc Types:")
        for k, v in obj.items():
            print(f"{k}: {v} ({type(v)})")

    # DAG chaining
    print_numpy_misc(get_numpy_misc())

pydantic_serde()

Operating System

Ubuntu 22.04.3 LTS

Versions of Apache Airflow Providers

No response

Deployment

Docker-Compose

Deployment details

I run Airflow in Docker by following the guide. We need to update the docker-compose.yaml to build from the following Docker image. We can create a requirements.txt to install numpy 2.2.6
https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html

FROM apache/airflow:3.0.2

COPY requirements.txt requirements.txt

RUN pip install --no-cache-dir -r requirements.txt

USER ${AIRFLOW_UID}

requirements.txt

numpy==2.2.6

Anything else?

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions