A Python operator pauses the run on a LONG column that holds a null and a value past 2^53 #9077

What happened?

A Python operator, such as Sort or a Python UDF, pauses the workflow when its input has a LONG column that holds a null and a value past 2^53. The run stops with:

TypeError: Unmatched type for field 'big', expected AttributeType.LONG, got 9007199254741012.0 (<class 'float'>) instead.

The value is already wrong before that check runs. Table.from_tuple_likes builds the operator's input with pandas.DataFrame.from_records, and pandas 2.2.3 turns an integer column that holds a None into float64. A float64 holds every integer only up to 2^53, so 9007199254740993 becomes 9007199254740992. Tuple.cast_to_schema turns an integral float back into an int only inside that window, as #6278 decided, so a value outside it fails validation instead of being written out wrong.

The cause is the input conversion. Building the frame with a nullable Int64 dtype for INTEGER and LONG columns would keep both the null and the value. That changes the dtype every Python operator is handed, so it seems worth agreeing on before a change.

How to reproduce?

The conversion alone, with pandas 2.2.3:

import pandas as pd
df = pd.DataFrame.from_records([(9007199254740993,), (None,)], columns=["big"])
df["big"].dtype   # float64
int(df["big"][0]) # 9007199254740992

End to end, a workflow that reads a CSV with a LONG column holding 9007199254740993 and an empty cell, and sorts it with Sort, pauses on the error above.

Version/Branch

1.4.0-incubating-SNAPSHOT (main)

Commit Hash (Optional)

3ba45ee

What browsers are you seeing the problem on?

No response

Relevant log output

TypeError: Unmatched type for field 'big', expected AttributeType.LONG, got 9007199254741012.0 (<class 'float'>) instead. The yielded columns and their types must match the operator's output columns