Skip to content

Commit 83769ae

Browse files
authored
feat: Python Support for Large Binary (apache#4100)
<!-- Thanks for sending a pull request (PR)! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: [Contributing to Texera](https://github.com/apache/texera/blob/main/CONTRIBUTING.md) 2. Ensure you have added or run the appropriate tests for your PR 3. If the PR is work in progress, mark it a draft on GitHub. 4. Please write your PR title to summarize what this PR proposes, we are following Conventional Commits style for PR titles as well. 5. Be sure to keep the PR description updated to reflect all changes. --> ### What changes were proposed in this PR? <!-- Please clarify what changes you are proposing. The purpose of this section is to outline the changes. Here are some tips for you: 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, you can clarify why it is a bug. 3. If it is a refactoring, clarify what has been changed. 3. It would be helpful to include a before-and-after comparison using screenshots or GIFs. 4. Please consider writing useful notes for better and faster reviews. --> This PR introduces Python support for the `large_binary` attribute type, enabling Python UDF operators to process data larger than 2 GB. Data is offloaded to MinIO (S3), and the tuple retains only a pointer (URI). This mirrors the existing Java LargeBinary implementation, ensuring cross-language compatibility. (See apache#4067 for system diagram and apache#4111 for renaming) ## Key Features ### 1. MinIO/S3 Integration - Utilizes the shared `texera-large-binaries` bucket. - Implements lazy initialization of S3 clients and automatic bucket creation. ### 2. Streaming I/O - **`LargeBinaryOutputStream`:** Writes data to S3 using multipart uploads (64KB chunks) to prevent blocking the main execution. - **`LargeBinaryInputStream`:** Lazily downloads data only when the read operation begins. Implements standard Python `io.IOBase`. ### 3. Tuple & Iceberg Compatibility - `largebinary` instances are automatically serialized to URI strings for Iceberg storage and Arrow tables. - Uses a magic suffix (`__texera_large_binary_ptr`) to distinguish pointers from standard strings. ### 4. Serialization - Pointers are stored as strings with metadata (`texera_type: LARGE_BINARY`). Auto-conversion ensures UDFs always see `largebinary` instances, not raw strings. ## User API Usage ### 1. Creating & Writing (Output) Use `LargeBinaryOutputStream` to stream large data into a new object. ```python from pytexera import largebinary, LargeBinaryOutputStream # Create a new handle large_binary = largebinary() # Stream data to S3 with LargeBinaryOutputStream(large_binary) as out: out.write(my_large_data_bytes) # Supports bytearray, bytes, etc. ``` ### 2. Reading (Input) Use `LargeBinaryInputStream` to read data back. It supports all standard Python stream methods. ```python from pytexera import LargeBinaryInputStream with LargeBinaryInputStream(large_binary) as stream: # Option A: Read everything all_data = stream.read() # Option B: Chunked reading chunk = stream.read(1024) # Option C: Iteration for line in stream: process(line) ``` ## Dependencies - `boto3`: Required for S3 interactions. - `StorageConfig`: Uses existing configuration for endpoints/credentials. ## Future Direction - Support for R UDF Operators - Check apache#4123 ### Any related issues, documentation, discussions? <!-- Please use this section to link other resources if not mentioned already. 1. If this PR fixes an issue, please include `Fixes apache#1234`, `Resolves apache#1234` or `Closes apache#1234`. If it is only related, simply mention the issue number. 2. If there is design documentation, please add the link. 3. If there is a discussion in the mailing list, please add the link. --> Design: apache#3787 ### How was this PR tested? <!-- If tests were added, say they were added here. Or simply mention that if the PR is tested with existing test cases. Make sure to include/update test cases that check the changes thoroughly including negative and positive cases if possible. If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future. If tests were not added, please describe why they were not added and/or why it was difficult to add. --> Tested by running this workflow multiple times and check MinIO dashboard to see whether six objects are created and deleted. Specify the file scan operator's property to use any file bigger than 2GB. [Large Binary Python.json](https://github.com/user-attachments/files/24062982/Large.Binary.Python.json) ### Was this PR authored or co-authored using generative AI tooling? <!-- If generative AI tooling has been used in the process of authoring this PR, please include the phrase: 'Generated-by: ' followed by the name of the tool and its version. If no, write 'No'. Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details. --> No. --------- Signed-off-by: Chris <[email protected]>
1 parent 56eec40 commit 83769ae

33 files changed

Lines changed: 2214 additions & 49 deletions

‎amber/operator-requirements.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,3 +23,4 @@ pybase64==1.3.2
2323
torch==2.8.0
2424
scikit-learn==1.5.0
2525
transformers==4.57.3
26+
boto3==1.40.53

‎amber/src/main/python/core/architecture/packaging/output_manager.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -262,7 +262,7 @@ def tuple_to_frame(self, tuples: typing.List[Tuple]) -> DataFrame:
262262
return DataFrame(
263263
frame=Table.from_pydict(
264264
{
265-
name: [t[name] for t in tuples]
265+
name: [t.get_serialized_field(name) for t in tuples]
266266
for name in self.get_port().get_schema().get_attr_names()
267267
},
268268
schema=self.get_port().get_schema().as_arrow_schema(),

‎amber/src/main/python/core/models/__init__.py‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@
1515
# specific language governing permissions and limitations
1616
# under the License.
1717

18+
import builtins
1819
from inspect import Traceback
1920
from typing import NamedTuple
2021

@@ -36,7 +37,7 @@
3637

3738

3839
class ExceptionInfo(NamedTuple):
39-
exc: type
40+
exc: builtins.type
4041
value: Exception
4142
tb: Traceback
4243

‎amber/src/main/python/core/models/schema/__init__.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,12 +16,14 @@
1616
# under the License.
1717

1818
from .attribute_type import AttributeType
19+
from core.models.type.large_binary import largebinary
1920
from .field import Field
2021
from .schema import Schema
2122

2223

2324
__all__ = [
2425
"AttributeType",
26+
"largebinary",
2527
"Field",
2628
"Schema",
2729
]
Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
"""
19+
Utilities for converting between Arrow schemas and Amber schemas,
20+
handling LARGE_BINARY metadata preservation.
21+
"""
22+
23+
import pyarrow as pa
24+
from typing import Mapping
25+
26+
from core.models.schema.attribute_type import AttributeType
27+
from core.models.schema.attribute_type_utils import (
28+
detect_attribute_type_from_arrow_field,
29+
create_arrow_field_with_metadata,
30+
)
31+
32+
33+
def arrow_schema_to_attr_types(arrow_schema: pa.Schema) -> dict[str, AttributeType]:
34+
"""
35+
Converts an Arrow schema to a dictionary of attribute name to AttributeType.
36+
Handles LARGE_BINARY metadata detection.
37+
38+
:param arrow_schema: PyArrow schema that may contain LARGE_BINARY metadata
39+
:return: Dictionary mapping attribute names to AttributeTypes
40+
"""
41+
attr_types = {}
42+
for attr_name in arrow_schema.names:
43+
field = arrow_schema.field(attr_name)
44+
attr_types[attr_name] = detect_attribute_type_from_arrow_field(field)
45+
return attr_types
46+
47+
48+
def attr_types_to_arrow_schema(
49+
attr_types: Mapping[str, AttributeType],
50+
) -> pa.Schema:
51+
"""
52+
Converts a mapping of attribute name to AttributeType into an Arrow schema.
53+
Adds metadata for LARGE_BINARY types.
54+
Preserves the order of attributes from the input mapping.
55+
56+
:param attr_types: Mapping of attribute names to AttributeTypes (e.g., OrderedDict)
57+
:return: PyArrow schema with metadata for LARGE_BINARY types
58+
"""
59+
fields = [
60+
create_arrow_field_with_metadata(attr_name, attr_type)
61+
for attr_name, attr_type in attr_types.items()
62+
]
63+
return pa.schema(fields)

‎amber/src/main/python/core/models/schema/attribute_type.py‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020
from bidict import bidict
2121
from enum import Enum
2222
from pyarrow import lib
23+
from core.models.type.large_binary import largebinary
2324

2425

2526
class AttributeType(Enum):
@@ -37,6 +38,7 @@ class AttributeType(Enum):
3738
DOUBLE = 5
3839
TIMESTAMP = 6
3940
BINARY = 7
41+
LARGE_BINARY = 8
4042

4143

4244
RAW_TYPE_MAPPING = bidict(
@@ -48,6 +50,7 @@ class AttributeType(Enum):
4850
"BOOLEAN": AttributeType.BOOL,
4951
"TIMESTAMP": AttributeType.TIMESTAMP,
5052
"BINARY": AttributeType.BINARY,
53+
"LARGE_BINARY": AttributeType.LARGE_BINARY,
5154
}
5255
)
5356

@@ -59,6 +62,7 @@ class AttributeType(Enum):
5962
AttributeType.BOOL: pa.bool_(),
6063
AttributeType.BINARY: pa.binary(),
6164
AttributeType.TIMESTAMP: pa.timestamp("us"),
65+
AttributeType.LARGE_BINARY: pa.string(), # Serialized as URI string
6266
}
6367

6468
FROM_ARROW_MAPPING = {
@@ -83,6 +87,7 @@ class AttributeType(Enum):
8387
AttributeType.BOOL: bool,
8488
AttributeType.BINARY: bytes,
8589
AttributeType.TIMESTAMP: datetime.datetime,
90+
AttributeType.LARGE_BINARY: largebinary,
8691
}
8792

8893
FROM_PYOBJECT_MAPPING = {
@@ -92,4 +97,5 @@ class AttributeType(Enum):
9297
bool: AttributeType.BOOL,
9398
bytes: AttributeType.BINARY,
9499
datetime.datetime: AttributeType.TIMESTAMP,
100+
largebinary: AttributeType.LARGE_BINARY,
95101
}
Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
"""
19+
Utilities for converting between AttributeTypes and Arrow field types,
20+
handling LARGE_BINARY metadata preservation.
21+
"""
22+
23+
import pyarrow as pa
24+
25+
from core.models.schema.attribute_type import (
26+
AttributeType,
27+
FROM_ARROW_MAPPING,
28+
TO_ARROW_MAPPING,
29+
)
30+
31+
# Metadata key used to mark LARGE_BINARY fields in Arrow schemas
32+
TEXERA_TYPE_METADATA_KEY = b"texera_type"
33+
LARGE_BINARY_METADATA_VALUE = b"LARGE_BINARY"
34+
35+
36+
def detect_attribute_type_from_arrow_field(field: pa.Field) -> AttributeType:
37+
"""
38+
Detects the AttributeType from an Arrow field, checking metadata for LARGE_BINARY.
39+
40+
:param field: PyArrow field that may contain metadata
41+
:return: The detected AttributeType
42+
"""
43+
# Check metadata for LARGE_BINARY type
44+
# (can be stored by either Scala ArrowUtils or Python)
45+
is_large_binary = (
46+
field.metadata
47+
and field.metadata.get(TEXERA_TYPE_METADATA_KEY) == LARGE_BINARY_METADATA_VALUE
48+
)
49+
50+
if is_large_binary:
51+
return AttributeType.LARGE_BINARY
52+
else:
53+
return FROM_ARROW_MAPPING[field.type.id]
54+
55+
56+
def create_arrow_field_with_metadata(
57+
attr_name: str, attr_type: AttributeType
58+
) -> pa.Field:
59+
"""
60+
Creates a PyArrow field with appropriate metadata for the given AttributeType.
61+
62+
:param attr_name: Name of the attribute
63+
:param attr_type: The AttributeType
64+
:return: PyArrow field with metadata if needed
65+
"""
66+
metadata = (
67+
{TEXERA_TYPE_METADATA_KEY: LARGE_BINARY_METADATA_VALUE}
68+
if attr_type == AttributeType.LARGE_BINARY
69+
else None
70+
)
71+
72+
return pa.field(attr_name, TO_ARROW_MAPPING[attr_type], metadata=metadata)

‎amber/src/main/python/core/models/schema/schema.py‎

Lines changed: 8 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -22,8 +22,10 @@
2222
from core.models.schema.attribute_type import (
2323
AttributeType,
2424
RAW_TYPE_MAPPING,
25-
FROM_ARROW_MAPPING,
26-
TO_ARROW_MAPPING,
25+
)
26+
from core.models.schema.arrow_schema_utils import (
27+
arrow_schema_to_attr_types,
28+
attr_types_to_arrow_schema,
2729
)
2830

2931

@@ -85,22 +87,17 @@ def _from_arrow_schema(self, arrow_schema: pa.Schema) -> None:
8587
:return:
8688
"""
8789
self._name_type_mapping = OrderedDict()
90+
attr_types = arrow_schema_to_attr_types(arrow_schema)
91+
# Preserve field order from arrow_schema
8892
for attr_name in arrow_schema.names:
89-
arrow_type = arrow_schema.field(attr_name).type # type: ignore
90-
attr_type = FROM_ARROW_MAPPING[arrow_type.id]
91-
self.add(attr_name, attr_type)
93+
self.add(attr_name, attr_types[attr_name])
9294

9395
def as_arrow_schema(self) -> pa.Schema:
9496
"""
9597
Creates a new pyarrow.Schema according to the current Schema.
9698
:return: pyarrow.Schema
9799
"""
98-
return pa.schema(
99-
[
100-
pa.field(attr_name, TO_ARROW_MAPPING[attr_type])
101-
for attr_name, attr_type in self._name_type_mapping.items()
102-
]
103-
)
100+
return attr_types_to_arrow_schema(self._name_type_mapping)
104101

105102
def get_attr_names(self) -> List[str]:
106103
"""

‎amber/src/main/python/core/models/schema/test_schema.py‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,3 +89,68 @@ def test_convert_from_raw_schema(self, raw_schema, schema):
8989
def test_convert_from_arrow_schema(self, arrow_schema, schema):
9090
assert schema == Schema(arrow_schema=arrow_schema)
9191
assert schema.as_arrow_schema() == arrow_schema
92+
93+
def test_large_binary_in_raw_schema(self):
94+
"""Test creating schema with LARGE_BINARY from raw schema."""
95+
raw_schema = {
96+
"regular_field": "STRING",
97+
"large_binary_field": "LARGE_BINARY",
98+
}
99+
schema = Schema(raw_schema=raw_schema)
100+
assert schema.get_attr_type("regular_field") == AttributeType.STRING
101+
assert schema.get_attr_type("large_binary_field") == AttributeType.LARGE_BINARY
102+
103+
def test_large_binary_in_arrow_schema_with_metadata(self):
104+
"""Test creating schema with LARGE_BINARY from Arrow schema with metadata."""
105+
arrow_schema = pa.schema(
106+
[
107+
pa.field("regular_field", pa.string()),
108+
pa.field(
109+
"large_binary_field",
110+
pa.string(),
111+
metadata={b"texera_type": b"LARGE_BINARY"},
112+
),
113+
]
114+
)
115+
schema = Schema(arrow_schema=arrow_schema)
116+
assert schema.get_attr_type("regular_field") == AttributeType.STRING
117+
assert schema.get_attr_type("large_binary_field") == AttributeType.LARGE_BINARY
118+
119+
def test_large_binary_as_arrow_schema_includes_metadata(self):
120+
"""Test that LARGE_BINARY fields include metadata in Arrow schema."""
121+
schema = Schema()
122+
schema.add("regular_field", AttributeType.STRING)
123+
schema.add("large_binary_field", AttributeType.LARGE_BINARY)
124+
125+
arrow_schema = schema.as_arrow_schema()
126+
127+
# Regular field should have no metadata
128+
regular_field = arrow_schema.field("regular_field")
129+
assert (
130+
regular_field.metadata is None
131+
or b"texera_type" not in regular_field.metadata
132+
)
133+
134+
# LARGE_BINARY field should have metadata
135+
large_binary_field = arrow_schema.field("large_binary_field")
136+
assert large_binary_field.metadata is not None
137+
assert large_binary_field.metadata.get(b"texera_type") == b"LARGE_BINARY"
138+
assert (
139+
large_binary_field.type == pa.string()
140+
) # LARGE_BINARY is stored as string
141+
142+
def test_round_trip_large_binary_schema(self):
143+
"""Test round-trip conversion of schema with LARGE_BINARY."""
144+
original_schema = Schema()
145+
original_schema.add("field1", AttributeType.STRING)
146+
original_schema.add("field2", AttributeType.LARGE_BINARY)
147+
original_schema.add("field3", AttributeType.INT)
148+
149+
# Convert to Arrow and back
150+
arrow_schema = original_schema.as_arrow_schema()
151+
round_trip_schema = Schema(arrow_schema=arrow_schema)
152+
153+
assert round_trip_schema == original_schema
154+
assert round_trip_schema.get_attr_type("field1") == AttributeType.STRING
155+
assert round_trip_schema.get_attr_type("field2") == AttributeType.LARGE_BINARY
156+
assert round_trip_schema.get_attr_type("field3") == AttributeType.INT

0 commit comments

Comments
 (0)