Bug description

create_table() accepts pa.json_() (PyArrow's arrow.json extension type) but internally lance-core converts it to lance.json (backed by large_binary with ARROW:extension:name: lance.json field metadata). However, table.add() does not account for this: the cast layer in lancedb tries to cast Utf8 → LargeBinary, and the resulting field carries the input's arrow.json metadata instead of the table's lance.json metadata. Lance-core then sees a schema mismatch (json vs large_binary) and rejects the append.

Minimal reproduction

import pyarrow as pa
import lancedb
import tempfile

with tempfile.TemporaryDirectory() as tmp:
    db = lancedb.connect(tmp)

    # Create table with pa.json_() — internally stored as lance.json
    schema = pa.schema([pa.field("data", pa.json_(), nullable=True)])
    tbl = db.create_table("t", schema=schema)

    # Verify internal conversion happened
    f = tbl.schema.field("data")
    assert f.type == pa.large_binary()
    assert f.metadata[b"ARROW:extension:name"] == b"lance.json"

    # Append pa.json_() data — this fails
    arr = pa.array([None, '{"a": 1}'], type=pa.json_())
    tbl.add(pa.table({"data": arr}))

Error:

RuntimeError: lance error: Append with different schema:
  `data` should have type json but type was large_binary

Expected behavior

table.add() should accept pa.json_() data on a table that was created with pa.json_().

Fix

In build_field_exprs, detect when the input field is an arrow.json extension and the table field is lance.json, and pass through without casting. Lance-core's write path already handles the arrow.json → lance.json conversion (including JSONB encoding):

if is_arrow_json_field(input_field) && is_json_field(table_field) {
    result.push((input_expr, input_field.clone()));
    continue;
}

Workaround

Manually produce large_binary arrays with lance.json field metadata before calling add():

field = pa.field("data", pa.large_binary(), nullable=True, metadata={
    b"ARROW:extension:name": b"lance.json",
    b"ARROW:extension:metadata": b"",
})
arr = pa.array([None, b'{"a": 1}'], type=pa.large_binary())
tbl.add(pa.table({"data": arr}, schema=pa.schema([field])))

Versions

• lancedb==0.30.0b5, pyarrow==23.0.1