You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Avro schema parser uses type name instead of field name [AVRO] #8928
The schema parser will use "record2" as the field name, despite it actually being the type name, the field name should be "f1", this means conversion from ArrowSchemas that don't contain the schema json in the metadata will always fail, and in general the schemas will not be applicable to the output recordbatch.
To Reproduce
Create an avro file with the above writer schema, then use the following arrow schema to create a reader_schema
It will error on mismatch in field names because the writer schema will have the field "f1" correctly, but the newly created reader_schema will have "record2"
@EmilyMatt I carefully reviewed this issue + the code over the past few hours. I realized the child metadata required for f1 was missing in your reproduction description.
I know this is burdensome and potentially heavy on the caller. The challenge here has been inter-oping Avro's type naming system with Arrow Schema. I couldn't find another way to do it which:
Didn't involve using metadata.
Wouldn't cause explicit failures / extra complexity for callers just wanting to encode Arrow data--not originally from an Avro source--into Avro.
The reason for the name generation logic was to make the implementation less rigid for callers going from pure Arrow (with the metadata missing) to Avro (i.e. point 2). My thinking was if there wasn't an original Avro schema, then generating one in this manner wouldn't necessarily be incorrect so long as it was in spec and repeatable with optional overrides (such as via the metadata).
Additionally I included the Arrow Schema's SCHEMA_METADATA_KEY logic as a flexible alternative for callers with the complete Avro schema so they could just inject it into the Arrow Schema for downstream use. This is approach did come in handy moving the DataFusion work forward.
Basically I was attempting to balance flexibility, practicality, and correctness, which was somewhat challenging in this scenario. Furthermore, I intended to take this rougher metadata based foundation and extend the crate's public api to gradually offload complexity from the caller while maintaining backwards compatibility. #8923 is example of that in practice.
However, I'm 100% open to and hoping for a better approach and already planned to refactor the Arrow Schema -> AvroSchema conversion logic in arrow-avro/src/schema.rs anyway due to it's brittleness. (Which is why I didn't catch the missing metadata sooner lol. If there's a bug in arrow-avro, I'm 98% confident it would pop-up in that schema conversion logic, so was already expecting it).
Anyway when I ran this reproducing test below in the arrow-avro/src/schema.rs file, with the child metadata now inserted, it passed with the reader and writer schemas matching exactly.
#[test]fntest_reader_schema_from_arrow_preserves_nested_field_name(){// Writer schema as described in issuelet writer_json = r#"{ "namespace": "ns1", "name": "main", "type": "record", "fields": [ { "name": "f1", "type": { "type": "record", "namespace": "ns2", "name": "record2", "fields": [ { "name": "f1_1", "type": "string" } ] } } ] }"#;// Parse the writer schema into the internal Avro representationlet writer_schema:Schema =
serde_json::from_str(writer_json).expect("writer schema JSON should parse");letmut md = HashMap::new();
md.insert(AVRO_NAMESPACE_METADATA_KEY.to_string(),"ns1".to_string());
md.insert(AVRO_NAME_METADATA_KEY.to_string(),"main".to_string());letmut f1_md = HashMap::new();
f1_md.insert(AVRO_NAMESPACE_METADATA_KEY.to_string(),"ns2".to_string());
f1_md.insert(AVRO_NAME_METADATA_KEY.to_string(),"record2".to_string());let arrow_reader_schema = ArrowSchema::new_with_metadata(vec![ArrowField::new("f1",DataType::Struct(Fields::from(vec![ArrowField::new("f1_1",DataType::Utf8,false,)])),false,).with_metadata(f1_md),],
md,);// Build the Avro reader schema from the Arrow schemalet reader_avro = AvroSchema::try_from(&arrow_reader_schema).expect("Arrow -> Avro schema conversion should succeed");let reader_schema:Schema = reader_avro
.schema().expect("generated Avro schema JSON should parse");// Both writer and reader should be top-level recordslet writer_record = match&writer_schema {Schema::Complex(ComplexType::Record(r)) => r,
other => panic!("expected writer to be a record, got: {other:?}"),};let reader_record = match&reader_schema {Schema::Complex(ComplexType::Record(r)) => r,
other => panic!("expected reader to be a record, got: {other:?}"),};// Sanity: writer field name is "f1"assert_eq!(writer_record.fields.len(),1);assert_eq!(writer_record.fields[0].name,"f1");assert_eq!(
reader_record.fields[0].name,"f1","Reader schema must preserve the Arrow field name `f1` and \ must not substitute the nested record's type name (`record2`) as the field name");println!("{reader_schema:#?}\n\n{writer_schema:#?}");// Critical assert: If the bug is present, the reader schema and writer schema will not matchassert_eq!(reader_schema, writer_schema);let _resolved = AvroFieldBuilder::new(&writer_schema).with_reader_schema(&reader_schema).with_utf8view(false).with_strict_mode(false).build().expect("Writer and generated reader schemas should resolve without field name mismatches",);}
I do think the schema.rs constant names + relevant docs being better / more descriptive would help here as well.
Honestly I think my main interest is the cases where the ArrowSchema does not contain any metadata.
If I provide a valid arrow schema, do an AvroSchema::try_from on it, and provide it as the read_schema, I expected the two to be compatible.
Currently it crashes, which is my main issue^^
Honestly I think my main interest is the cases where the ArrowSchema does not contain any metadata. If I provide a valid arrow schema, do an AvroSchema::try_from on it, and provide it as the read_schema, I expected the two to be compatible. Currently it crashes, which is my main issue^^
That makes sense and I 100% understand where you're coming from.
Right now AvroSchema::try_from(&ArrowSchema) can only work with whatever information is actually present in the ArrowSchema. When there’s no Avro metadata at all, the Arrow side won't contain:
The original Avro record type names, namespaces, etc. (i.e. ns2.record2), or
Any indication that a given Struct used to be a named Avro record.
Because AvroSchema::try_from was originally written for the Writer, missing Schema metadata was interpreted as there being no originally sourced Avro schema, therefore we were free to generate one with the information available.
So for something like your nested f1 struct, AvroSchema::try_from has to invent a legal Avro record name. The current implementation does that by using the field name ("f1") when avro.name / avro.namespace aren’t present. That means the generated reader schema ends up with:
writer: nested record name = "record2" (namespace "ns2")
reader: nested record name = "f1" (no namespace)
When you then feed that generated schema into the resolver as read_schema, Avro resolution correctly fails with a record‑name mismatch. From Avro’s perspective those are two different types, even though the shape matches.
Essentially there are two slightly different expectations here:
What the code currently guarantees:
From an Arrow schema with no/missing Avro metadata, arrow-avro will generate a valid Avro schema that correctly describes the Arrow layout, but we can't promise it will be compatible with a pre‑existing Avro writer schema that used specific named record types / namespaces, etc.
More streamlined approach:
If the ArrowSchema is structurally compatible with the writer schema, then AvroSchema::try_from + read_schema should just work, even without metadata.
For schemas that don’t rely on Avro named types, those two line up and things work fine. For named records (like your record2), they diverge because Arrow simply doesn’t carry all the needed information unless we stash them in metadata.
Practically speaking, the patterns available are:
If you already have the Avro writer schema JSON, reuse that in the read_schema (or store it in SCHEMA_METADATA_KEY on the Arrow Schema), or
When you do want to round‑trip via Arrow, attach avro.name / avro.namespace on the relevant struct/dictionary fields so the generated reader schema can match the writer’s names.
I totally agree this is not obvious from the API, and from your perspective it just looks like you passed a perfectly valid Arrow schema and the resolution crashed. So at a minimum we should:
Document this limitation more clearly, and
Consider whether we can support a more permissive resolution mode for reader schemas derived from Arrow (i.e. fall back to structural matching when the reader has no avro.name/avro.namespace), without breaking spec‑compliant cases.
Extend the public API to simplify as much of this metadata related complexity as possible. Perhaps new logic to make it easy to derive a resolution compatible reader schema from a specific writer schema?
If you’re interested, we could track that as a follow‑up enhancement: something along the lines of "support using pure Arrow schemas as reader schemas without requiring Avro naming metadata", and then discuss what trade‑offs / flags that would need.
Activity
jecsand838 commentedon Nov 26, 2025
@EmilyMatt Thank you for opening this issue. I'll take a look at it today.
jecsand838 commentedon Nov 26, 2025
take
jecsand838 commentedon Nov 27, 2025
@EmilyMatt I carefully reviewed this issue + the code over the past few hours. I realized the child metadata required for
f1was missing in your reproduction description.Including this metadata would like this:
I know this is burdensome and potentially heavy on the caller. The challenge here has been inter-oping Avro's type naming system with Arrow Schema. I couldn't find another way to do it which:
The reason for the name generation logic was to make the implementation less rigid for callers going from pure Arrow (with the metadata missing) to Avro (i.e. point 2). My thinking was if there wasn't an original Avro schema, then generating one in this manner wouldn't necessarily be incorrect so long as it was in spec and repeatable with optional overrides (such as via the metadata).
Additionally I included the Arrow Schema's
SCHEMA_METADATA_KEYlogic as a flexible alternative for callers with the complete Avro schema so they could just inject it into the ArrowSchemafor downstream use. This is approach did come in handy moving the DataFusion work forward.Basically I was attempting to balance flexibility, practicality, and correctness, which was somewhat challenging in this scenario. Furthermore, I intended to take this rougher metadata based foundation and extend the crate's public api to gradually offload complexity from the caller while maintaining backwards compatibility. #8923 is example of that in practice.
However, I'm 100% open to and hoping for a better approach and already planned to refactor the Arrow
Schema->AvroSchemaconversion logic inarrow-avro/src/schema.rsanyway due to it's brittleness. (Which is why I didn't catch the missing metadata sooner lol. If there's a bug inarrow-avro, I'm 98% confident it would pop-up in that schema conversion logic, so was already expecting it).Anyway when I ran this reproducing test below in the
arrow-avro/src/schema.rsfile, with the child metadata now inserted, it passed with the reader and writer schemas matching exactly.I do think the
schema.rsconstant names + relevant docs being better / more descriptive would help here as well.Let me know if you think this resolves the issue.
CC: @alamb @nathaniel-d-ef
EmilyMatt commentedon Nov 27, 2025
Honestly I think my main interest is the cases where the ArrowSchema does not contain any metadata.
If I provide a valid arrow schema, do an AvroSchema::try_from on it, and provide it as the read_schema, I expected the two to be compatible.
Currently it crashes, which is my main issue^^
jecsand838 commentedon Nov 27, 2025
That makes sense and I 100% understand where you're coming from.
Right now
AvroSchema::try_from(&ArrowSchema)can only work with whatever information is actually present in theArrowSchema. When there’s no Avro metadata at all, the Arrow side won't contain:ns2.record2), orStructused to be a named Avro record.Because
AvroSchema::try_fromwas originally written for theWriter, missingSchemametadata was interpreted as there being no originally sourced Avro schema, therefore we were free to generate one with the information available.So for something like your nested
f1struct,AvroSchema::try_fromhas to invent a legal Avro record name. The current implementation does that by using the field name ("f1") whenavro.name/avro.namespacearen’t present. That means the generated reader schema ends up with:When you then feed that generated schema into the resolver as
read_schema, Avro resolution correctly fails with a record‑name mismatch. From Avro’s perspective those are two different types, even though the shape matches.Essentially there are two slightly different expectations here:
What the code currently guarantees:
From an Arrow schema with no/missing Avro metadata,
arrow-avrowill generate a valid Avro schema that correctly describes the Arrow layout, but we can't promise it will be compatible with a pre‑existing Avro writer schema that used specific named record types / namespaces, etc.More streamlined approach:
If the ArrowSchema is structurally compatible with the writer schema, then
AvroSchema::try_from+read_schemashould just work, even without metadata.For schemas that don’t rely on Avro named types, those two line up and things work fine. For named records (like your
record2), they diverge because Arrow simply doesn’t carry all the needed information unless we stash them in metadata.Practically speaking, the patterns available are:
read_schema(or store it inSCHEMA_METADATA_KEYon the ArrowSchema), oravro.name/avro.namespaceon the relevant struct/dictionary fields so the generated reader schema can match the writer’s names.I totally agree this is not obvious from the API, and from your perspective it just looks like you passed a perfectly valid Arrow schema and the resolution crashed. So at a minimum we should:
avro.name/avro.namespace), without breaking spec‑compliant cases.If you’re interested, we could track that as a follow‑up enhancement: something along the lines of "support using pure Arrow schemas as reader schemas without requiring Avro naming metadata", and then discuss what trade‑offs / flags that would need.
feat: Implement an AsyncReader for avro using ObjectStore (#8930)
Jefffrey commentedon Jul 29, 2026
@jecsand838 does this issue need further action or we can close as unplanned now that theres a followup?
jecsand838 commentedon Jul 29, 2026
@Jefffrey I think we can close it