-
Notifications
You must be signed in to change notification settings - Fork 8.3k
Avro Schema Evolution Support #11986
Copy link
Copy link
Closed
Labels
comp-formatsInput/output formats (CSV/JSON/Parquet/ORC/Arrow/Protobuf/etc.).Input/output formats (CSV/JSON/Parquet/ORC/Arrow/Protobuf/etc.).questionQuestion?Question?question-answered
Description
I use AvroConfulent data format with schema registry to consume Kafka events to clickhouse.
- сurrently, Avro schemas are cached once resolved
- after evolving Avro schema(add fields) clickhouse doesn't fetch new schema and failed with an error on SELECT raws from Kafka table
- recreate Kafka table, change format_avro_schema_registry_url and restart clickhouse server doesn't refresh the cache
- if I create the same Table on another Kafka topic with the same format everything works correctly, and I guess that schema cache works by schema name
As I understand these lines of code and comments in PR, fetch a new Avro schema should work automatically:
https://github.com/ClickHouse/ClickHouse/pull/8953/files/927e572d39432d22ae96e087674a5124c6e2931b#diff-4c0061c616a674630b9e2e74706e6255R61
#8571
AvroConfluent only caches schemas per an instance of an InputFormat. This means SchemaRegistry will be queried each time a batch from Kafka is processed.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
comp-formatsInput/output formats (CSV/JSON/Parquet/ORC/Arrow/Protobuf/etc.).Input/output formats (CSV/JSON/Parquet/ORC/Arrow/Protobuf/etc.).questionQuestion?Question?question-answered