Repository navigation
Commit 31945b1
committed
[fix](paimon) Scope schema retention and fence fallback snapshots
### What problem does this PR solve?
Related PR: #67904
Problem Summary: Latest-schema retention was implicitly enabled by a schema ID,
changing ordinary metadata publication for unrelated connectors. Make retention
an explicit Paimon opt-in and remove the collateral Iceberg/Hive changes. Keep
schema and physical coordinates in one immutable statement pin, including the
fallback branch data fence, so reloads cannot silently select replacement data.
Preserve upstream snapshot-aware partition listing when integrating master.
### Release note
Preserve Paimon statement schemas and fallback data coordinates. The connector
plugin API advances to 8.0 for explicit latest-schema publication.
### Check List (For Author)
- Test: Related FE/connector suites, full SPI suite, FE build, standalone FE
Checkstyle, and Native/JNI regression queries.
- Behavior changed: Yes; only explicitly opted-in latest pins publish retained
schemas, and changed fallback generations are rejected.
- Does this need documentation: No; existing Paimon query semantics are restored.1,584 files changed
Lines changed: 63277 additions & 58843 deletions
File tree
- .github
- workflows
- be
- benchmark
- src
- agent
- cloud
- common
- metrics
- core
- column
- variant_v2
- data_type_serde
- data_type
- value/variant
- exec
- common
- exchange
- operator
- join
- pipeline
- scan
- sink/writer
- maxcompute
- sort
- exprs
- aggregate
- function
- ai
- array
- cast
- variant_v2
- lambda_function
- table_function
- format_v2
- jni
- orc
- parquet
- reader
- table
- format
- jni
- parquet
- table
- transformer
- glibc-compatibility
- information_schema
- io
- fs
- benchmark
- hdfs
- runtime/index_policy
- service
- http/action
- storage
- compaction
- index
- inverted
- analyzer
- common_grams
- query
- similarity
- token_filter
- tokenizer/ngram
- snii
- compaction
- format
- query
- internal
- reader
- stats
- writer
- iterator
- rowset
- segment
- variant
- v2
- tablet
- transform
- udf/python
- util
- test
- ai
- cloud
- core
- column
- data_type_serde
- data_type
- jsonb
- value
- data/vec/native
- exec
- common
- operator
- pipeline
- scan
- sort
- exprs
- aggregate
- function
- cast
- format_v2
- jni
- orc
- parquet
- table
- format
- native
- parquet
- table/iceberg
- io
- cache
- fs
- hdfs
- runtime/index_policy
- storage
- index
- inverted
- ananlyzer
- common_grams
- compaction
- similarity
- token_filter
- tokenizer
- snii
- bench
- compaction
- format
- query
- writer
- rowset
- segment
- test_data
- segment_flusher_format
- complex_v2_not_nullable_with_variant_bloom_horizontal_compressed
- complex_v2_not_nullable_without_variant_bloom_horizontal_compressed
- complex_v2_nullable_with_variant_bloom_vertical_uncompressed
- complex_v2_nullable_without_variant_bloom_vertical_compressed
- complex_v3_not_nullable_with_variant_bloom_vertical_compressed
- complex_v3_not_nullable_without_variant_bloom_vertical_uncompressed
- complex_v3_nullable_with_variant_bloom_horizontal_compressed
- complex_v3_nullable_without_variant_bloom_horizontal_compressed
- fixed_partial_sequence_variant_row_store_vertical
- variant_row_store_1_vertical
- variant_row_store_2_horizontal
- variant_row_store_segment_creator
- variant/legacy_v1_compaction
- transform
- variant
- testutil
- udf/python
- util
- variant
- bin
- cloud
- src
- common
- meta-service
- meta-store
- recycler
- test
- conf
- docker
- compilation
- runtime/all-in-one/4.1
- compose
- resource
- conf
- thirdparties
- docker-compose
- hudi
- scripts/create_preinstalled_scripts/hudi
- iceberg
- minio
- polaris
- sqlserver/init
- fe
- be-java-extensions
- hadoop-deps
- src/main/java/org/apache/hadoop/fs
- hadoop-hudi-scanner
- src
- main
- java/org/apache/doris/hudi
- resources
- META-INF/services
- test/java/org/apache/doris/hudi
- hive-apache-shade
- src/test/java/org/apache/doris/hive/shade
- hive-udf-shade
- iceberg-metadata-scanner
- src
- main
- java/org/apache/doris
- datasource/iceberg/s3tables
- iceberg
- resources
- META-INF/services
- test/java/org/apache/doris/iceberg
- java-common
- src
- main
- java/org/apache/doris/common
- classloader
- jdbc
- jni
- utils
- vec
- resources
- test/java/org/apache/doris/common
- classloader
- jni
- java-udf
- src
- main
- java/org/apache/doris/udf
- resources
- META-INF/services
- test/java/org/apache/doris/udf
- java-writer
- src
- main
- java/org/apache/doris/writer
- resources
- META-INF/services
- test/java/org/apache/doris/writer
- jdbc-scanner
- src
- main
- java/org/apache/doris/jdbc
- resources
- META-INF/services
- test/java/org/apache/doris/jdbc
- jni-bootstrap
- src
- main/java/org/apache/doris/jni/bootstrap
- test/java/org/apache/doris/jni
- bootstrap
- testplugin
- jni-spi
- src
- main
- java/org/apache/doris/jni/spi
- utils
- vec
- resources-filtered/META-INF/doris
- test
- java/org/apache/doris/jni/spi
- vec
- resources
- lakesoul-scanner
- src/main
- java/org/apache/doris/lakesoul
- arrow
- resources
- max-compute-connector
- src
- main
- java/org/apache/doris/maxcompute
- resources
- META-INF/services
- test/java/org/apache/doris/maxcompute
- paimon-scanner
- src
- main
- java/org/apache/doris/paimon
- resources
- META-INF/services
- test/java/org/apache/doris/paimon
- plugin-toolkit
- src
- main/java/org/apache/doris/jni/toolkit
- jdbc
- test/java/org/apache/doris/jni/toolkit/jdbc
- preload-extensions
- src/main/resources
- trino-connector-scanner
- src
- main
- java/org/apache/doris/trinoconnector
- resources
- META-INF/services
- test/java/org/apache/doris/trinoconnector
- testing
- check/checkstyle
- fe-common/src/main/java/org/apache/doris
- common
- job/cdc/request
- fe-connector
- fe-connector-adbc/src
- main/java/org/apache/doris/connector/adbc
- test/java/org/apache/doris/connector/adbc
- fe-connector-cache/src
- main/java/org/apache/doris/connector/cache
- test/java/org/apache/doris/connector/cache
- fe-connector-hive/src
- main/java/org/apache/doris/connector/hive
- test/java/org/apache/doris/connector/hive
- fe-connector-hms-hive-shade
- fe-connector-hms/src
- main/java/org/apache/doris/connector/hms
- test/java/org/apache/doris/connector/hms
- fe-connector-hudi
- src
- main/java/org/apache/doris/connector/hudi
- test/java/org/apache/doris/connector/hudi
- fe-connector-iceberg/src
- main/java/org/apache
- doris/connector/iceberg
- dlf
- iceberg
- test/java/org/apache/doris/connector/iceberg
- fe-connector-jdbc/src
- main/java/org/apache/doris/connector/jdbc/client
- test/java/org/apache/doris/connector/jdbc/client
- fe-connector-maxcompute/src
- main/java/org/apache/doris/connector/maxcompute
- test/java/org/apache/doris/connector/maxcompute
- fe-connector-paimon/src
- main/java/org/apache/doris/connector/paimon
- test/java/org/apache/doris/connector/paimon
- fe-connector-spi/src
- main/java/org/apache/doris/connector/spi
- mvcc
- test
- java/org/apache/doris/connector/spi
- mvcc
- resources
- fe-connector-trino/src/main/java/org/apache/doris/connector/trino
- fe-core/src
- main/java/org/apache/doris
- alter
- analysis
- invertedindex
- arrowflight
- auth2
- protocol
- results
- sessions
- tokens
- catalog
- stream
- cloud
- catalog
- rpc
- common/util
- connector
- datasource
- connector/converter
- doris
- jdbc/client
- metacache
- mvcc
- scan
- httpv2
- restv2
- rest
- indexpolicy
- job/extensions
- insert/streaming
- mtmv
- mtmv
- ivm
- agg
- mysql
- privilege
- protocol
- nereids
- glue/translator
- jobs/executor
- load
- processor/post
- rules
- analysis
- exploration/mv
- expression
- rules
- rewrite
- trees
- expressions/functions
- agg
- executable
- scalar
- plans
- commands
- info
- insert
- logical
- physical
- util
- planner
- qe
- protocol
- runtime
- service
- arrowflight/sessions
- statistics/analysis
- tablefunction
- tls/server
- transaction
- tso
- test
- java/org/apache/doris
- alter
- analysis
- invertedindex
- arrowflight
- auth2
- protocol
- sessions
- catalog
- cloud
- catalog
- rpc
- common
- connector
- datasource
- connector/converter
- jdbc/client
- metacache
- mvcc
- plugin
- scan
- httpv2/rest
- indexpolicy
- mtmv
- ivm
- agg
- mysql
- protocol
- nereids
- glue/translator
- load
- postprocess
- properties
- rules
- analysis
- exploration/mv
- expression
- rules
- rewrite
- trees
- expressions/functions
- agg
- executable
- scalar
- plans
- commands
- logical
- util
- planner
- qe
- protocol
- statistics/analysis
- transaction
- tso
- utframe
- resources/protocol-golden
- fe-type/src/main/java/org/apache/doris/catalog
- fs_brokers/cdc_client/src
- main/java/org/apache/doris/cdcclient
- service
- sink
- utils
- test/java/org/apache/doris/cdcclient
- itcase
- utils
- gensrc
- proto
- thrift
- regression-test
- data
- datatype_p0/nested_types/ddl
- export_p0/outfile/parquet
- external_table_p0
- iceberg
- jdbc
- paimon
- remote_doris
- tvf/insert
- external_table_p2
- hudi
- jdbc
- function_p0/cast
- inverted_index_p0
- analyzer
- storage_format
- load_p0/stream_load
- mtmv_p0/ivm
- nereids_function_p0
- array
- scalar_function
- nereids_p0
- local_shuffle
- nereids_rules_p0
- date_function_rewrite
- eliminate_aggregate_casewhen
- eliminate_gby_key
- fill_up_missing_slots
- infer_agg_not_null
- infer_predicate
- merge_one_row_relation
- push_down_top_n
- query_p0
- partition_topn
- sql_functions
- conditional_functions
- json_function
- string_functions
- table_function
- recursive_cte
- variant_p0
- compaction
- doc_mode
- schema_change
- mv
- predefine
- sql
- rqg
- schema_change
- v2
- with_index
- framework/src/main/groovy/org/apache/doris/regression/suite
- java-udf-src/src/main/java/org/apache/doris/udf
- collect
- date
- sketch
- pipeline/common
- plugins
- suites
- arrow_flight_sql_p0
- auth_p0
- cloud_p0
- cache/warm_up
- datatype_p0
- nested_types/ddl
- timestamp_ns
- ddl_p0
- create_view_nereids
- test_create_table_generated_column
- doc/sql-manual/sql-function
- export_p0
- outfile
- parquet
- export_p2
- external_table_p0
- dialect_compatible/sql/presto/scalar/timestamp
- iceberg
- jdbc
- paimon
- remote_doris
- tvf/insert
- external_table_p2
- hudi
- jdbc
- fault_injection_p0
- function_p0/cast
- inverted_index_p0
- analyzer
- load
- storage_format
- job_p0/streaming_job/cdc
- load_p0/stream_load
- mtmv_p0/ivm
- nereids_function_p0/scalar_function
- nereids_p0
- local_shuffle
- nereids_rules_p0
- column_pruning
- date_function_rewrite
- eliminate_aggregate_casewhen
- eliminate_gby_key
- fill_up_missing_slots
- infer_agg_not_null
- infer_predicate
- merge_one_row_relation
- push_down_top_n
- point_query_p0
- query_p0
- cache
- cast
- datatype
- partition_topn
- sql_functions
- conditional_functions
- json_function
- string_functions
- table_function
- recursive_cte
- schema_change_p0
- search
- sql_block_rule_p0
- variant_p0
- compaction
- doc_mode
- predefine
- schema_change
- with_index
- ext_meta
- mv
- predefine
- sql
- rqg
- schema_change
- v2
- predefine
- sql
- rqg
- schema_change
- with_index
- with_index
- variant_p1/compaction
- samples/datalake
- deltalake_and_kudu
- hudi
- iceberg_and_paimon
- thirdparty
- patches
- test
- tools/be-java-plugins
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
61 | 61 | | |
62 | 62 | | |
63 | 63 | | |
| 64 | + | |
64 | 65 | | |
65 | 66 | | |
66 | 67 | | |
| |||
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
196 | 196 | | |
197 | 197 | | |
198 | 198 | | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
199 | 204 | | |
200 | 205 | | |
201 | 206 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
60 | 60 | | |
61 | 61 | | |
62 | 62 | | |
63 | | - | |
| 63 | + | |
64 | 64 | | |
65 | | - | |
| 65 | + | |
66 | 66 | | |
67 | 67 | | |
68 | 68 | | |
| |||
711 | 711 | | |
712 | 712 | | |
713 | 713 | | |
714 | | - | |
| 714 | + | |
715 | 715 | | |
716 | 716 | | |
717 | 717 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
76 | 76 | | |
77 | 77 | | |
78 | 78 | | |
79 | | - | |
80 | | - | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
81 | 84 | | |
82 | 85 | | |
83 | 86 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
60 | 60 | | |
61 | 61 | | |
62 | 62 | | |
| 63 | + | |
63 | 64 | | |
64 | 65 | | |
65 | 66 | | |
66 | 67 | | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
67 | 73 | | |
68 | 74 | | |
69 | 75 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
636 | 636 | | |
637 | 637 | | |
638 | 638 | | |
639 | | - | |
640 | 639 | | |
641 | 640 | | |
642 | 641 | | |
| |||
686 | 685 | | |
687 | 686 | | |
688 | 687 | | |
689 | | - | |
690 | | - | |
691 | | - | |
| 688 | + | |
| 689 | + | |
692 | 690 | | |
693 | | - | |
694 | | - | |
695 | | - | |
696 | | - | |
697 | | - | |
698 | | - | |
699 | | - | |
700 | | - | |
701 | | - | |
702 | | - | |
703 | | - | |
704 | | - | |
705 | | - | |
706 | | - | |
707 | | - | |
708 | | - | |
709 | | - | |
710 | | - | |
711 | | - | |
| 691 | + | |
| 692 | + | |
| 693 | + | |
| 694 | + | |
| 695 | + | |
| 696 | + | |
| 697 | + | |
| 698 | + | |
| 699 | + | |
| 700 | + | |
| 701 | + | |
| 702 | + | |
712 | 703 | | |
713 | 704 | | |
714 | 705 | | |
| |||
1164 | 1155 | | |
1165 | 1156 | | |
1166 | 1157 | | |
| 1158 | + | |
| 1159 | + | |
| 1160 | + | |
| 1161 | + | |
1167 | 1162 | | |
1168 | 1163 | | |
1169 | 1164 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
| 29 | + | |
29 | 30 | | |
30 | 31 | | |
31 | 32 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
99 | 99 | | |
100 | 100 | | |
101 | 101 | | |
| 102 | + | |
102 | 103 | | |
103 | 104 | | |
104 | 105 | | |
| |||
2592 | 2593 | | |
2593 | 2594 | | |
2594 | 2595 | | |
| 2596 | + | |
| 2597 | + | |
| 2598 | + | |
2595 | 2599 | | |
2596 | | - | |
2597 | | - | |
2598 | | - | |
| 2600 | + | |
| 2601 | + | |
| 2602 | + | |
2599 | 2603 | | |
2600 | 2604 | | |
2601 | 2605 | | |
| |||
0 commit comments