Merge upstream apache/iceberg main - #39

Bumps [zizmorcore/zizmor-action](https://github.com/zizmorcore/zizmor-action) from 0.6.1 to 0.6.2.
- [Release notes](https://github.com/zizmorcore/zizmor-action/releases)
- [Commits](zizmorcore/zizmor-action@6fc4b00...3dc1ecc)

---
updated-dependencies:
- dependency-name: zizmorcore/zizmor-action
  dependency-version: 0.6.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…7574)

Bumps [gradle/actions/setup-gradle](https://github.com/gradle/actions) from 6.2.0 to 6.3.0.
- [Release notes](https://github.com/gradle/actions/releases)
- [Commits](gradle/actions@3f131e8...9c97196)

---
updated-dependencies:
- dependency-name: gradle/actions/setup-gradle
  dependency-version: 6.3.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [actions/setup-java](https://github.com/actions/setup-java) from 5.6.0 to 5.7.0.
- [Release notes](https://github.com/actions/setup-java/releases)
- [Commits](actions/setup-java@03ad4de...b6effb0)

---
updated-dependencies:
- dependency-name: actions/setup-java
  dependency-version: 5.7.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…o 8.9.0 (apache#17569)

Build: Bump com.diffplug.spotless:spotless-plugin-gradle

Bumps [com.diffplug.spotless:spotless-plugin-gradle](https://github.com/diffplug/spotless) from 8.8.0 to 8.9.0.
- [Release notes](https://github.com/diffplug/spotless/releases)
- [Changelog](https://github.com/diffplug/spotless/blob/main/CHANGES.md)
- [Commits](diffplug/spotless@gradle/8.8.0...gradle/8.9.0)

---
updated-dependencies:
- dependency-name: com.diffplug.spotless:spotless-plugin-gradle
  dependency-version: 8.9.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps the codeql-action group with 3 updates: [github/codeql-action/init](https://github.com/github/codeql-action), [github/codeql-action/analyze](https://github.com/github/codeql-action) and [github/codeql-action/upload-sarif](https://github.com/github/codeql-action).


Updates `github/codeql-action/init` from 4.37.3 to 4.37.4
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](github/codeql-action@e4fba86...f205ea1)

Updates `github/codeql-action/analyze` from 4.37.3 to 4.37.4
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](github/codeql-action@e4fba86...f205ea1)

Updates `github/codeql-action/upload-sarif` from 4.37.3 to 4.37.4
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](github/codeql-action@e4fba86...f205ea1)

---
updated-dependencies:
- dependency-name: github/codeql-action/init
  dependency-version: 4.37.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: codeql-action
- dependency-name: github/codeql-action/analyze
  dependency-version: 4.37.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: codeql-action
- dependency-name: github/codeql-action/upload-sarif
  dependency-version: 4.37.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: codeql-action
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…ache#17482)

…#17437)

…d 1.20 (apache#17437) (apache#17590)

…e#16789)

…he#17445) (apache#17591)

…er (apache#17606)

Backports apache#16789

apache#17497)

Motivation: Scan planning responses index delete files by position in the delete-files array.
The index map was keyed on the delete file location alone, but a single Puffin file can hold a deletion vector for each of several data files. Every DV in that file shares a location, so each new DV entry overwrote the previous one and every task referencing a DV in that Puffin file resolved to the same last index.

Changes: Extract DeleteFileSet's private wrapper into a public DeleteFileWrapper, mirroring CharSequenceWrapper, and key the index map on it so serialization uses the same delete file identity that builds the delete-files array.
Fail with a clear message when a task references a delete file that is not in that array.

…#17554)

Add read.split.adaptive-size.enabled, read.data-planning-mode, and read.delete-planning-mode to the read properties table with their defaults and accepted values.

…nerationUtil (apache#17391)

* Core: Add nested field traversal in AppendBenchmark and FileGenerationUtil

* expand testBoundsForNestedTypes to cover all metricsMode

* Core: Resolve metrics mode by field id in FileGenerationUtil

Use MetricsConfig.columnMode(int) instead of manually resolving the
column name via Schema.findColumnName. The int overload performs the
same id-to-name lookup internally and correctly handles column aliases,
removing duplicated logic across the generator and its test.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* fix schema used in UT

---------

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

…17393)

…#17544)

max-file-group-input-files is read and validated by
SizeBasedFileRewritePlanner but was missing from validOptions(), so
RewriteDataFilesSparkAction rejected it with "Cannot use options
[max-file-group-input-files]". Flink exposes the option through
RewriteDataFiles.maxFileGroupInputFiles() where it works, while Spark
rejected it. The option was added in apache#14837 without updating validOptions().

Add the option to validOptions() and cover it with a rewrite that sets it,
which fails without this change, plus a planner test that it caps the number
of input files per group.

…17592)

* Respect plaintext files in EncyptingFileIO bulkDecrypt

* unit tests

---------

Co-authored-by: Thomas Powell <[email protected]>

…17125)

AvroEncoderUtil.decode validated the magic bytes with
Preconditions.checkState(cond, "Unrecognized header bytes: 0x%02X 0x%02X",
header0, header1). Guava's Preconditions formats messages with
Strings.lenientFormat, which only understands the %s placeholder, so the
%02X specifiers were emitted verbatim and the two byte arguments were
appended as decimal in a trailing bracket. A mismatched header therefore
reported "Unrecognized header bytes: 0x%02X 0x%02X [16, 32]" instead of the
intended hex, obscuring the actual bytes.

…pache#17624)

Thanks @eye-gu for the Fix!

…d SparkSchemaUtil (apache#17626)

Spark: Remove deprecated methods scheduled for 1.12.0 removal

- SparkSchemaUtil.convertWithFreshIds(Schema, StructType) and
  prune(Schema, StructType, List<Expression>) in Spark 4.0 and 4.1
- SparkReadConf/SparkWriteConf constructors taking an unused branch
  argument in Spark 4.1

Spark 3.5 is unaffected: these overloads were never deprecated there.

…he#17640)

…) in ManifestFilterManager (apache#15713)

…17424)

spark/v4.2 arrived with the upstream merge and has no Vortex integration:
no iceberg-vortex dependency, and its SparkFormatModels registers no
Vortex model (0 Vortex references, against 10 in v3.5, v4.0 and v4.1).

Its test classes nonetheless inherit parameter matrices from the shared
iceberg-data test bases, which the fork hardcoded to include VORTEX. The
TCK therefore asks for a Vortex writer that was never registered:

  java.lang.IllegalArgumentException: Format model is not registered for
  format VORTEX and type class org.apache.spark.sql.catalyst.InternalRow
      at FormatModelRegistry.modelFor(FormatModelRegistry.java:198)
      at RegistryBasedFileWriterFactory.newDataWriter(...)
      at TestFileWriterFactory.writeData(TestFileWriterFactory.java:348)

Four classes inherit a Vortex-parameterised base, not just the one that
surfaced: TestSparkFileWriterFactory, TestSparkRollingFileWriters,
TestSparkPositionDeltaWriters and TestSparkPartitioningWriters. All four
now override parameters() to drop VORTEX, which is the pattern the fork
already uses for the Flink modules -- they have no Vortex integration
either and exclude it the same way.

This is a stopgap for the test matrix, not a fix for the underlying gap:
porting the Vortex Spark integration to 4.2 is separate work, and until
it lands v4.2 simply has no Vortex support to exercise.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01GxC1sZXqWV6UibNP9qaD9w

spark/v4.2 arrived with the upstream merge without any Vortex
integration, which left the fork's Vortex Spark code out of the default
build entirely (upstream moved defaultSparkVersions to 4.2).

Upstream's v4.2 is byte-identical to v4.1 for every main source the fork
modifies -- SparkBatch, BaseBatchReader, BatchDataReader,
SparkColumnarReaderFactory and SparkFormatModels all have zero drift --
so this is the fork's v4.1 Vortex delta replayed onto v4.2 paths, which
applied without conflicts.

New main sources: SparkVortexReader, SparkVortexValueReaders,
SparkVortexWriter, VectorizedSparkVortexReaders, VortexArrowColumnVector
and VortexBatchReadConf. SparkFormatModels now registers the Vortex model
for InternalRow and ColumnarBatch, matching v4.1.

build.gradle gains the iceberg-vortex dependency (excluding vortex-jni so
the shaded ':all' artifact wins), vortex-spark213, vortex-jni:all, the
Arrow 19 compileOnly that VortexArrowColumnVector needs for view vectors,
and the shadow-jar relocation exclusions for the Arrow C-data JNI classes.

Vortex test coverage comes across too: TestSparkVortexReader,
TestSparkVortexUnknown, TestSparkVortexReaderDeletes,
TestSparkVortexVariantRead, TestVortexVectorizedRowLineage,
TestVortexVectorizedTypePromotion and TestVortexMergeOnReadDelete.

This reverts the parameter-matrix exclusions added in the previous commit:
v4.2 now has a Vortex model to exercise, so the shared test bases should
run their VORTEX parameters rather than skip them.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01GxC1sZXqWV6UibNP9qaD9w

Adding Vortex to spark/v4.2 shifts both generated baselines, which until
now only covered v3.5, v4.0 and v4.1.

runtime-deps.txt goes from 40 to 46 entries, matching v4.1: vortex-jni
and vortex-spark_2.13 arrive, Arrow moves from 15.0 to the 19.0 set that
vortex-jni resolves (adding arrow-c-data and
arrow-memory-netty-buffer-patch), flatbuffers moves to 25.2, unshaded
guava appears via vortex-spark, and eclipse-collections drops out.

baseline-class-uniqueness.lock files are new for both the spark and
spark-runtime projects, recording the duplicate classes vortex-jni's
shaded jar shares with netty, arrow-c-data and RoaringBitmap -- the same
entries the other Spark versions already carry.

Both were produced by their generators (generateRuntimeDeps and
checkClassUniqueness --write-locks), not written by hand.
checkRuntimeDeps and checkClassUniqueness pass on v4.2.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01GxC1sZXqWV6UibNP9qaD9w