Skip to content

[bugfix](txn) return when txn state is null when doing abort txn - #18045

Merged
dataroaring merged 1 commit into
apache:masterfrom
ByteYue:fix_txn_state_null_when_abort_txn
Mar 23, 2023
Merged

dataroaring merged 1 commit into
apache:masterfrom
ByteYue:fix_txn_state_null_when_abort_txn

Conversation

@ByteYue

@ByteYue ByteYue commented Mar 23, 2023 •

Copy link
Copy Markdown
Contributor

Proposed changes

follower #17893
When doing broker load, if we encountered one error before the txn starts, then there would be one abort txn rpc. Unfortunately, there is no txn state for the corresponding txn, so the following procedure would throw NPE when accessing the null state.
This pr makes it just return if there is no corresponding txn state.

Problem summary

Describe your changes.

Checklist(Required)

  • Does it affect the original behavior
  • Has unit tests been added
  • Has document been added or modified
  • Does it need to update dependencies
  • Is this PR support rollback (If NO, please explain WHY)

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@ByteYue
ByteYue force-pushed the fix_txn_state_null_when_abort_txn branch from c923227 to 9b8f932 Compare March 23, 2023 06:53

@morningman morningman left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman self-assigned this Mar 23, 2023
@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Mar 23, 2023
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@ByteYue

ByteYue commented Mar 23, 2023 •

Copy link
Copy Markdown
Contributor Author

run buildall

@waywtdcc

Copy link
Copy Markdown

Is there any issue to record this error?

@ByteYue

ByteYue commented Mar 23, 2023

Copy link
Copy Markdown
Contributor Author

Is there any issue to record this error?

Hi there, you can refer to #17893 to check the log. We at first encountered this error in one of our customer's environment which has 20+ BE and 4 FE. This pr is mainly to improve #17893

@hello-stephen

Copy link
Copy Markdown
Contributor

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.65 seconds
stream load tsv: 465 seconds loaded 74807831229 Bytes, about 153 MB/s
stream load json: 24 seconds loaded 2358488459 Bytes, about 93 MB/s
stream load orc: 74 seconds loaded 1101869774 Bytes, about 14 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230323083948_clickbench_pr_119467.html

@dataroaring dataroaring left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring
dataroaring merged commit 3bb3c36 into apache:master Mar 23, 2023
@morningman morningman added usercase Important user case type label dev/1.2.4-merged and removed dev/1.2.3 labels Mar 28, 2023
mongo360 pushed a commit to mongo360/doris that referenced this pull request Jul 12, 2023
laskoviymishka added a commit to laskoviymishka/doris that referenced this pull request Oct 13, 2026
### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: Upgrade the Iceberg Java SDK from 1.11.0 to 1.12.0.
1.12.0 brings the client side of REST catalog labels (`Labels`,
`SupportsLabels`, labels on `LoadTableResponse`, see apache/iceberg#15750,
apache#18045, apache#18046), which Doris needs before it can expose catalog-provided
labels for iceberg tables. This change only upgrades the dependency and
adapts Doris to it.

- Bump `iceberg.version` 1.11.0 -> 1.12.0, and `avro.version`
  1.12.1 -> 1.12.2 to stay consistent with what iceberg 1.12.0 is built
  against (the pom requires the two to move together).
- Bump the explicit caffeine pin in fe-connector-iceberg 2.9.3 -> 3.2.4.
  Iceberg 1.12.0 moved to caffeine 3. The pin is a direct dependency, so
  it wins Maven mediation; leaving it at 2.9.3 would package an older
  caffeine into the iceberg plugin than iceberg-core is built against.
- Re-vendor `org.apache.iceberg.DeleteFileIndex` from 1.12.0 and re-apply
  the Doris changes (public class/builder methods, schemasById comment,
  equality-field-id sets). The vendored class shadows the SDK class in
  the same package, and 1.12.0 changed it (an extra `specsById` constructor
  argument and delete/data file partition-match validation), so keeping
  the 1.11.0 copy would break the SDK's own callers at runtime.
  Since Doris passes `specsById` to the builder, the cached scan planning
  path now also rejects delete files whose partition spec or tuple does
  not match the data file, the same as upstream iceberg.
- Pass `table.specs()` to `ManifestFiles.read` in `IcebergManifestCache`.
  Iceberg 1.12.0 removed the deprecated `read(ManifestFile, FileIO)`
  overload, which read the partition specs from the manifest file's own
  metadata (only possible for Avro manifests). This matches the delete
  manifest path next to it (`readDeleteManifest(..., table.specs())`)
  and `IcebergRemoveOrphanFilesAction`.
- Keep the AWS SDK at 2.29.52 and default iceberg-aws's HTTP client to
  `urlconnection`. In 1.12.0, iceberg-aws's default `http-client.type`
  (`apache`) builds an `Apache5HttpClient` from
  `software.amazon.awssdk:apache5-client`, which first shipped in AWS SDK
  2.32. Doris pins 2.29.52 on purpose (apache#58566: newer SDKs add default S3
  integrity checksums that break S3-compatible services), so with the
  default every S3FileIO, Glue and STS client iceberg builds failed with
  `NoClassDefFoundError: software/amazon/awssdk/http/apache5/Apache5HttpClient$Builder`.
  `IcebergCatalogFactory.buildBaseCatalogProperties`, the base of every
  catalog flavor, now sets `http-client.type=urlconnection` unless the
  catalog sets it. S3FileIO keeps the setting when it is serialized, so
  the BE iceberg metadata scanner, which rebuilds the FileIO from it,
  uses the same client; `url-connection-client` is added to that scanner,
  which only bundled `apache-client` and `netty-nio-client`. An explicit
  `http-client.type=apache` is rejected at CREATE/ALTER CATALOG instead of
  failing on the first S3 or Glue call.

`SchemaAwareDataTableScan` (the other vendored class) needs no change:
`DataTableScan`, `BaseTableScan`, `SnapshotScan` and `TableScanContext`
are identical between 1.11.0 and 1.12.0.

### Release note

Upgrade the Iceberg Java SDK to 1.12.0. Iceberg's AWS clients (S3FileIO,
Glue, STS) now use the URLConnection HTTP client by default, and the
`http-client.type=apache` catalog property is no longer accepted.

### Check List (For Author)

- Test: Unit Test
    - Added IcebergCatalogFactoryTest#buildBaseDefaultsIcebergAwsHttpClientToUrlConnection
      (builds a real S3 client builder with the default) and
      #buildBaseKeepsAnExplicitHttpClientType, and
      IcebergCatalogPropertiesTest#createTimeRulesRejectTheApacheHttpClientType;
      IcebergCatalogOptionsSnapshotTest snapshots now include the default.
    - `build.sh --fe` passes. Ran every iceberg FE unit test with
      run-fe-ut.sh (fe-connector-iceberg, fe-connector-metastore-iceberg,
      be-java-extensions/iceberg-metadata-scanner): 1630 tests, 0 failures,
      5 skipped.
    - Iceberg regression suites not run.
- Behavior changed: Yes.
    - Iceberg's AWS clients use the URLConnection HTTP client by default,
      so `http-client.apache.*` properties no longer apply, and
      `http-client.type=apache` is rejected at CREATE/ALTER CATALOG.
    - Iceberg scans whose delete files have a partition spec or partition
      tuple that does not match the data file now fail with a
      ValidationException (iceberg 1.12.0 behavior) instead of applying
      the deletes.
- Does this need documentation: No
laskoviymishka added a commit to laskoviymishka/doris that referenced this pull request Oct 13, 2026
### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: Upgrade the Iceberg Java SDK from 1.11.0 to 1.12.0.
1.12.0 brings the client side of REST catalog labels (`Labels`,
`SupportsLabels`, labels on `LoadTableResponse`, see apache/iceberg#15750,
apache#18045, apache#18046), which Doris needs before it can expose catalog-provided
labels for iceberg tables. This change only upgrades the dependency and
adapts Doris to it.

- Bump `iceberg.version` 1.11.0 -> 1.12.0, and `avro.version`
  1.12.1 -> 1.12.2 to stay consistent with what iceberg 1.12.0 is built
  against (the pom requires the two to move together).
- Bump the explicit caffeine pin in fe-connector-iceberg 2.9.3 -> 3.2.4.
  Iceberg 1.12.0 moved to caffeine 3. The pin is a direct dependency, so
  it wins Maven mediation; leaving it at 2.9.3 would package an older
  caffeine into the iceberg plugin than iceberg-core is built against.
- Re-vendor `org.apache.iceberg.DeleteFileIndex` from 1.12.0 and re-apply
  the Doris changes (public class/builder methods, schemasById comment,
  equality-field-id sets). The vendored class shadows the SDK class in
  the same package, and 1.12.0 changed it (an extra `specsById` constructor
  argument and delete/data file partition-match validation), so keeping
  the 1.11.0 copy would break the SDK's own callers at runtime.
  Since Doris passes `specsById` to the builder, the cached scan planning
  path now also rejects delete files whose partition spec or tuple does
  not match the data file, the same as upstream iceberg.
- Pass `table.specs()` to `ManifestFiles.read` in `IcebergManifestCache`.
  Iceberg 1.12.0 removed the deprecated `read(ManifestFile, FileIO)`
  overload, which read the partition specs from the manifest file's own
  metadata (only possible for Avro manifests). This matches the delete
  manifest path next to it (`readDeleteManifest(..., table.specs())`)
  and `IcebergRemoveOrphanFilesAction`.
- Keep the AWS SDK at 2.29.52 and default iceberg-aws's HTTP client to
  `urlconnection`. In 1.12.0, iceberg-aws's default `http-client.type`
  (`apache`) builds an `Apache5HttpClient` from
  `software.amazon.awssdk:apache5-client`, which first shipped in AWS SDK
  2.32. Doris pins 2.29.52 on purpose (apache#58566: newer SDKs add default S3
  integrity checksums that break S3-compatible services), so with the
  default every S3FileIO, Glue and STS client iceberg builds failed with
  `NoClassDefFoundError: software/amazon/awssdk/http/apache5/Apache5HttpClient$Builder`.
  `IcebergCatalogFactory.buildBaseCatalogProperties`, the base of every
  catalog flavor, now sets `http-client.type=urlconnection` unless the
  catalog sets it. S3FileIO keeps the setting when it is serialized, so
  the BE iceberg metadata scanner, which rebuilds the FileIO from it,
  uses the same client; `url-connection-client` is added to that scanner,
  which only bundled `apache-client` and `netty-nio-client`. An explicit
  `http-client.type=apache` is rejected at CREATE/ALTER CATALOG instead of
  failing on the first S3 or Glue call.

- Report the mixed-version case clearly. FE sends iceberg system-table
  splits to the BE as Java-serialized iceberg objects, and iceberg 1.12.0
  changed the serialized form of classes every split carries (`BaseFile`,
  `PartitionData`, their anonymous inner classes; 24 serializable classes
  changed `serialVersionUID`). Unlike 1.11.0, where only `Schema` changed
  and is shimmed, this cannot be bridged by remapping UIDs, so while FE and
  BE run different iceberg versions (a rolling upgrade) iceberg system
  tables cannot be read. `IcebergSerializationCompat` now turns the bare
  `InvalidClassException` / `ClassNotFoundException` into an error naming
  the cause and the BE's iceberg version, and the shim comments no longer
  claim a shared wire across versions.

`SchemaAwareDataTableScan` (the other vendored class) needs no change:
`DataTableScan`, `BaseTableScan`, `SnapshotScan` and `TableScanContext`
are identical between 1.11.0 and 1.12.0.

### Release note

Upgrade the Iceberg Java SDK to 1.12.0. Iceberg's AWS clients (S3FileIO,
Glue, STS) now use the URLConnection HTTP client by default, and the
`http-client.type=apache` catalog property is no longer accepted. During a
rolling upgrade from a release on Iceberg 1.11, Iceberg system tables
(`tbl$snapshots`, `tbl$files`, ...) are unavailable until FE and BE are
both upgraded.

### Check List (For Author)

- Test: Unit Test
    - Added IcebergCatalogFactoryTest#buildBaseDefaultsIcebergAwsHttpClientToUrlConnection
      (builds a real S3 client builder with the default) and
      #buildBaseKeepsAnExplicitHttpClientType, and
      IcebergCatalogPropertiesTest#createTimeRulesRejectTheApacheHttpClientType;
      IcebergCatalogOptionsSnapshotTest snapshots now include the default.
    - Added IcebergSerializationCompatTest#reportsATaskFromAnotherIcebergVersion
      (a DataFile carrying Iceberg 1.11's BaseFile UID) and
      #reportsAClassThisIcebergDoesNotHave.
    - Verified the mixed-version break outside the build: a DataFile written
      with iceberg 1.11.0 cannot be read with 1.12.0 and vice versa, even
      after rewriting every changed class UID in the stream.
    - `build.sh --fe` passes. Ran every iceberg FE unit test with
      run-fe-ut.sh (fe-connector-iceberg, fe-connector-metastore-iceberg,
      be-java-extensions/iceberg-metadata-scanner): 1650 tests, 0 failures,
      5 skipped.
    - Iceberg regression suites not run.
- Behavior changed: Yes.
    - Iceberg's AWS clients use the URLConnection HTTP client by default,
      so `http-client.apache.*` properties no longer apply, and
      `http-client.type=apache` is rejected at CREATE/ALTER CATALOG.
    - Iceberg scans whose delete files have a partition spec or partition
      tuple that does not match the data file now fail with a
      ValidationException (iceberg 1.12.0 behavior) instead of applying
      the deletes.
    - While FE and BE run different iceberg versions (rolling upgrade from
      iceberg 1.11), iceberg system tables fail with an explicit
      "different Iceberg version" error until both sides are upgraded.
- Does this need documentation: No
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/1.2.4-merged reviewed usercase Important user case type label

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants