Add docs for Table Maintenance in Flink by JeonDaehong · Pull Request #13360 · apache/iceberg

JeonDaehong · 2025-06-20T14:43:10Z

I have completed writing the documentation for Table Maintenance in Flink.

Please review it and kindly merge if everything looks good.

This is my first time contributing to open source documentation, so if any revisions are needed, I would appreciate your feedback. I will make the necessary changes and submit another PR accordingly.

Thank you very much.

( Docs update for Flink Table Maintenance API - Fixes #13307 )

@deprecated

…( remove @deprecated annotations no longer needed after JUnit5 migration )

…estIcebergSink.java

…cy to 1.19 and 1.20

JeonDaehong · 2025-06-20T14:44:19Z

Previous commits are coming along with my changes...
Do I need to delete my forked repository and fork it again?

JeonDaehong · 2025-06-20T14:44:40Z

This is my first time working on documentation, so I would appreciate plenty of feedback. Thank you!

pvary · 2025-06-20T15:27:41Z

+```java
+StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+TableLoader tableLoader = TableLoader.fromHadoopTable("path/to/table");
+TriggerLockFactory lockFactory = TriggerLockFactory.defaultLockFactory();


What is this?

It seems I misunderstood that part and wrote it incorrectly.
I'll remove it.

pvary · 2025-06-20T15:29:01Z

+        .partialProgressEnabled(true)
+        .partialProgressMaxCommits(10))


I would turn this off in the main example. This becomes interesting if the table is big, and otherwise it add too many small commits

pvary · 2025-06-20T15:30:13Z

+|--------|-------------|---------|
+| `uidSuffix(String)` | Unique identifier suffix for the job | Random UUID |
+| `rateLimit(Duration)` | Minimum interval between task executions | 60 seconds |
+| `lockCheckDelay(Duration)` | Delay for checking lock availability | 30 seconds |


I'm not sure that based on this doc, the average user will know what is this lock, and why it is needed

pvary · 2025-06-20T15:31:41Z

+| `scheduleOnDataFileCount(int)` | Trigger after N data files | No automatic scheduling | int |
+| `scheduleOnDataFileSize(long)` | Trigger after total data file size (bytes) | No automatic scheduling | long |
+| `scheduleOnIntervalSecond(long)` | Trigger after time interval (seconds) | No automatic scheduling | long |
+| `parallelism(int)` | Parallelism for this specific task | Inherits from TableMaintenance | int |


I think it is important to highlight that this parallelism is only for deletion

pvary · 2025-06-20T15:33:31Z

+| `scheduleOnDataFileCount(int)` | Trigger after N data files | 1000 | int |
+| `scheduleOnDataFileSize(long)` | Trigger after total data file size (bytes) | 100GB | long |
+| `scheduleOnIntervalSecond(long)` | Trigger after time interval (seconds) | 600 (10 minutes) | long |
+| `maxRewriteBytes(long)` | Maximum bytes to rewrite per execution | Long.MAX_VALUE | long |


Describe what it is used for - namely to reduce the work done in a single run if there are too many data to rewrite

pvary · 2025-06-20T15:34:52Z

+
+| Configuration Key | Description | Default Value | Type |
+|-------------------|-------------|---------------|------|
+| `iceberg.maintenance.rate-limit-seconds` | Rate limit in seconds | 60 | long |


Could this be just another column like in https://iceberg.apache.org/docs/nightly/flink-configuration/#read-options?

pvary · 2025-06-20T15:37:50Z

@JeonDaehong: Thanks for working on this.
Added some specific comments.

As a more general one:

Could we start some high level description about the feature?
Can we have some generic part where we discuss the configuration options?
We can put the full config list after these

Guosmilesmile

@JeonDaehong Thank you very much for your contribution.

Besides the points below, I have also left some comments regarding details.

I think there might be some missing explanations about the role of locks here.
For some of the more complex configuration options, I think we need to add what they are used for.
It seems that the configuration options in the RewriteDataFiles Configuration section are not fully listed.
As Peter mentioned, could we start with some high-level description of the feature?

Guosmilesmile · 2025-06-21T00:21:11Z

+| `rateLimit(Duration)` | Minimum interval between task executions | 60 seconds |
+| `lockCheckDelay(Duration)` | Delay for checking lock availability | 30 seconds |
+| `parallelism(int)` | Default parallelism for maintenance tasks | System default |
+| `maxReadBack(int)` | Max snapshots to check during initialization | 100 |


May be Maximum number of snapshots checked when started at the first time. Only available when the change stream is not provided ?

Guosmilesmile · 2025-06-21T00:59:50Z

+
+| Configuration Key | Description | Default Value | Type |
+|-------------------|-------------|---------------|------|
+| `iceberg.maintenance.rate-limit-seconds` | Rate limit in seconds | 60 | long |


Regarding the configuration items for this part, I'm a bit unfamiliar. Are these all the configuration items? And where are they used?

Guosmilesmile · 2025-06-21T01:11:24Z

+    .table(table)
+    .tableLoader(tableLoader)
+    .setAll(properties)
+    .addPostCommitTopology(


Users neither need to nor can call the addPostCommitTopology method. Compression in the sink is enabled through configuration.

Guosmilesmile · 2025-06-21T01:17:18Z

+
+| Method | Description | Default Value | Type |
+|--------|-------------|---------------|------|
+| `targetFileSizeBytes(long)` | Target size for rewritten files | Table property or 512MB | long |


Are there some configurations that we haven't listed, like minInputFiles, rewriteAll, maxFileGroupSizeBytes, and so on?

JeonDaehong · 2025-06-21T01:49:23Z

@Guosmilesmile @pvary

Thank you both so much for the feedback :D

I've carefully reviewed and read through all of your comments.
Based on your suggestions, I’ll make the necessary revisions and submit a new PR within the next two weeks!

By the way, I noticed that commit history from a previous unrelated issue is still showing up in this PR.
In this case, should I delete my forked repository and re-fork it from the upstream repo?

Guosmilesmile · 2025-06-21T02:50:32Z

By the way, I noticed that commit history from a previous unrelated issue is still showing up in this PR.
In this case, should I delete my forked repository and re-fork it from the upstream repo?

You can try to Sync fork first and update branch, then rebase the code.If you are not familiar with the process, make sure to back up your code first.

JeonDaehong · 2025-06-21T04:23:31Z

By the way, I noticed that commit history from a previous unrelated issue is still showing up in this PR.
In this case, should I delete my forked repository and re-fork it from the upstream repo?

You can try to Sync fork first and update branch, then rebase the code.If you are not familiar with the process, make sure to back up your code first.

Thank you for the guidance! I’m not very familiar with the rebase process, so I’ll review some docs and try it carefully with a backup first.

JeonDaehong · 2025-07-10T12:08:20Z

@pvary @Guosmilesmile

Sorry for the delay.

I was away for a while due to a business trip and a service launch, so I wasn't able to work on the PR.

Now that I have more time, I’ll slowly get back to working on the documentation and submit the PR again.

Thank you for your patience.

JeonDaehong · 2025-08-09T05:11:47Z

I will submit it again as a new PR.

JeonDaehong and others added 15 commits May 10, 2025 10:23

[test] - Jeondaehong: migrate Flink v2.0 tests from JUnit4 to JUnit5 …

d2d6cb5

…( remove @deprecated annotations no longer needed after JUnit5 migration )

Update flink/v2.0/flink/src/test/java/org/apache/iceberg/flink/sink/T…

5387116

…estIcebergSink.java

Update flink/v2.0/build.gradle

238714b

Merge branch 'apache:main' into main

437d1d5

Fix test timeout issue caused by shared MiniCluster

031334c

Merge branch 'apache:main' into main

ec37024

Flink: Backport the removal of MiniClusterWithClientResource dependen…

80deb49

…cy to 1.19 and 1.20

fix : typo fix

0a31e12

apply Spotless formatting to Flink 1.19 and 1.20 modules

01a68f2

add missing import for MiniClusterExtension in TestIcebergSourceFailover

f129338

Merge branch 'apache:main' into main

bb611fd

Merge branch 'apache:main' into main

622cffe

Merge branch 'apache:main' into main

1f42828

Merge branch 'apache:main' into main

435ea43

docs: add documentation for Flink Table Maintenance API

dd111f6

github-actions Bot added the docs label Jun 20, 2025

pvary reviewed Jun 20, 2025

View reviewed changes

Guosmilesmile reviewed Jun 21, 2025

View reviewed changes

JeonDaehong closed this Aug 9, 2025

JeonDaehong mentioned this pull request Aug 9, 2025

Docs: Add docs for Table Maintenance in Flink #13771

Closed

This was referenced Aug 18, 2025

Docs: Add docs for Table Maintenance in Flink #13852

Closed

Docs: Add docs for Table Maintenance in Flink #13853

Merged

		.partialProgressEnabled(true)
		.partialProgressMaxCommits(10))

Conversation

JeonDaehong commented Jun 20, 2025

Uh oh!

JeonDaehong commented Jun 20, 2025

Uh oh!

JeonDaehong commented Jun 20, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pvary commented Jun 20, 2025

Uh oh!

Guosmilesmile left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JeonDaehong commented Jun 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Guosmilesmile commented Jun 21, 2025

Uh oh!

JeonDaehong commented Jun 21, 2025

Uh oh!

JeonDaehong commented Jul 10, 2025

Uh oh!

JeonDaehong commented Aug 9, 2025

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

JeonDaehong commented Jun 21, 2025 •

edited

Loading