Docs: Add docs for Table Maintenance in Flink by JeonDaehong · Pull Request #13771 · apache/iceberg

JeonDaehong · 2025-08-09T05:37:29Z

This PR adds documentation for issue
( Docs update for Flink Table Maintenance API - Fixes #13307 )

It incorporates feedback received from the previously closed PR #13360.

Please review and let me know if any changes are needed. I will update promptly.

- Introduced comprehensive documentation for Flink Table Maintenance in Apache Iceberg. - Covers automated snapshot expiration, file compaction, and orphan file cleanup. - Includes configuration options, examples, scheduling, and best practices. - Provides lock management details and troubleshooting tips.

JeonDaehong · 2025-08-09T05:38:03Z

@pvary @Guosmilesmile
I apologize for the delay in submitting the updated PR due to personal reasons.

Guosmilesmile

@JeonDaehong Thank you very much for your PR! Regarding the documentation part, I feel there are a few points that need to be discussed:

A new md file was added but not included in mkdocs.yml. I think this page will not take effect. I feel it might be unnecessary to add a new page; instead, we could add the content to the existing flink-actions.md, distinguishing between streaming and batch. However, this is just my personal opinion, and I would like to hear @pvary Peter’s thoughts.
Could we add a section about which table maintenance operations Flink currently supports, what their purposes are, and describe the corresponding configurations within each relevant feature, rather than putting all configurations in one section?
Regarding the locking part, there is no Java API configuration provided; the configuration options given are for SQL, and SQL only uses them when compaction is enabled in sink v2. I suggest separating this and describing it in an independent section.

Guosmilesmile · 2025-08-09T14:18:45Z

+```java
+StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+TableLoader tableLoader = TableLoader.fromHadoopTable("path/to/table");
+TriggerLockFactory lockFactory = TriggerLockFactory.defaultLockFactory();


I think we do not have a method called defaultLockFactory; instead, we need to create a specific LockFactory.

Guosmilesmile · 2025-08-09T14:30:03Z

+| `scheduleOnCommitCount(int)` | Trigger after N commits | No automatic scheduling | int |
+| `scheduleOnDataFileCount(int)` | Trigger after N data files | No automatic scheduling | int |
+| `scheduleOnDataFileSize(long)` | Trigger after total data file size (bytes) | No automatic scheduling | long |
+| `scheduleOnIntervalSecond(long)` | Trigger after time interval (seconds) | No automatic scheduling | long |
+| `parallelism(int)` | Parallelism for this specific task | Inherits from TableMaintenance | int |


Can we extract the common configurations belonging to the MaintenanceTaskBuilder, instead of having them appear in each functional configuration separately?

Guosmilesmile · 2025-08-09T14:35:13Z

+
+You can also configure maintenance behavior through Flink configuration:
+
+| Configuration Key | Description | Default Value | Type |
+|-------------------|-------------|---------------|------|
+| `iceberg.maintenance.rate-limit-seconds` | Rate limit in seconds | 60 | long |
+| `iceberg.maintenance.lock-check-delay-seconds` | Lock check delay in seconds | 30 | long |
+| `iceberg.maintenance.rewrite.max-bytes` | Maximum rewrite bytes | Long.MAX_VALUE | long |
+| `iceberg.maintenance.rewrite.schedule.commit-count` | Schedule on commit count | 10 | int |
+| `iceberg.maintenance.rewrite.schedule.data-file-count` | Schedule on data file count | 1000 | int |
+| `iceberg.maintenance.rewrite.schedule.data-file-size` | Schedule on data file size | 100GB | long |
+| `iceberg.maintenance.rewrite.schedule.interval-second` | Schedule interval in seconds | 600 | long |


From my perspective, this part seems a bit strange. How is it used? I can't think of where these configurations come from.

Guosmilesmile · 2025-08-09T14:42:53Z

+
+| Method | Description | Default Value | Type |
+|--------|-------------|---------------|------|
+| `maxSnapshotAge(Duration)` | Maximum age of snapshots to retain | No limit | Duration |


Default value is 5 days ago

Guosmilesmile · 2025-08-09T14:45:04Z

+| `partialProgressEnabled(boolean)` | Enable partial progress commits | false | boolean |
+| `partialProgressMaxCommits(int)` | Maximum commits for partial progress | 10 | int |
+| `scheduleOnCommitCount(int)` | Trigger after N commits | 10 | int |
+| `scheduleOnDataFileCount(int)` | Trigger after N data files | 1000 | int |
+| `scheduleOnDataFileSize(long)` | Trigger after total data file size (bytes) | 100GB | long |
+| `scheduleOnIntervalSecond(long)` | Trigger after time interval (seconds) | 600 (10 minutes) | long |


Same note as above.

Guosmilesmile · 2025-08-09T14:51:36Z

+| Method | Description | Default Value | Type |
+|--------|-------------|---------------|------|
+| `targetFileSizeBytes(long)` | Target size for rewritten files | Table property or 512MB | long |
+| `partialProgressEnabled(boolean)` | Enable partial progress commits | false | boolean |


Perhaps "Enable partial progress commits, allowing compacted data files to be committed in batches." would be easier to understand.

Guosmilesmile · 2025-08-09T14:55:49Z

+
+
+
+### RewriteDataFiles Configuration


Some configurations are still missing from the documentation:
minFileSizeBytes
maxFileSizeBytes
minInputFiles
deleteFileThreshold
rewriteAll
maxFileGroupSizeBytes
maxFilesToRewrite
filter

Guosmilesmile · 2025-08-09T14:58:18Z

+## Scheduling Options
+
+Maintenance tasks can be triggered based on various conditions:
+
+### Time-based Scheduling
+```java
+ExpireSnapshots.builder()
+    .scheduleOnIntervalSecond(3600)
+```
+
+### Commit-based Scheduling
+```java
+RewriteDataFiles.builder()
+    .scheduleOnCommitCount(50)
+```
+
+### Data Volume-based Scheduling
+```java
+RewriteDataFiles.builder()
+    .scheduleOnDataFileCount(500)
+    .scheduleOnDataFileSize(50L * 1024 * 1024 * 1024)
+```


It would be better to integrate these with the common configuration section provided above.

Guosmilesmile · 2025-08-09T15:01:43Z

+    .addPostCommitTopology(
+        TableMaintenance.forTable(env, tableLoader, TriggerLockFactory.defaultLockFactory())
+            .rateLimit(Duration.ofMinutes(10))
+            .add(ExpireSnapshots.builder().scheduleOnCommitCount(10))
+            .add(RewriteDataFiles.builder().scheduleOnDataFileCount(50))
+    )


It is not invoked in this way; specific cases can be found here.
https://github.com/apache/iceberg/blob/main/flink/v2.0/flink/src/test/java/org/apache/iceberg/flink/sink/TestIcebergSinkCompact.java

I have reviewed it. Thank you !

Guosmilesmile · 2025-08-09T15:06:33Z

+## Troubleshooting
+
+### Common Issues
+
+**OutOfMemoryError during file deletion:**
+```java
+.deleteBatchSize(500)
+```
+
+**Lock conflicts:**
+```java
+.lockCheckDelay(Duration.ofMinutes(1))
+.rateLimit(Duration.ofMinutes(10))
+```
+
+**Slow rewrite operations:**
+```java
+.partialProgressEnabled(true)
+.partialProgressMaxCommits(3)
+.maxRewriteBytes(1024 * 1024 * 1024)
+```


What is the specific scenario here? Providing numerical recommendations directly without detailed reasoning can seem a bit odd.

JeonDaehong · 2025-08-10T00:54:53Z

@JeonDaehong Thank you very much for your PR! Regarding the documentation part, I feel there are a few points that need to be discussed:

A new md file was added but not included in mkdocs.yml. I think this page will not take effect. I feel it might be unnecessary to add a new page; instead, we could add the content to the existing flink-actions.md, distinguishing between streaming and batch. However, this is just my personal opinion, and I would like to hear @pvary Peter’s thoughts.

Could we add a section about which table maintenance operations Flink currently supports, what their purposes are, and describe the corresponding configurations within each relevant feature, rather than putting all configurations in one section?

Regarding the locking part, there is no Java API configuration provided; the configuration options given are for SQL, and SQL only uses them when compaction is enabled in sink v2. I suggest separating this and describing it in an independent section.

@Guosmilesmile
Thank you for your feedback.

Based on your comments, I will make the necessary changes and commit them again.

First, I will continue adding the content to the flink-table-maintenance.md file and update the mkdocs.yml file accordingly.

If @pvary prefers the content to be appended to flink-action.md instead, I will revise it and move it there at that time.

Thank you.

docs: add flink-table-maintenacne

Based on the maintainer's feedback, I have revised the content.

JeonDaehong · 2025-08-12T22:26:47Z

Hello :D
Sorry to keep asking when you’re busy. Would it be alright if I asked for your feedback when you have time?

@Guosmilesmile @pvary
I have made revisions based on the feedback you provided. Since this is my first time contributing directly to the docs, I appreciate your understanding if there are any shortcomings.

If you have any further feedback, I will gladly make additional updates accordingly.

Also, I have added the entry to the yml file for now. However, if @pvary suggests that it would be better to add this content to flink-actions.md, I will revise it to include it under the actions section.

Thank you always for your help! :D

Guosmilesmile · 2025-08-13T03:12:10Z

Sorry, I've been quite busy lately. I'll find some time to review it in a few days.

JeonDaehong · 2025-08-13T06:47:43Z

Sorry, I've been quite busy lately. I'll find some time to review it in a few days.

Please take a look whenever you have time. 😄
Thank you as always for your attention whenever I open a PR.

I have reviewed all the work you’ve done in your recent PRs. It seems you’ve been working quite busily lately.

I’ll also continue to study hard and contribute through PRs so that I can become a Maintainer and take part in the core development of Iceberg’s code. 😄

Guosmilesmile

Many of the configuration items need to be rechecked for their default values. Thank you.

Guosmilesmile · 2025-08-15T05:10:13Z

+jdbcProps.put("user", "flink");
+jdbcProps.put("password", "flinkpw");


I think this part should be jdbc.user and jdbc.password .

Guosmilesmile · 2025-08-15T05:18:29Z

+| `scheduleOnCommitCount(int)` | Trigger after N commits | No automatic scheduling | int |
+| `scheduleOnDataFileCount(int)` | Trigger after N data files | No automatic scheduling | int |
+| `scheduleOnDataFileSize(long)` | Trigger after total data file size (bytes) | No automatic scheduling | long |
+| `scheduleOnIntervalSecond(long)` | Trigger after time interval (seconds) | No automatic scheduling | long |


The configuration here is not complete, and there is no scheduleOnIntervalSecond . It should be scheduleOnInterval. Can we refer to https://github.com/apache/iceberg/blob/main/flink/v2.0/flink/src/main/java/org/apache/iceberg/flink/maintenance/api/MaintenanceTaskBuilder.java for adjustment?

Guosmilesmile · 2025-08-15T05:24:19Z

+| `maxRewriteBytes(long)` | Maximum bytes to rewrite per execution | Long.MAX_VALUE | long |
+
+
+### Scheduling Examples


I personally think that the "Scheduling Examples" section is not necessary. Instead, we could add a part before "Configuration Options" to explain which table maintenance features are currently supported by Flink. Each feature could be explained separately, detailing how to use it, somewhat like this: https://iceberg.incubator.apache.org/docs/nightly/spark-procedures/#expire_snapshots

Guosmilesmile · 2025-08-15T05:27:42Z

+| `retainLast(int)` | Minimum number of snapshots to retain | 1 | int |
+| `deleteBatchSize(int)` | Number of files to delete in each batch | 1000 | int |
+| `planningWorkerPoolSize(int)` | Number of worker threads for planning snapshot expiration | System default | int |
+| `cleanExpiredMetadata(boolean)` | Remove expired metadata files when expiring snapshots | true | boolean |


I think the default value is false, please help to check it agin ,thanks

Guosmilesmile · 2025-08-15T05:30:36Z

+| `minFileSizeBytes(long)` | Minimum size of files eligible for compaction | 0 | long |
+| `maxFileSizeBytes(long)` | Maximum size of files eligible for compaction | Long.MAX_VALUE | long |
+| `minInputFiles(int)` | Minimum number of files to trigger rewrite | 1 | int |
+| `deleteFileThreshold(double)` | Fraction of delete files to trigger rewrite | 0.0 | double |


Defaults to Integer.MAX_VALUE, which means this feature is not enabled by default.

Guosmilesmile · 2025-08-15T05:44:22Z

+```
+
+
+## Flink Configuration Options


I think the content of this part is incorrect, and it is not necessary.

Guosmilesmile · 2025-08-15T05:45:30Z

+IcebergSink.forRowData(dataStream)
+    .table(table)
+    .tableLoader(tableLoader)
+    .setAll(flinkConf)
+    .append();
+```
+


It would be great if we could add some examples for the SQL part.

Guosmilesmile · 2025-08-15T05:53:15Z

+flink-maintenance.lock.type=jdbc
+flink-maintenance.lock.lock-id=catalog.db.table
+flink-maintenance.lock.jdbc.uri=jdbc:postgresql://localhost:5432/iceberg
+flink-maintenance.lock.jdbc.user=flink
+flink-maintenance.lock.jdbc.password=flinkpw
+flink-maintenance.lock.jdbc.init-lock-tables=true


This is only used in SQL and is only applicable when writing with compression. I suggest presenting the configuration items in a tabular format.

Guosmilesmile · 2025-08-15T05:54:06Z

+flink-maintenance.lock.type=zookeeper
+flink-maintenance.lock.zookeeper.uri=zk://zk1:2181,zk2:2181
+flink-maintenance.lock.zookeeper.session-timeout-ms=60000
+flink-maintenance.lock.zookeeper.connection-timeout-ms=15000
+flink-maintenance.lock.zookeeper.max-retries=3
+flink-maintenance.lock.zookeeper.base-sleep-ms=3000
+```


Same note as above.

Guosmilesmile · 2025-08-15T05:54:51Z

+flink-maintenance.lock.zookeeper.max-retries=3
+flink-maintenance.lock.zookeeper.base-sleep-ms=3000
+```
+Use ZooKeeper-based locks only in high-availability or multi-process coordination environments.


I feel that this is not necessary.

JeonDaehong · 2025-08-15T05:59:05Z

Many of the configuration items need to be rechecked for their default values. Thank you.

Thank you for the feedback.
I’ll check and fix it.

I’ll also review the default values in the related classes. :D

github-actions Bot added the docs label Aug 9, 2025

Guosmilesmile reviewed Aug 9, 2025

View reviewed changes

JeonDaehong added 3 commits August 10, 2025 16:20

Merge branch 'apache:main' into main

78e16e0

docs: add flink-table-maintenacne

73b7e26

docs: add flink-table-maintenacne

Docs: Flink Table Maintenance for Iceberg (#13307)

05deff5

Based on the maintainer's feedback, I have revised the content.

Guosmilesmile reviewed Aug 15, 2025

View reviewed changes

JeonDaehong closed this by deleting the head repository Aug 18, 2025

This was referenced Aug 18, 2025

Docs: Add docs for Table Maintenance in Flink #13852

Closed

Docs: Add docs for Table Maintenance in Flink #13853

Merged

		jdbcProps.put("user", "flink");
		jdbcProps.put("password", "flinkpw");

		\| `maxRewriteBytes(long)` \| Maximum bytes to rewrite per execution \| Long.MAX_VALUE \| long \|


		### Scheduling Examples

Conversation

JeonDaehong commented Aug 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JeonDaehong commented Aug 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Guosmilesmile left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JeonDaehong commented Aug 10, 2025

Uh oh!

JeonDaehong commented Aug 12, 2025

Uh oh!

Guosmilesmile commented Aug 13, 2025

Uh oh!

JeonDaehong commented Aug 13, 2025

Uh oh!

Guosmilesmile left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JeonDaehong commented Aug 15, 2025

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

JeonDaehong commented Aug 9, 2025 •

edited

Loading

JeonDaehong commented Aug 9, 2025 •

edited

Loading