Skip to content

Commit b78c656

Browse files
authored
[fix](iceberg) Commit every Iceberg file under the table's own location (apache#69392)
### What problem does this PR solve? Issue Number: None Related PR: apache#69082 Problem Summary: **In short.** DELETE, UPDATE and MERGE on an Iceberg table recorded the delete files they write (v2 position delete files and v3 deletion vector files) under the internal `s3://` address BE writes them at, instead of the table's own location (`oss://`, `s3a://`, `abfss://`, ...). Doris could not always read such a table back, and other engines and maintenance procedures resolved those files with the wrong file system. This PR commits delete files under the table's location. It also makes every Iceberg file Doris writes take both of its paths from one place in BE and pass one check in FE, so the next writer cannot repeat the mistake without failing. #### Background Every Iceberg sink FE sends carries two forms of the table's data location: - a normalized one BE can address the storage with (`output_path`): FE rewrites `oss://`, `cos://`, `cosn://`, `obs://`, `bos://`, `gs://`, `s3a://`, `s3n://` and Azure `abfs[s]://` / `wasb[s]://` locations (except OneLake) to `s3://`; - the raw one Iceberg metadata must record: `original_output_path` on `TIcebergTableSink`, `table_location` on `TIcebergDeleteSink`, both on `TIcebergMergeSink`. BE writes each file under the first form and reports it in `TIcebergCommitData.file_path`, which FE writes into the manifests as it is. So the reported path must be in the second form. #### The problem, and what it cost Reproduce on an Iceberg table located at `oss://bucket/...`: ```sql CREATE TABLE t (id INT, v STRING) PROPERTIES ('format-version' = '2'); INSERT INTO t VALUES (1, 'a'), (2, 'b'), (3, 'c'); DELETE FROM t WHERE id = 1; SELECT file_path FROM t$delete_files; -- s3://bucket/.../data/delete_pos_xxx.parquet SELECT file_path FROM t$data_files; -- oss://bucket/.../data/... ``` - Whether Doris can still read the table depends on the catalog and storage binding. In a real Azure ADLS Hadoop catalog test with explicit SharedKey, Doris SELECT failed with `No storage properties found for schema: s3`, and Spark failed with `UnsupportedFileSystemException: No FileSystem for scheme "s3"`. - Other engines and maintenance operations such as `expire_snapshots` / `remove_orphan_files` may resolve the wrong file system from the recorded scheme, depending on the engine and FileIO. Doris's own `remove_orphan_files` refuses to run on such a table (`Cannot determine whether listed and reachable file locations are equivalent`). - Cause: data files were written under `write_path` and committed under `original_write_path` (`VIcebergPartitionWriter`), but `VIcebergDeleteSink` had no such split; it reported the `output_path`-based path it wrote. - Why it went unnoticed: the two writers mapped the paths in different ways, `VIcebergDeleteFileWriter` reported its write path by default, and FE accepted any path. CI did run on such a location: the Iceberg REST fixture's warehouse is `s3a://`, which FE normalizes to `s3://`. But the DML suites only checked that delete file paths contain `/data/delete_pos_`, and a stray `test_iceberg_update_delete_basic.out.org` in the repository records the wrong `s3://warehouse/wh/...` paths. #### How this PR fixes it 1. `[fix](be) Commit Iceberg delete files under the table's own location`, `[test](be) Verify Iceberg delete file commit paths` and `[test](be) Initialize the Arrow memory pool in the Iceberg position delete write test`: `VIcebergDeleteSink` writes under `output_path` and commits position delete files and deletion vectors under `table_location`. An FE unit test pins that both sinks carrying `table_location` name the same directory as `output_path`, and `DataSinks.thrift` documents the contract. The position delete write test sets up the Arrow memory pool the Parquet writer allocates from, so it also passes when run alone (CI review bot finding on this PR). 2. `[refactor](be) Take both paths of every Iceberg file from one output location`: - The new `IcebergOutputLocation` (header-only) holds one directory in both forms and gives the write path and the commit path of a file name, and of a partition subdirectory. - The data writer (`VIcebergTableWriter` / `VIcebergPartitionWriter`), the delete file writer and the Puffin path all take their paths from it. `VIcebergDeleteFileWriter` now reports the commit path itself instead of its write path. - A table or delete sink without both location fields is rejected in `open()` / `init_properties()` instead of committing a write path or a relative path. The FE of every branch (3.1, 4.0, 4.1, 4.2, master) sets both fields on every Iceberg sink. - Directories join file names with exactly one `/`. 3. `[improvement](fe) Reject Iceberg commit data reported outside the sink's location`: - `planWrite` registers on the transaction the raw location(s) each sink it builds carries, read back from the built sink. - `IcebergConnectorTransaction.addCommitData`, where every report from every writer enters FE, rejects a file that is not under a registered location, or has no path. FE then answers the report as not accepted, and BE deletes the files under the paths it wrote them at and cancels the query, so a wrong path fails the statement and leaves no orphan files. - Thrift comments state the contract, including that "the same directory" can differ in more than the scheme (Azure moves the account out of the authority). Comments only, no protocol change. 4. `[test](regression) Check that every Iceberg write records its files under the table location`: - The new `test_iceberg_write_file_locations` runs INSERT, INSERT OVERWRITE PARTITION, branch INSERT, DELETE, UPDATE, MERGE and `rewrite_data_files` on a v2 table, and INSERT, DELETE, UPDATE and MERGE on a v3 table. After each operation every file in `$all_files` must start with the table's LOCATION plus `/data/`. A new write operation belongs in this suite too. - `test_iceberg_update_delete_basic` checks the prefix of its delete files as well, and the stray `.out.org` is removed. | Scenario | Before | After | | --- | --- | --- | | DELETE / UPDATE / MERGE on a v2 table at `oss://`, `s3a://`, `abfss://`, ... | position delete files recorded as `s3://...` | recorded under the table's location | | The same on a v3 table | deletion vector (Puffin) files recorded as `s3://...` | recorded under the table's location | | A BE writer reports a file outside the location its sink carries | committed as reported | the statement fails; BE deletes the file | | A sink plan without `original_output_path` / `table_location` (no supported FE sends one) | data files committed as `/p=1/x.parquet`, delete files as `s3://...` | rejected before any file is written | | `write.data.path` ending with `/` | data files at `.../data//p=1/x.parquet` | `.../data/p=1/x.parquet`, as Iceberg's own location provider places them | Delete files that an older version already committed with `s3://` keep that address until they are rewritten or expired. Upgrade BE before FE, as usual. An upgraded FE rejects the `s3://` delete file paths that a BE without this fix reports, so until that BE is upgraded, DELETE / UPDATE / MERGE on a table whose location FE normalizes fails instead of committing the wrong path. #### Classes, and how they call each other - `IcebergWritePlanProvider.planWrite` (FE) builds the sink and registers the raw location(s) it carries with the transaction (`commitLocationsOf`). - `IcebergConnectorTransaction.addCommitData` (FE) accepts a reported file only under a registered location; `IcebergWriterHelper` then turns the fragments into `DataFile` / `DeleteFile`. - `IcebergOutputLocation` (BE, new) is one directory in its write form and its commit form. - `VIcebergTableWriter` builds the data location in `open()` and hands each `VIcebergPartitionWriter` its partition directory (`WriteInfo.location`). `VIcebergDeleteSink` writes position delete files through `VIcebergDeleteFileWriter` and deletion vectors into one Puffin file, both from its location. `VIcebergMergeSink` builds one of each. ``` FE planWrite ── sink: output_path + raw location ──────────────► BE │ addCommitLocation(raw location) IcebergOutputLocation(output_path, raw location) ▼ ├─ VIcebergTableWriter ─► VIcebergPartitionWriter data files IcebergConnectorTransaction ├─ VIcebergDeleteSink ──► VIcebergDeleteFileWriter position deletes ▲ addCommitData: file_path under a registered │ └─► Puffin writer deletion vectors │ location, else reject the report └─ VIcebergMergeSink ──► both of the above └──────────── TIcebergCommitData.file_path = commit path ◄───────┘ ``` Not changed: Thrift fields (comments only), the scan side, and how FE normalizes locations.
1 parent 6802f57 commit b78c656

28 files changed

Lines changed: 956 additions & 223 deletions

‎be/src/exec/sink/viceberg_delete_sink.cpp‎

Lines changed: 28 additions & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -148,14 +148,16 @@ Status VIcebergDeleteSink::init_properties(ObjectPool* pool) {
148148
_compress_type = delete_sink.compress_type;
149149
}
150150

151-
// Get output path and table location
152-
if (delete_sink.__isset.output_path) {
153-
_output_path = delete_sink.output_path;
154-
}
155-
156-
if (delete_sink.__isset.table_location) {
157-
_table_location = delete_sink.table_location;
151+
// Every FE version sends both forms of the data location (see IcebergOutputLocation). Without
152+
// the table's own form the commit could only record the address BE wrote the file at, e.g.
153+
// s3://b/db/t/data/delete_pos_x.parquet for an oss:// table, which other engines resolve with
154+
// the wrong file system.
155+
if (delete_sink.output_path.empty() || delete_sink.table_location.empty()) {
156+
return Status::InternalError(
157+
"Iceberg delete sink must carry output_path and table_location, got '{}' and '{}'",
158+
delete_sink.output_path, delete_sink.table_location);
158159
}
160+
_output_location.emplace(delete_sink.output_path, delete_sink.table_location);
159161

160162
// Get Hadoop configuration
161163
if (delete_sink.__isset.hadoop_config) {
@@ -222,7 +224,7 @@ Status VIcebergDeleteSink::open(RuntimeState* state, RuntimeProfile* profile) {
222224

223225
LOG(INFO) << fmt::format(
224226
"VIcebergDeleteSink opened: delete_type={}, output_path={}, format_version={}",
225-
to_string(_delete_type), _output_path, _format_version);
227+
to_string(_delete_type), _t_sink.iceberg_delete_sink.output_path, _format_version);
226228

227229
return Status::OK();
228230
}
@@ -481,18 +483,16 @@ Status VIcebergDeleteSink::_collect_position_deletes(
481483

482484
Status VIcebergDeleteSink::_write_position_delete_files(
483485
const std::map<std::string, IcebergFileDeletion>& file_deletions) {
486+
DORIS_CHECK(_output_location.has_value());
484487
constexpr size_t kBatchSize = 4096;
485488
for (const auto& [data_file_path, deletion] : file_deletions) {
486489
if (deletion.rows_to_delete.isEmpty()) {
487490
continue;
488491
}
489-
// Generate unique delete file path
490-
std::string delete_file_path = _generate_delete_file_path(data_file_path);
491-
492492
// Create delete file writer
493493
auto writer = VIcebergDeleteFileWriterFactory::create_writer(
494-
TFileContent::POSITION_DELETES, delete_file_path, _file_format_type,
495-
_compress_type);
494+
TFileContent::POSITION_DELETES, *_output_location,
495+
_delete_file_name(data_file_path), _file_format_type, _compress_type);
496496

497497
// Build column names for position delete
498498
std::vector<std::string> column_names = {"file_path", "pos"};
@@ -506,7 +506,7 @@ Status VIcebergDeleteSink::_write_position_delete_files(
506506
writer->open(_state, _state->runtime_profile(), _position_delete_output_expr_ctxs,
507507
column_names, _hadoop_conf, _file_type, _broker_addresses);
508508
if (writer->file_system() != nullptr) {
509-
_created_files.emplace_back(writer->file_system(), delete_file_path);
509+
_created_files.emplace_back(writer->file_system(), writer->write_path());
510510
}
511511
RETURN_IF_ERROR(open_status);
512512

@@ -543,7 +543,7 @@ Status VIcebergDeleteSink::_write_position_delete_files(
543543
_delete_file_count++;
544544

545545
VLOG(1) << fmt::format("Written position delete file: path={}, rows={}, referenced_file={}",
546-
delete_file_path, commit_data.row_count, data_file_path);
546+
writer->write_path(), commit_data.row_count, data_file_path);
547547
}
548548

549549
return Status::OK();
@@ -643,6 +643,7 @@ std::string VIcebergDeleteSink::_get_file_extension() const {
643643

644644
Status VIcebergDeleteSink::_write_deletion_vector_files(
645645
const std::map<std::string, IcebergFileDeletion>& file_deletions) {
646+
DORIS_CHECK(_output_location.has_value());
646647
std::vector<DeletionVectorBlob> blobs;
647648
for (const auto& [data_file_path, deletion] : file_deletions) {
648649
if (deletion.rows_to_delete.isEmpty()) {
@@ -687,13 +688,15 @@ Status VIcebergDeleteSink::_write_deletion_vector_files(
687688
return Status::OK();
688689
}
689690

690-
std::string puffin_path = _generate_puffin_file_path();
691+
const std::string puffin_file_name = _puffin_file_name();
691692
int64_t puffin_file_size = 0;
692-
RETURN_IF_ERROR(_write_puffin_file(puffin_path, &blobs, &puffin_file_size));
693+
RETURN_IF_ERROR(_write_puffin_file(_output_location->write_path(puffin_file_name), &blobs,
694+
&puffin_file_size));
695+
const std::string commit_path = _output_location->commit_path(puffin_file_name);
693696

694697
for (const auto& blob : blobs) {
695698
TIcebergCommitData commit_data;
696-
commit_data.__set_file_path(puffin_path);
699+
commit_data.__set_file_path(commit_path);
697700
commit_data.__set_row_count(blob.merged_count);
698701
commit_data.__set_affected_rows(blob.delete_count);
699702
commit_data.__set_file_size(puffin_file_size);
@@ -803,37 +806,15 @@ std::string VIcebergDeleteSink::_build_puffin_footer_json(
803806
return {buffer.GetString(), buffer.GetSize()};
804807
}
805808

806-
std::string VIcebergDeleteSink::_generate_delete_file_path(
807-
const std::string& referenced_data_file) {
808-
// Generate unique delete file name using UUID
809-
std::string uuid = generate_uuid_string();
810-
std::string file_name;
811-
812-
std::string file_extension = _get_file_extension();
813-
file_name =
814-
fmt::format("delete_pos_{}_{}{}", uuid,
815-
std::hash<std::string> {}(referenced_data_file) % 10000000, file_extension);
816-
817-
// Combine with output path or table location
818-
std::string base_path = _output_path.empty() ? _table_location : _output_path;
819-
820-
// Ensure base path ends with /
821-
if (!base_path.empty() && base_path.back() != '/') {
822-
base_path += '/';
823-
}
824-
825-
// Delete files are data files in Iceberg, write under data location
826-
return fmt::format("{}{}", base_path, file_name);
809+
std::string VIcebergDeleteSink::_delete_file_name(const std::string& referenced_data_file) const {
810+
// Delete files live in the data location itself, next to the partition directories.
811+
return fmt::format("delete_pos_{}_{}{}", generate_uuid_string(),
812+
std::hash<std::string> {}(referenced_data_file) % 10000000,
813+
_get_file_extension());
827814
}
828815

829-
std::string VIcebergDeleteSink::_generate_puffin_file_path() {
830-
std::string uuid = generate_uuid_string();
831-
std::string file_name = fmt::format("delete_dv_{}.puffin", uuid);
832-
std::string base_path = _output_path.empty() ? _table_location : _output_path;
833-
if (!base_path.empty() && base_path.back() != '/') {
834-
base_path += '/';
835-
}
836-
return fmt::format("{}{}", base_path, file_name);
816+
std::string VIcebergDeleteSink::_puffin_file_name() const {
817+
return fmt::format("delete_dv_{}.puffin", generate_uuid_string());
837818
}
838819

839820
} // namespace doris

‎be/src/exec/sink/viceberg_delete_sink.h‎

Lines changed: 7 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -24,12 +24,14 @@
2424
#include <cstdint>
2525
#include <map>
2626
#include <memory>
27+
#include <optional>
2728
#include <string>
2829
#include <vector>
2930

3031
#include "common/status.h"
3132
#include "core/block/block.h"
3233
#include "exec/sink/writer/async_result_writer.h"
34+
#include "exec/sink/writer/iceberg/iceberg_output_location.h"
3335
#include "exec/sink/writer/iceberg/viceberg_delete_file_writer.h"
3436
#include "exprs/vexpr_fwd.h"
3537
#include "roaring/roaring64map.hh"
@@ -114,12 +116,9 @@ class VIcebergDeleteSink final : public AsyncResultWriter {
114116

115117
std::string _build_puffin_footer_json(const std::vector<DeletionVectorBlob>& blobs);
116118

117-
std::string _generate_puffin_file_path();
118-
119-
/**
120-
* Generate unique delete file path
121-
*/
122-
std::string _generate_delete_file_path(const std::string& referenced_data_file = "");
119+
// Unique file names, relative to the data location (_output_location).
120+
std::string _puffin_file_name() const;
121+
std::string _delete_file_name(const std::string& referenced_data_file) const;
123122

124123
/**
125124
* Get $row_id column index from block
@@ -163,9 +162,8 @@ class VIcebergDeleteSink final : public AsyncResultWriter {
163162
TFileFormatType::type _file_format_type = TFileFormatType::FORMAT_PARQUET;
164163
TFileCompressType::type _compress_type = TFileCompressType::SNAPPYBLOCK;
165164

166-
// Output directory for delete files
167-
std::string _output_path;
168-
std::string _table_location;
165+
// The data location delete files are written to and committed under, set by init_properties().
166+
std::optional<IcebergOutputLocation> _output_location;
169167

170168
TFileType::type _file_type = TFileType::FILE_HDFS;
171169
std::vector<TNetworkAddress> _broker_addresses;
Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
#pragma once
19+
20+
#include <string>
21+
#include <string_view>
22+
#include <utility>
23+
24+
#include "common/check.h"
25+
26+
namespace doris {
27+
28+
// The directory an Iceberg writer puts its files in, in both forms of the same location. BE
29+
// creates, writes and deletes a file under write_dir, the form BE addresses the storage with.
30+
// Iceberg metadata records the file under commit_dir, the table's own form, which every engine
31+
// reading the table resolves. FE sends both: TIcebergTableSink output_path / original_output_path
32+
// and TIcebergDeleteSink output_path / table_location. Example: an OSS table with data location
33+
// oss://b/db/t/data reaches BE as write_dir s3://b/db/t/data and commit_dir oss://b/db/t/data, so
34+
// delete_pos_1.parquet is written to s3://b/db/t/data/delete_pos_1.parquet and committed as
35+
// oss://b/db/t/data/delete_pos_1.parquet.
36+
//
37+
// Every file a writer reports takes both of its paths from one IcebergOutputLocation, so it
38+
// cannot commit the form it wrote: delete files used to be committed as s3://..., and other
39+
// engines resolved them with a file system the table is not on.
40+
class IcebergOutputLocation {
41+
public:
42+
IcebergOutputLocation(std::string write_dir, std::string commit_dir)
43+
: _write_dir(std::move(write_dir)), _commit_dir(std::move(commit_dir)) {
44+
DORIS_CHECK(!_write_dir.empty());
45+
DORIS_CHECK(!_commit_dir.empty());
46+
}
47+
48+
// The same subdirectory in both forms, e.g. the partition directory "p=1/q=2".
49+
IcebergOutputLocation subdirectory(std::string_view relative_dir) const {
50+
DORIS_CHECK(!relative_dir.empty());
51+
return {_join(_write_dir, relative_dir), _join(_commit_dir, relative_dir)};
52+
}
53+
54+
// The path BE writes file_name at.
55+
std::string write_path(std::string_view file_name) const {
56+
return _join(_write_dir, file_name);
57+
}
58+
59+
// The path Iceberg metadata records file_name at.
60+
std::string commit_path(std::string_view file_name) const {
61+
return _join(_commit_dir, file_name);
62+
}
63+
64+
private:
65+
// FE takes write.data.path verbatim, so a directory may end with '/'. Both forms join with
66+
// exactly one separator, as Iceberg's own location provider does: oss://b/t/data and
67+
// oss://b/t/data/ both join "p=1" into oss://b/t/data/p=1.
68+
static std::string _join(const std::string& dir, std::string_view name) {
69+
std::string path = dir;
70+
if (path.back() != '/') {
71+
path.push_back('/');
72+
}
73+
path.append(name);
74+
return path;
75+
}
76+
77+
std::string _write_dir;
78+
std::string _commit_dir;
79+
};
80+
81+
} // namespace doris

‎be/src/exec/sink/writer/iceberg/viceberg_delete_file_writer.cpp‎

Lines changed: 10 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -44,11 +44,13 @@ std::unique_ptr<iceberg::Schema> build_position_delete_schema() {
4444
}
4545

4646
VIcebergDeleteFileWriter::VIcebergDeleteFileWriter(TFileContent::type delete_type,
47-
const std::string& output_path,
47+
const IcebergOutputLocation& location,
48+
const std::string& file_name,
4849
TFileFormatType::type file_format,
4950
TFileCompressType::type compress_type)
5051
: _delete_type(delete_type),
51-
_output_path(output_path),
52+
_write_path(location.write_path(file_name)),
53+
_commit_path(location.commit_path(file_name)),
5254
_file_format(file_format),
5355
_compress_type(compress_type) {}
5456

@@ -74,7 +76,7 @@ Status VIcebergDeleteFileWriter::open(RuntimeState* state, RuntimeProfile* profi
7476
fs_properties.broker_addresses = &broker_addresses;
7577
}
7678

77-
io::FileDescription file_description = {.path = _output_path, .fs_name {}};
79+
io::FileDescription file_description = {.path = _write_path, .fs_name {}};
7880
_fs = DORIS_TRY(FileFactory::create_fs(fs_properties, file_description));
7981

8082
io::FileWriterOptions file_writer_options = {.used_by_s3_committer = false};
@@ -131,7 +133,7 @@ Status VIcebergDeleteFileWriter::close(TIcebergCommitData& commit_data) {
131133
}
132134

133135
// Fill commit data (use __set_ to mark optional fields as present)
134-
commit_data.__set_file_path(_output_path);
136+
commit_data.__set_file_path(_commit_path);
135137
commit_data.__set_row_count(_written_rows);
136138
commit_data.__set_affected_rows(_written_rows);
137139
commit_data.__set_file_size(_file_size);
@@ -152,9 +154,10 @@ Status VIcebergDeleteFileWriter::close(TIcebergCommitData& commit_data) {
152154

153155
// Factory method
154156
std::unique_ptr<VIcebergDeleteFileWriter> VIcebergDeleteFileWriterFactory::create_writer(
155-
TFileContent::type delete_type, const std::string& output_path,
156-
TFileFormatType::type file_format, TFileCompressType::type compress_type) {
157-
return std::make_unique<VIcebergDeleteFileWriter>(delete_type, output_path, file_format,
157+
TFileContent::type delete_type, const IcebergOutputLocation& location,
158+
const std::string& file_name, TFileFormatType::type file_format,
159+
TFileCompressType::type compress_type) {
160+
return std::make_unique<VIcebergDeleteFileWriter>(delete_type, location, file_name, file_format,
158161
compress_type);
159162
}
160163

‎be/src/exec/sink/writer/iceberg/viceberg_delete_file_writer.h‎

Lines changed: 13 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@
2323

2424
#include "common/status.h"
2525
#include "core/block/block.h"
26+
#include "exec/sink/writer/iceberg/iceberg_output_location.h"
2627
#include "exprs/vexpr_fwd.h"
2728
#include "gen_cpp/DataSinks_types.h"
2829
#include "io/fs/file_writer.h"
@@ -50,8 +51,10 @@ class VFileFormatTransformer;
5051
*/
5152
class VIcebergDeleteFileWriter {
5253
public:
53-
VIcebergDeleteFileWriter(TFileContent::type delete_type, const std::string& output_path,
54-
TFileFormatType::type file_format,
54+
// Writes file_name under location; the commit data reports the file under the table's own form
55+
// of that directory, not the form it was written at.
56+
VIcebergDeleteFileWriter(TFileContent::type delete_type, const IcebergOutputLocation& location,
57+
const std::string& file_name, TFileFormatType::type file_format,
5558
TFileCompressType::type compress_type);
5659

5760
~VIcebergDeleteFileWriter();
@@ -96,9 +99,13 @@ class VIcebergDeleteFileWriter {
9699

97100
std::shared_ptr<io::FileSystem> file_system() const { return _fs; }
98101

102+
// The path the file is written at, which is what cleanup must delete.
103+
const std::string& write_path() const { return _write_path; }
104+
99105
private:
100106
TFileContent::type _delete_type;
101-
std::string _output_path;
107+
std::string _write_path;
108+
std::string _commit_path;
102109
TFileFormatType::type _file_format;
103110
TFileCompressType::type _compress_type = TFileCompressType::SNAPPYBLOCK;
104111

@@ -122,8 +129,9 @@ class VIcebergDeleteFileWriter {
122129
class VIcebergDeleteFileWriterFactory {
123130
public:
124131
static std::unique_ptr<VIcebergDeleteFileWriter> create_writer(
125-
TFileContent::type delete_type, const std::string& output_path,
126-
TFileFormatType::type file_format, TFileCompressType::type compress_type);
132+
TFileContent::type delete_type, const IcebergOutputLocation& location,
133+
const std::string& file_name, TFileFormatType::type file_format,
134+
TFileCompressType::type compress_type);
127135
};
128136

129137
} // namespace doris

‎be/src/exec/sink/writer/iceberg/viceberg_partition_writer.cpp‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -93,7 +93,7 @@ Status VIcebergPartitionWriter::open(RuntimeState* state, RuntimeProfile* profil
9393
if (!_write_info.broker_addresses.empty()) {
9494
fs_properties.broker_addresses = &(_write_info.broker_addresses);
9595
}
96-
_path = fmt::format("{}/{}", _write_info.write_path, _get_target_file_name());
96+
_path = _write_info.location.write_path(_get_target_file_name());
9797
io::FileDescription file_description = {.path = _path, .fs_name {}};
9898
_fs = DORIS_TRY(FileFactory::create_fs(fs_properties, file_description));
9999
io::FileWriterOptions file_writer_options = {.used_by_s3_committer = false};
@@ -210,8 +210,7 @@ Status VIcebergPartitionWriter::_build_iceberg_commit_data(TIcebergCommitData* c
210210
DCHECK(commit_data != nullptr);
211211
DCHECK(_file_format_transformer != nullptr);
212212

213-
commit_data->__set_file_path(
214-
fmt::format("{}/{}", _write_info.original_write_path, _get_target_file_name()));
213+
commit_data->__set_file_path(_write_info.location.commit_path(_get_target_file_name()));
215214
commit_data->__set_row_count(_row_count);
216215
commit_data->__set_affected_rows(_row_count);
217216
commit_data->__set_file_size(_file_format_transformer->written_len());

0 commit comments

Comments
 (0)