Skip to content

Commit 0c1b19d

Browse files
candiduslynxcandiduslynx
authored
feat: Add use_path_style (#13551)
Fixes #13260
1 parent ed0735f commit 0c1b19d

File tree

4 files changed

+15
-4
lines changed

4 files changed

+15
-4
lines changed

‎plugins/destination/s3/client/client.go‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,7 @@ func New(ctx context.Context, logger zerolog.Logger, spec []byte, opts plugin.Ne
6767
baseEndpoint := c.spec.Endpoint
6868
o.BaseEndpoint = &baseEndpoint
6969
}
70+
o.UsePathStyle = c.spec.UsePathStyle
7071
})
7172
c.uploader = manager.NewUploader(c.s3Client)
7273
c.downloader = manager.NewDownloader(c.s3Client)

‎plugins/destination/s3/client/spec.go‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,9 @@ type Spec struct {
1818
Path string `json:"path,omitempty"`
1919
Athena bool `json:"athena,omitempty"`
2020
TestWrite *bool `json:"test_write,omitempty"`
21-
Endpoint string `json:"endpoint,omitempty"`
21+
22+
Endpoint string `json:"endpoint,omitempty"`
23+
UsePathStyle bool `json:"use_path_style,omitempty"`
2224

2325
BatchSize *int64 `json:"batch_size"`
2426
BatchSizeBytes *int64 `json:"batch_size_bytes"`

‎website/pages/docs/plugins/destinations/s3/_configuration.mdx‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,8 @@ spec:
2323
# no_rotate: false
2424
# athena: false # <- set this to true for Athena compatibility
2525
# test_write: true # tests the ability to write to the bucket before processing the data
26+
# endpoint: "" # Endpoint to use for S3 API calls.
27+
# use_path_style: false
2628
# batch_size: 10000 # 10K entries
2729
# batch_size_bytes: 52428800 # 50 MiB
2830
# batch_timeout: 30s # 30 seconds

‎website/pages/docs/plugins/destinations/s3/overview.mdx‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -67,19 +67,25 @@ This is the (nested) spec used by the CSV destination Plugin.
6767
If set to `true`, the plugin will write to one file per table.
6868
Otherwise, for every batch a new file will be created with a different `.<UUID>` suffix.
6969

70-
- `athena` (`boolean`) (optional, default `false`)
70+
- `athena` (`boolean`) (optional) (default: `false`)
7171

7272
When `athena` is set to `true`, the S3 plugin will sanitize keys in JSON columns to be compatible with the Hive Metastore / Athena.
7373
This allows tables to be created with a Glue Crawler and then queried via Athena, without changes to the table schema.
7474

75-
- `test_write` (`boolean`) (optional, default `true`)
75+
- `test_write` (`boolean`) (optional) (default: `true`)
7676

7777
Ensure write access to the given bucket and path by writing a test object on each sync.
7878
If you are sure that the bucket and path are writable, you can set this to `false` to skip the test.
7979

80-
- `endpoint` (`string`) (optional, default empty)
80+
- `endpoint` (`string`) (optional) (default: empty)
8181

8282
Endpoint to use for S3 API calls. This is useful for S3-compatible storage services such as MinIO.
83+
Note: if you want to use path-style addressing, i.e., `https://s3.amazonaws.com/BUCKET/KEY`, `use_path_style` should be enabled, too.
84+
85+
- `use_path_style` (`boolean`) (optional) (default: `false`)
86+
87+
Allows to use path-style addressing in the `endpoint` option, i.e., `https://s3.amazonaws.com/BUCKET/KEY`.
88+
By default, the S3 client will use virtual hosted bucket addressing when possible (`https://BUCKET.s3.amazonaws.com/KEY`).
8389

8490
- `batch_size` (`integer`) (optional) (default: `10000`)
8591

0 commit comments

Comments
 (0)