Skip to content

Commit 1e7edab

Browse files
HaochengLIUthisisnic
authored andcommitted
apacheGH-41973: Expose new S3 option check_directory_existence_before_creation
1 parent 7f645d4 commit 1e7edab

8 files changed

Lines changed: 31 additions & 11 deletions

File tree

‎r/NEWS.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,8 @@ post](https://arrow.apache.org/blog/2025/02/16/19.0.1-release/) and
5353
* Fix crash in ParquetFileWriter$WriteTable and add WriteBatch (#42241)
5454
* Fix bindings in Math group generics (@aboyoun, #43162)
5555
* Fix pull on a grouped query returns the wrong column (#43172)
56+
# arrow 17.0.0.9000
57+
* Expose an option `check_directory_existence_before_creation` in `S3FileSystem` which defaults to `FALSE`. If it's set to false, when creating a directory the code will not check if it already exists or not. It's an optimization to try directory creation and catch the error, rather than issue two dependent I/O calls. If set to `TRUE`, when creating a directory the code will only create the directory when necessary at the cost of extra I/O calls. This can be used for key/value cloud storage which has a hard rate limit to number of object mutation operations or scenarios such as the directories already exist and you do not have creation access.
5658

5759
# arrow 17.0.0
5860

‎r/R/arrowExports.R‎

Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎r/R/filesystem.R‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -156,6 +156,13 @@ FileSelector$create <- function(base_dir, allow_not_found = FALSE, recursive = F
156156
#' buckets if `$CreateDir()` is called on the bucket level (default `FALSE`).
157157
#' - `allow_bucket_deletion`: logical, if TRUE, the filesystem will delete
158158
#' buckets if`$DeleteDir()` is called on the bucket level (default `FALSE`).
159+
#' - `check_directory_existence_before_creation`: logical, if `FALSE`, when creating a directory the code will
160+
#' . not check if it already exists or not. It's an optimization to try directory creation and catch the error,
161+
#' rather than issue two dependent I/O calls.
162+
#' if `TRUE`, when creating a directory the code will only create the directory when necessary
163+
#' at the cost of extra I/O calls. This can be used for key/value cloud storage which has
164+
#' a hard rate limit to number of object mutation operations or scenarios such as
165+
#' the directories already exist and you do not have creation access (default `FALSE`).
159166
#' - `request_timeout`: Socket read time on Windows and macOS in seconds. If
160167
#' negative, the AWS SDK default (typically 3 seconds).
161168
#' - `connect_timeout`: Socket connection timeout in seconds. If negative, AWS
@@ -411,7 +418,8 @@ S3FileSystem$create <- function(anonymous = FALSE, ...) {
411418
invalid_args <- intersect(
412419
c(
413420
"access_key", "secret_key", "session_token", "role_arn", "session_name",
414-
"external_id", "load_frequency", "allow_bucket_creation", "allow_bucket_deletion"
421+
"external_id", "load_frequency", "allow_bucket_creation", "allow_bucket_deletion",
422+
"check_directory_existence_before_creation"
415423
),
416424
names(args)
417425
)
@@ -459,6 +467,7 @@ default_s3_options <- list(
459467
background_writes = TRUE,
460468
allow_bucket_creation = FALSE,
461469
allow_bucket_deletion = FALSE,
470+
check_directory_existence_before_creation = FALSE,
462471
connect_timeout = -1,
463472
request_timeout = -1
464473
)

‎r/man/FileSystem.Rd‎

Lines changed: 3 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎r/src/arrowExports.cpp‎

Lines changed: 5 additions & 4 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎r/src/filesystem.cpp‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -289,7 +289,8 @@ std::shared_ptr<fs::S3FileSystem> fs___S3FileSystem__create(
289289
std::string region = "", std::string endpoint_override = "", std::string scheme = "",
290290
std::string proxy_options = "", bool background_writes = true,
291291
bool allow_bucket_creation = false, bool allow_bucket_deletion = false,
292-
double connect_timeout = -1, double request_timeout = -1) {
292+
bool check_directory_existence_before_creation = false, double connect_timeout = -1,
293+
double request_timeout = -1) {
293294
// We need to ensure that S3 is initialized before we start messing with the
294295
// options
295296
StopIfNotOk(fs::EnsureS3Initialized());
@@ -330,6 +331,8 @@ std::shared_ptr<fs::S3FileSystem> fs___S3FileSystem__create(
330331

331332
s3_opts.allow_bucket_creation = allow_bucket_creation;
332333
s3_opts.allow_bucket_deletion = allow_bucket_deletion;
334+
s3_opts.check_directory_existence_before_creation =
335+
check_directory_existence_before_creation;
333336

334337
s3_opts.request_timeout = request_timeout;
335338
s3_opts.connect_timeout = connect_timeout;

‎r/tests/testthat/test-s3-minio.R‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -46,15 +46,17 @@ fs <- S3FileSystem$create(
4646
scheme = "http",
4747
endpoint_override = paste0("localhost:", minio_port),
4848
allow_bucket_creation = TRUE,
49-
allow_bucket_deletion = TRUE
49+
allow_bucket_deletion = TRUE,
50+
check_directory_existence_before_creation = TRUE,
5051
)
5152
limited_fs <- S3FileSystem$create(
5253
access_key = minio_key,
5354
secret_key = minio_secret,
5455
scheme = "http",
5556
endpoint_override = paste0("localhost:", minio_port),
5657
allow_bucket_creation = FALSE,
57-
allow_bucket_deletion = FALSE
58+
allow_bucket_deletion = FALSE,
59+
check_directory_existence_before_creation = FALSE,
5860
)
5961
now <- as.character(as.numeric(Sys.time()))
6062
fs$CreateDir(now)

‎r/vignettes/fs.Rmd‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -190,7 +190,7 @@ Also note that parameters in the URI need to be
190190

191191
For S3, only the following options can be included in the URI as query parameters
192192
are `region`, `scheme`, `endpoint_override`, `access_key`, `secret_key`, `allow_bucket_creation`,
193-
and `allow_bucket_deletion`. For GCS, the supported parameters are `scheme`, `endpoint_override`,
193+
`allow_bucket_deletion` and `check_directory_existence_before_creation`. For GCS, the supported parameters are `scheme`, `endpoint_override`,
194194
and `retry_limit_seconds`.
195195

196196
In GCS, a useful option is `retry_limit_seconds`, which sets the number of seconds

0 commit comments

Comments
 (0)