Skip to content

Commit bd2c070

Browse files
authored
GH-41973: Expose new S3 option check_directory_existence_before_creation - manual rebase (#46619)
See #41998 - rebase was too messy due to PR age so did this manually. * GitHub Issue: #41973 Lead-authored-by: Haocheng Liu <[email protected]> Co-authored-by: Nic Crane <[email protected]> Signed-off-by: Nic Crane <[email protected]>
1 parent 5890232 commit bd2c070

12 files changed

Lines changed: 64 additions & 32 deletions

File tree

‎r/NEWS.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,8 @@
2222
## Minor improvements and fixes
2323

2424
- Added bindings for atan, sinh, cosh, tanh, asinh, acosh, and tanh, and expm1 (#44953)
25+
- Expose an option `check_directory_existence_before_creation` in `S3FileSystem`
26+
to reduce I/O calls on cloud storage (@HaochengLIU, #41998)
2527

2628
# arrow 20.0.0
2729

‎r/R/arrowExports.R‎

Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎r/R/dplyr-funcs-datetime.R‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -856,7 +856,8 @@ register_bindings_hms <- function() {
856856
Expression$create("multiply_checked", days, 86400)
857857

858858
return(numeric_to_time32(total_secs))
859-
}
859+
},
860+
notes = "subsecond times not supported"
860861
)
861862

862863
register_binding(
@@ -880,6 +881,7 @@ register_bindings_hms <- function() {
880881
)
881882
return(datetime_to_time32(as_date_time))
882883
}
883-
}
884+
},
885+
notes = "subsecond times not supported"
884886
)
885887
}

‎r/R/dplyr-funcs-doc.R‎

Lines changed: 7 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@
2121
#'
2222
#' The `arrow` package contains methods for 37 `dplyr` table functions, many of
2323
#' which are "verbs" that do transformations to one or more tables.
24-
#' The package also has mappings of 221 R functions to the corresponding
24+
#' The package also has mappings of 222 R functions to the corresponding
2525
#' functions in the Arrow compute library. These allow you to write code inside
2626
#' of `dplyr` methods that call R functions, including many in packages like
2727
#' `stringr` and `lubridate`, and they will get translated to Arrow and run
@@ -83,7 +83,7 @@
8383
#' Functions can be called either as `pkg::fun()` or just `fun()`, i.e. both
8484
#' `str_sub()` and `stringr::str_sub()` work.
8585
#'
86-
#' In addition to these functions, you can call any of Arrow's 271 compute
86+
#' In addition to these functions, you can call any of Arrow's 280 compute
8787
#' functions directly. Arrow has many functions that don't map to an existing R
8888
#' function. In other cases where there is an R function mapping, you can still
8989
#' call the Arrow function directly if you don't want the adaptations that the R
@@ -96,7 +96,6 @@
9696
#'
9797
#' * [`add_filename()`][arrow::add_filename()]
9898
#' * [`cast()`][arrow::cast()]
99-
#' * [`one()`][arrow::one()]
10099
#'
101100
#' ## base
102101
#'
@@ -215,6 +214,11 @@
215214
#' * [`n()`][dplyr::n()]
216215
#' * [`n_distinct()`][dplyr::n_distinct()]
217216
#'
217+
#' ## hms
218+
#'
219+
#' * [`as_hms()`][hms::as_hms()]
220+
#' * [`hms()`][hms::hms()]
221+
#'
218222
#' ## lubridate
219223
#'
220224
#' * [`am()`][lubridate::am()]
@@ -297,11 +301,6 @@
297301
#' * [`ymd_hms()`][lubridate::ymd_hms()]: `locale` argument not supported
298302
#' * [`yq()`][lubridate::yq()]: `locale` argument not supported
299303
#'
300-
#' ## hms
301-
#'
302-
#' * [`hms()`][hms::hms()]: subsecond times not supported
303-
#' * [`hms()`][hms::as_hms()]: subsecond times not supported
304-
#'
305304
#' ## methods
306305
#'
307306
#' * [`is()`][methods::is()]

‎r/R/filesystem.R‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -156,6 +156,10 @@ FileSelector$create <- function(base_dir, allow_not_found = FALSE, recursive = F
156156
#' buckets if `$CreateDir()` is called on the bucket level (default `FALSE`).
157157
#' - `allow_bucket_deletion`: logical, if TRUE, the filesystem will delete
158158
#' buckets if`$DeleteDir()` is called on the bucket level (default `FALSE`).
159+
#' - `check_directory_existence_before_creation`: logical, check if directory
160+
#' already exists or not before creation. Helpful for cloud storage operations
161+
#' where object mutation operations are rate limited or existing directories
162+
#' are read-only. (default `FALSE`).
159163
#' - `request_timeout`: Socket read time on Windows and macOS in seconds. If
160164
#' negative, the AWS SDK default (typically 3 seconds).
161165
#' - `connect_timeout`: Socket connection timeout in seconds. If negative, AWS
@@ -411,7 +415,8 @@ S3FileSystem$create <- function(anonymous = FALSE, ...) {
411415
invalid_args <- intersect(
412416
c(
413417
"access_key", "secret_key", "session_token", "role_arn", "session_name",
414-
"external_id", "load_frequency", "allow_bucket_creation", "allow_bucket_deletion"
418+
"external_id", "load_frequency", "allow_bucket_creation", "allow_bucket_deletion",
419+
"check_directory_existence_before_creation"
415420
),
416421
names(args)
417422
)
@@ -459,6 +464,7 @@ default_s3_options <- list(
459464
background_writes = TRUE,
460465
allow_bucket_creation = FALSE,
461466
allow_bucket_deletion = FALSE,
467+
check_directory_existence_before_creation = FALSE,
462468
connect_timeout = -1,
463469
request_timeout = -1
464470
)

‎r/data-raw/docgen.R‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -149,6 +149,10 @@ tidyselect <- grep("^tidyselect::", readLines("R/reexports-tidyselect.R"), value
149149
# HACK: remove the _random_along UDF we're using (fix in ARROW-17974)
150150
docs[["_random_along"]] <- NULL
151151

152+
# TODO - update the script to add this back in - will fail CI as tries to link
153+
# to non-existent function as arrow::one only exists as registered binding
154+
docs[["arrow::one"]] <- NULL
155+
152156
docs <- c(docs, setNames(rep(list(NULL), length(tidyselect)), tidyselect))
153157

154158
fun_df <- tibble::tibble(

‎r/man/FileSystem.Rd‎

Lines changed: 4 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎r/man/acero.Rd‎

Lines changed: 18 additions & 10 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎r/src/arrowExports.cpp‎

Lines changed: 6 additions & 5 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎r/src/filesystem.cpp‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -289,7 +289,8 @@ std::shared_ptr<fs::S3FileSystem> fs___S3FileSystem__create(
289289
std::string region = "", std::string endpoint_override = "", std::string scheme = "",
290290
std::string proxy_options = "", bool background_writes = true,
291291
bool allow_bucket_creation = false, bool allow_bucket_deletion = false,
292-
double connect_timeout = -1, double request_timeout = -1) {
292+
bool check_directory_existence_before_creation = false, double connect_timeout = -1,
293+
double request_timeout = -1) {
293294
// We need to ensure that S3 is initialized before we start messing with the
294295
// options
295296
StopIfNotOk(fs::EnsureS3Initialized());
@@ -331,6 +332,9 @@ std::shared_ptr<fs::S3FileSystem> fs___S3FileSystem__create(
331332
s3_opts.allow_bucket_creation = allow_bucket_creation;
332333
s3_opts.allow_bucket_deletion = allow_bucket_deletion;
333334

335+
s3_opts.check_directory_existence_before_creation =
336+
check_directory_existence_before_creation;
337+
334338
s3_opts.request_timeout = request_timeout;
335339
s3_opts.connect_timeout = connect_timeout;
336340

0 commit comments

Comments
 (0)