While the stores appear to be filesystems, underneath

They cannot be used as a direct replacement for a cluster filesystem such as HDFS

*except where this is explicitly stated*.

* Changes to stored objects may not be immediately visible, both in directory listings and actual data access.

* The means by which directories are emulated may make working with them slow.

* Rename operations may be very slow and, on failure, leave the store in an unknown state.

* Seeking within a file may require new HTTP calls, hurting performance.

1. Reading and writing data can be significantly slower than working with a normal filesystem.

1. Some directory structures may be very inefficient to scan during query split calculation.

*Important: never check authentication secrets into source code repositories,

especially public ones*

configuration and security options.

This does less renaming at the end of a job than the "version 1" algorithm.

As it still uses `rename()` to commit files, it is unsafe to use

when the object store does not have consistent metadata/listings.

As storing temporary files can run up charges; delete

directories called `"_temporary"` on a regular basis to avoid this.

For optimal performance when working with Parquet data use the following settings:

Here is the documentation on the standard connectors both from Apache and the cloud providers.

* [Google Cloud Storage Connector for Spark and Hadoop](https://cloud.google.com/hadoop/google-cloud-storage-connector). From Google

<artifactId>spark-hadoop-cloud_2.11</artifactId>

<packaging>jar</packaging>

<description>

Contains support for cloud infrastructures, specifically the Hadoop JARs and

transitive dependencies needed to interact with the infrastructures,