Apache OpenDAL™ (/ˈoʊ.pən.dæl/, pronounced "OH-puhn-dal") is an Open Data Access Layer that keeps application storage code clean, from simple file reads and writes to streaming, byte-range reads, and concurrent uploads.
Use OpenDAL when your application talks to a single S3 bucket or local directory and you want one API for everyday operations, large transfers, and retries. Keep the same operator model when you test in memory, develop on local disk, or connect to 50+ storage services from Rust, Python, Node.js, Java, Go, and more.
- Clean code for a single backend: read, write, stat, list, copy, and delete through one
Operator. Paths, metadata, and error kinds follow one model across operations. - Built for demanding I/O: stream large objects, read only the byte ranges you need, and upload parts concurrently on services with multipart or block writes. Add retries, timeouts, observability, and concurrency limits as layers.
- Portable across environments and backends: use
memoryin tests,fsin local development, and object storage, file systems, databases, or key-value services in production. Capability checks show where services differ. - Proven in data-intensive systems: projects such as Dify, RAGFlow, LlamaIndex, Databend, RisingWave, GreptimeDB, and Lance build on OpenDAL. Read how ScopeDB accesses object storage with OpenDAL.
Bindings share the service model and follow their own language conventions, so available options and layers vary by binding. Each language guide lists what its binding supports.
Each quickstart uses the in-memory service, so it runs without credentials. The second snippet in each language shows the same API against S3.
pip install opendalimport opendal
op = opendal.Operator("memory")
op.write("hello.txt", b"Hello, World!")
print(op.read("hello.txt").decode())
print(f"size = {op.stat('hello.txt').content_length} bytes")
op.delete("hello.txt")Point the operator at S3, add retries, and read only what you need:
import opendal
from opendal.layers import RetryLayer
op = opendal.Operator("s3", bucket="my-bucket", region="us-east-1").layer(
RetryLayer(max_times=3)
)
# Read only the first 64 KiB of an object.
header = op.read("data/events.parquet", offset=0, size=64 * 1024)
# Stream a large object instead of loading it into memory.
total = 0
with op.open("data/events.parquet", "rb") as f:
while chunk := f.read(1024 * 1024):
total += len(chunk)See the Python guide for async APIs, error handling, and production setup.
cargo add opendal
cargo add tokio --features fulluse opendal::services;
use opendal::Operator;
use opendal::Result;
#[tokio::main]
async fn main() -> Result<()> {
let op = Operator::new(services::Memory::default())?;
op.write("hello.txt", "Hello, World!").await?;
let bytes = op.read("hello.txt").await?;
let meta = op.stat("hello.txt").await?;
op.delete("hello.txt").await?;
println!("read {} bytes, stat reports {} bytes", bytes.len(), meta.content_length());
Ok(())
}Enable a service with its feature, for example cargo add opendal --features services-s3. This fragment reads a byte range and uploads a large object in concurrent parts:
use opendal::layers::RetryLayer;
use opendal::services::S3;
use opendal::Operator;
let op = Operator::new(S3::default().bucket("my-bucket").region("us-east-1"))?
.layer(RetryLayer::new().with_jitter());
// Read only the first 64 KiB of an object.
let header = op.read_with("data/events.parquet").range(0..64 * 1024).await?;
// Keep up to 4 part uploads in flight. `close` waits for them and commits the object.
let mut writer = op
.writer_with("archive/big.bin")
.chunk(8 * 1024 * 1024)
.concurrent(4)
.await?;
writer.write(first_part).await?;
writer.write(second_part).await?;
writer.close().await?;See the Rust core guide and the API reference for streaming readers, conditional operations, and layers.
Configure credentials and other options as described in each service's documentation, for example S3.
Each binding has its own independent version number, which may differ from the Rust core version. Check the binding's own version when you look for updates or compatibility information.
| Language | Package | Guide | API reference |
|---|---|---|---|
| Rust | opendal | Guide | docs.rs · dev |
| Python | opendal | Guide | dev |
| Node.js | opendal | Guide | dev |
| Java | opendal-java | Guide | javadoc.io · dev |
| Go | bindings/go | Guide | pkg.go.dev |
| Ruby | opendal | Guide | dev |
| C | Source | Guide | dev |
| C++ | Source | Guide | dev |
More bindings: .NET, D, Dart, Haskell, Lua, OCaml, PHP, Swift, and Zig.
OpenDAL connects to 50+ services through one Operator. A selection by category:
| Category | Services |
|---|---|
| Object storage | s3, gcs, azblob, oss, cos, obs, tos, b2, swift, vercel-blob, … |
| File storage | fs, hdfs, hdfs-native, webhdfs, azdls, azfile, alluxio, lakefs, ipfs, dbfs, … |
| Cloud SaaS | gdrive, dropbox, onedrive, hf, github, aliyun-drive, koofr, pcloud, … |
| Standard protocols | http, ftp, webdav, sftp, git |
| Databases | sqlite, mysql, postgresql, mongodb, surrealdb, d1, … |
| Key-value and embedded | redis, etcd, tikv, rocksdb, foundationdb, memcached, moka, foyer, … |
See the services documentation for every service and its configuration keys.
Layers wrap an operator to add cross-cutting behavior: RetryLayer, TimeoutLayer, ConcurrentLimitLayer, ThrottleLayer, LoggingLayer, TracingLayer, MetricsLayer, PrometheusLayer, OtelMetricsLayer, MimeGuessLayer, RouteLayer, FoyerLayer, and more. These links point to the Rust core; each binding guide lists the layers it exposes.
- ScopeDB on OpenDAL: sparse range reads, concurrent multipart writes, cloud authentication, and separate timeout and retry policies for reads and writes.
- Who uses OpenDAL: projects and companies that build on OpenDAL.
- Examples: Rust, Python, Node.js, and more.
- Website: https://opendal.apache.org
- Vision: https://opendal.apache.org/vision
- Rust release docs: https://docs.rs/opendal
- Rust dev docs: https://opendal.apache.org/docs/rust/opendal/
OpenDAL is an active open-source project. We are always open to people who want to use it or contribute to it. Here are some ways to go.
- Start with Contributing Guide.
- Submit Issues for bug report or feature requests.
- Start Discussions for questions or ideas.
- Talk to community directly at Discord.
- Report security vulnerabilities to private mailing list
The first and most prominent mentions must use the full form: Apache OpenDAL™ of the name for any individual usage (webpage, handout, slides, etc.) Depending on the context and writing style, you should use the full form of the name sufficiently often to ensure that readers clearly understand the association of both the OpenDAL project and the OpenDAL software product to the ASF as the parent organization.
For more details, see the Apache Product Name Usage Guide.
Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0
Apache OpenDAL, OpenDAL, and Apache are either registered trademarks or trademarks of the Apache Software Foundation.