Alternative backends

Since ONNX Runtime is written in C++, linking troubles often arise when attempting to use it in a Rust project—especially with WASM. v2.0.0-rc.12 of ort introduced support for alternative backends: ONNX runtimes that aren’t the ONNX Runtime.

As the Rust ML scene has evolved, many exciting new inference engines supporting ONNX models have popped up, like 🤗 Hugging Face’s candle, Burn , and tract. These libraries, being written in pure Rust, play much nicer when it comes to linking, and often support any platform Rust’s standard library does (which is a lot more than ONNX Runtime!) They’re also, of course, memory safe and 🦀blazingly🔥fast🚀

ort alternative backends are simply wrappers that implements the ONNX Runtime C API over another crate. Because they implement the same exact API as ONNX Runtime, using them in ort is as simple as adding one line of code!

Using an alternative backend

Alternative backends are experimental, and are constantly changing and growing — use them at your own risk!

We may not be able to provide the same level of support for different backends as we do with ONNX Runtime.

Install the alternative backend

We’ll use ort-tract for this example.

Cargo.toml


[dependencies]
ort-tract = "0.3.0+0.22"
...

Enable the `alternative-backend` feature

This instructs ort to not try to download/link to ONNX Runtime.

Cargo.toml


[dependencies.ort]
version = "=2.0.0-rc.12"
default-features = false # Disables the `download-binaries` feature since we don't need it
features = [
    "std",
    "ndarray",
    "alternative-backend"
]

Initialize the backend

Use ort::set_api to use the crate’s API implementation (replacing ort_tract with whichever backend crate you choose to use):


fn main() {
    // This should run as early in your application as possible - before you ever use `ort`!
    ort::set_api(ort_tract::api());
}

Done!

Be sure to check each backend’s docs page to see which APIs are and are not supported.

Available backends

ort currently has the following backends:

ort-candle, based on 🤗 Hugging Face candle
- 🔷 Supports: CPU, CUDA (though not available via ort-candle right now), WebAssembly
- ⚠️ Limited operator support; though most transformer models have good support.
ort-tract, based on tract
- 🔷 Supports: CPU, WebAssembly
- ✅ Great operator support
ort-web runs ONNX Runtime in the web
- 🔷 Supports: WebAssembly (with WebGL & WebGPU backends!)
- ✅ Great operator support - it’s the full ONNX Runtime!