Skip to content

index.mdx

Latest commit

 

History

History
77 lines (58 loc) · 3.23 KB

index.mdx

File metadata and controls

77 lines (58 loc) · 3.23 KB
title Alternative backends

Alternative backends

import { Callout, Steps } from 'nextra/components'; import Ort from '../../components/Ort';

Since ONNX Runtime is written in C++, linking troubles often arise when attempting to use it in a Rust project--especially with WASM. v2.0.0-rc.12 of introduced support for alternative backends: ONNX runtimes that aren't the ONNX Runtime.

As the Rust ML scene has evolved, many exciting new inference engines supporting ONNX models have popped up, like 🤗 Hugging Face's candle, Burn, and tract. These libraries, being written in pure Rust, play much nicer when it comes to linking, and often support any platform Rust's standard library does (which is a lot more than ONNX Runtime!) They're also, of course, memory safe and 🦀blazingly🔥fast🚀

alternative backends are simply wrappers that implements the ONNX Runtime C API over another crate. Because they implement the same exact API as ONNX Runtime, using them in is as simple as adding one line of code!

Using an alternative backend

Alternative backends are experimental, and are constantly changing and growing -- use them at your own risk! 
We may not be able to provide the same level of support for different backends as we do with ONNX Runtime.

Install the alternative backend

We'll use ort-tract for this example.

[dependencies]
ort-tract = "0.3.0+0.22"
...

Enable the alternative-backend feature

This instructs to not try to download/link to ONNX Runtime.

[dependencies.ort]
version = "=2.0.0-rc.12"
default-features = false # Disables the `download-binaries` feature since we don't need it
features = [
    "std",
    "ndarray",
    "alternative-backend"
]

Initialize the backend

Use ort::set_api to use the crate's API implementation (replacing ort_tract with whichever backend crate you choose to use):

fn main() {
    // This should run as early in your application as possible - before you ever use `ort`!
    ort::set_api(ort_tract::api());
}

Done!

Be sure to check each backend's docs page to see which APIs are and are not supported.

Available backends

currently has the following backends:

  • ort-candle, based on 🤗 Hugging Face candle
    • 🔷 Supports: CPU, CUDA (though not available via ort-candle right now), WebAssembly
    • ⚠️ Limited operator support; though most transformer models have good support.
  • ort-tract, based on tract
  • ort-web runs ONNX Runtime in the web
    • 🔷 Supports: WebAssembly (with WebGL & WebGPU backends!)
    • ✅ Great operator support - it's the full ONNX Runtime!