ort examples ✨

• gpt2: Classic GPT-2 language model text generation
• async-gpt2-api: GPT-2 behind an HTTP API with Session::run_async and axum
• yolov8: YOLOv8 object detection
• semantic-similarity: Semantic textual similarity using all-MiniLM-L6-v2 from SentenceTransformers (SBERT)
• modnet: MODNet portrait matting
  • cudarc: A variant of modnet inferring directly from CUDA buffers using the cudarc crate.
• phi-3-vision: Multimodal text generation using Microsoft's Phi-3 Vision model
• model-info: Simple CLI app to inspect an ONNX model's metadata
• train-clm: Trains a causal language model from scratch using ort
  • train-clm-simple: train-clm, but simpler! Uses a 🤗-esque training API.
• custom-ops: Example implementation of custom operator kernels

To run an example:

1. Clone the repository:

   $ git clone https://github.com/pykeio/ort

   To run an example for a specific version, add --branch:

   $ git clone https://github.com/pykeio/ort --branch v2.0.0-rc.12

2. Enter the root of the repository:

   $ cd ort

3. Run the example with cargo example-<name>:

   $ cargo example-gpt2

Log verbosity

Logs are very verbose by default, which can often clutter the output. The verbosity of ort's logging messages can be controlled with the RUST_LOG environment variable. To mostly silence ort:

$ RUST_LOG=ort=warn cargo example-gpt2

or, with PowerShell on Windows:

$env:RUST_LOG = 'ort=warn';
cargo example-gpt2

Execution providers

You can run an example with an execution provider by passing its feature flag:

$ cargo example-gpt2 --features cuda

Note that not all examples support all execution providers.

Backends

You can also use feature flags to run with different backends:

$ cargo example-gpt2 --features backend-tract

Note that not all examples may be supported by all alternative backends.

---

gpt2

🧑‍💻 View source | examples/gpt2/gpt2.rs

💡 This example supports all EPs & backends.

OpenAI's infamous GPT-2 language model running in ort. It uses a very small model and a very simple sampling algorithm, so it's not very impressive (especially not by today's standards), but the simplicity means it's a great learning resource for running other LLMs with ort!

async-gpt2-api

🧑‍💻 View source | examples/async-gpt2-api/async-gpt2-api.rs

💡 This example supports all EPs. No alternative backends currently support Session::run_async.

Like the gpt2 example, but it streams text generation over HTTP using the axum web server framework. Similar to OpenAI's API, the response is streamed back as Server-Sent Events (SSE), so they can easily be received from a JavaScript client.

The application creates an HTTP server on port 7216. Send a POST request to /generate to start generating!

yolov8

🧑‍💻 View source | examples/yolov8/yolov8.rs

💡 This example supports all EPs & backends.

This example implements YOLOv8 object detection using ort. It features loading images with the image crate, converting them to ort tensors with ndarray, processing the inferred bounding boxes, and displaying them with show-image. YOLO's architecture makes it fairly simple to adapt this example to other YOLO versions, like YOLOv10/11.

semantic-similarity

🧑‍💻 View source | examples/sentence-transformers/semantic-similarity.rs | 💖 Contributed by n12n

This example uses the versatile all-MiniLM-L6-v2 sentence embedding model for textual semantic similarity. It encodes some sample sentences and uses cosine similarity to compare them to a query sentence, à la the popular SentenceTransformers Python module (aka SBERT).

modnet

🧑‍💻 View source | examples/modnet/modnet.rs | 💖 Contributed by ling jia

📸 Photo by Charlotte May

💡 This example supports all EPs & backends.

This example implements the MODNet model for portrait matting. Though this example uses a static image for simplicity, with ort and a hardware-accelerated execution provider, MODNet can be run in real-time! ⚡

cudarc

🧑‍💻 View source | examples/cudarc/cudarc.rs

⚠️ This example only supports CUDA with the default (ONNX Runtime) backend.

This example is a variant of modnet that loads the image into a CUDA buffer using cudarc, and then creates an ort tensor from it. This can be useful for more complex scenarios where you need to perform on-GPU processing before passing data to an ort model.

phi-3-vision

🧑‍💻 View source | examples/phi-3-vision/src/main.rs | Contributed by web3nomad

💡 This example supports all EPs & backends.

ℹ️ This example requires additional setup before it can be run. See the example's README for more information.

phi-3-vision showcases Microsoft's Phi-3 Vision multimodal vision-language model. It first demonstrates text-only input by asking a simple question using only text - then, it asks Phi-3 Vision to interpret a graph. Unlike gpt2, Phi-3 Vision's model accepts token embeddings instead of token IDs, so two auxiliary models are needed to perform the text and image embeddings.

model-info

🧑‍💻 View source | examples/model-info/model-info.rs

💡 This example supports all backends. (And EPs, but those don't really do anything here.)

model-info is a simple application that prints a given model's metadata, as well as the shape & type of its inputs & outputs - like a mini, Rust-ified Netron!

train-clm

🧑‍💻 View source | examples/training/train-clm.rs

💡 This example supports all EPs with the default (ONNX Runtime) backend.

ℹ️ For more information, see the training README.

ort supports training too! You can use ort to train models from scratch, or finetune existing models to create personalized variants - all on-device, no EP needed!

The train-clm example trains a causal language model from scratch on OshiChats v2, a dataset of live text chat messages collected from various VTuber live streams, in order to create perhaps the worst language model ever. For more practical use, the example can be adapted for supervised fine-tuning (SFT) or LoRA finetuning of larger models.

This example uses ort's more advanced Trainer API, manually implementing the training loop. There's also the train-clm-simple example, which uses ort's simple Trainer API — a Hugging Face Trainer-like interface — though at the cost of some flexibility.

train-clm-simple

🧑‍💻 View source | examples/training/train-clm-simple.rs

💡 This example supports all EPs with the default (ONNX Runtime) backend.

ℹ️ For more information, see the training README.

This example is a variant of train-clm, but instead of implementing the training loop manually, it simply sets up a dataloader and lets training rip with trainer.run()!

trainer.train(
    TrainingArguments::new(dataloader)
        .with_lr(7e-5)
        .with_max_steps(5000)
        .with_ckpt_strategy(CheckpointStrategy::Steps(500))
        .with_callbacks(LoggerCallback::new())
)?

custom-ops

🧑‍💻 View source | examples/custom-ops/custom-ops.rs

💡 This example supports all EPs with the default (ONNX Runtime) backend.

You can also implement your own custom ONNX operators with ort! This example showcases two simple operator kernel implementations and their usage in sessions with OperatorDomain.