LiteRT-LM Overview

LiteRT-LM is a production-ready, open-source inference framework designed to deliver high-performance, cross-platform LLM deployments on edge devices.

• Cross-Platform Support: Run on Android, iOS, Web, Desktop, and IoT (e.g. Raspberry Pi).
• Hardware Acceleration: Get peak performance and system stability by leveraging GPU and NPU accelerators across diverse hardware.
• Multi-Modality: Build with LLMs that have vision and audio support.
• Tool Use: Function calling support for agentic workflows with constrained decoding for improved accuracy.
• Broad Model Support: Run Gemma, Llama, Phi-4, Qwen and more.

On-Device GenAI Showcase

The Google AI Edge Gallery is an experimental app designed to showcase on-device Generative AI capabilities running entirely offline using LiteRT-LM.

• Google Play: Use LLMs locally on supported Android devices.
• App Store: Experience on-device AI on your iOS device.
• GitHub Source: View the source code for the gallery app to learn how to integrate LiteRT-LM inside your own projects.

Featured Model: Gemma-3n-E2B

Start Building

The following snippets show how to get started with the LiteRT-LM CLI, as well as Python, Kotlin, and C++ APIs.

litert-lm run model.litertlm --prompt="What is the capital of France?"

engine = litert_lm.Engine("model.litertlm")

with engine.create_conversation() as conversation:
    response = conversation.send_message("What is the capital of France?")
    print(f"Response: {response['content'][0]['text']}")

val engineConfig = EngineConfig(
    modelPath = "/path/to/your/model.litertlm",
    backend = Backend.CPU(),
)

val engine = Engine(engineConfig)
engine.initialize()

val conversation = engine.createConversation()
print(conversation.sendMessage("What is the capital of France?"))  

auto model_assets = ModelAssets::Create(model_path);
CHECK_OK(model_assets);

auto engine_settings = EngineSettings::CreateDefault(
    model_assets,
    /*backend=*/litert::lm::Backend::CPU);

absl::StatusOr<std::unique_ptr<Engine>> engine = Engine::CreateEngine(engine_settings);
CHECK_OK(engine);

auto conversation_config = ConversationConfig::CreateDefault(**engine);
CHECK_OK(conversation_config);
absl::StatusOr<std::unique_ptr<Conversation>> conversation = Conversation::Create(**engine, *conversation_config);
CHECK_OK(conversation);

absl::StatusOr<Message> model_message = (*conversation)->SendMessage(
    JsonMessage{
        {"role", "user"},
        {"content", "What is the capital of France?"}
    });
CHECK_OK(model_message);

std::cout << *model_message << std::endl;

Supported Backends & Platforms

Supported Models

The following table lists models supported by LiteRT-LM. For more detailed performance numbers and model cards, visit the LiteRT Community on Hugging Face.

Reporting Issues

If you encounter a bug or have a feature request, report at LiteRT-LM GitHub Issues.