This app serves as a valuable resource to inspire your creativity and provide foundational code that you can customize and adapt for your particular use case.
Please dive in and start exploring our demo app today! We look forward to any feedback and are excited to see your innovative ideas.
From this demo app, you will learn many key concepts such as:
- How to prepare Llama models, build the ExecuTorch library, and model inferencing across delegates
- Expose the ExecuTorch library via JNI layer
- Familiarity with current ExecuTorch app-facing capabilities
The goal is for you to see the type of support ExecuTorch provides and feel comfortable with leveraging it for your use cases.
As a whole, the models that this app supports are (varies by delegate):
- Llama * Llama 3.2 Quantized 1B/3B * Llama 3.2 1B/3B in BF16 * Llama Guard 3 1B * Llama 3.1 8B * Llama 3 8B * Llama 2 7B
- LLaVA-1.5 vision model (only XNNPACK)
- Qwen 3 0.6B, 1.7B, and 4B
- Voxtral Mini 3B
- Gemma 3 4B
First it’s important to note that by default, the app depends on ExecuTorch library on Maven Central. It uses the latest org.pytorch:executorch-android package, which comes with all the default kernel libraries (portable, quantized, optimized), LLM customized libraries, and XNNPACK backend.
No modification is needed if you want to use the pre-built ExecuTorch library.
However, you can build your own ExecuTorch Android library (an AAR file). Copy the file to app/libs/executorch.aar. In gradle.properties file, add a line useLocalAar=true so that gradle uses the local AAR file.
This page contains the documentation for building the ExecuTorch Android library.
Currently ExecuTorch provides support across 4 delegates. Once you identify the delegate of your choice, select the README link to get a complete end-to-end instructions for environment set-up to exporting the models to build ExecuTorch libraries and apps to run on device:
| Delegate | Resource |
|---|---|
| XNNPACK (CPU-based library) | link |
| QNN (Qualcomm AI Accelerators) | link |
| MediaTek (MediaTek AI Accelerators) | link |
| Vulkan | link |
This section will provide the main steps to use the app, along with a code snippet of the ExecuTorch API.
For loading the app, development, and running on device we recommend Android Studio:
- Open Android Studio and select "Open an existing Android Studio project" to open the directory containing this README.md file.
- Run the app (^R). This builds and launches the app on the phone.
Below are the UI features for the app.
Select the settings widget to get started with picking a model, its parameters and any prompts.
Before selecting a model and tokenizer in the app, you need to push these files to your Android device. Use the following commands to copy the model (.pte) and tokenizer files to the device:
adb shell mkdir -p /data/local/tmp/llama
adb push <your_model>.pte /data/local/tmp/llama
adb push <your_tokenizer> /data/local/tmp/llamaReplace <your_model>.pte with your exported model file and <your_tokenizer> with your tokenizer file (e.g., tokenizer.bin or tokenizer.model).
Once you've selected the model, tokenizer, and model type you are ready to click on "Load Model" to have the app load the model and go back to the main Chat activity.
Optional Parameters:
- Temperature: Defaulted to 0, you can adjust the temperature for the model as well. The model will reload upon any adjustments.
- System Prompt: More for the advanced user, without any formatting, you can enter in a system prompt. For example, "you are a travel assistant" or "give me a response in a few sentences".
- User Prompt: More for the advanced user, if you would like to manually input a prompt then you can do so by modifying the
{{user prompt}}. You can also modify the special tokens as well. Once changed then go back to the main Chat activity to send.
// Upon returning to the Main Chat Activity
mModule = new LlmModule(
ModelUtils.getModelCategory(mCurrentSettingsFields.getModelType()),
modelPath,
tokenizerPath,
temperature,
dataPath);
int loadResult = mModule.load();modelCategory: Indicates whether it’s a text-only or vision modelmodelPath: Path to the .pte filetokenizerPath: Path to the tokenizer filetemperature: Model parameter to adjust the randomness of the model’s outputdataPath: Path to one or a list of .ptd files
Once the model is successfully loaded, enter any prompt and click the send (i.e., generate) button to send it to the model.
You can provide it more follow-up questions as well.
mModule.generate(prompt, sequence_length, MainActivity.this);prompt: User-formatted promptsequence_length: Number of tokens to generate in response to a promptMainActivity.this: Indicates that the callback functions (onResult(),onStats()) are present in this class.
[LLaVA-1.5: Only for XNNPACK delegate]
For LLaVA-1.5 implementation, select the exported LLaVA .pte and tokenizer file in the Settings menu and load the model. After this you can send an image from your gallery or take a live picture along with a text prompt to the model.
To show completion of the follow-up question, here is the complete detailed response from the model.
llama-android.mp4
llava-android.mp4
gemma-android.mp4
voxtral-S24.mp4
Ensure you have the following functions in your callback class that you provided in the mModule.generate(). For this example, it is MainActivity.this.
@Override
public void onResult(String result) {
//...result contains token from response
//.. onResult will continue to be invoked until response is complete
}
@Override
public void onStats(String stats) {
//... will be a json. See extension/llm/stats.h for the field definitions
}The app includes instrumentation tests for sanity checking and UI workflow validation.
-
SanityCheck - Basic model loading and generation test that verifies the LLM module can load a model and generate tokens.
-
UIWorkflowTest - UI-based tests that simulate user interactions:
testModelLoadingWorkflow: Tests the complete flow of selecting a model/tokenizer and loading ittestSendMessageAndReceiveResponse: Tests sending a message and receiving a response from the model
The test model (stories110M.pte) and tokenizer (tokenizer.model) are automatically downloaded when you run the tests via Gradle. The download task runs before the instrumentation tests execute.
If you want to manually prepare the model files, you can use the following commands:
# Install executorch python package first: https://docs.pytorch.org/executorch/stable/getting-started.html#installation
curl -C - -Ls "https://huggingface.co/karpathy/tinyllamas/resolve/main/stories110M.pt" --output stories110M.pt
curl -C - -Ls "https://raw.githubusercontent.com/karpathy/llama2.c/master/tokenizer.model" --output tokenizer.model
# Create params.json file
touch params.json
echo '{"dim": 768, "multiple_of": 32, "n_heads": 12, "n_layers": 12, "norm_eps": 1e-05, "vocab_size": 32000}' > params.json
# Export the model
python -m executorch.extension.llm.export.export_llm base.checkpoint=stories110M.pt base.params=params.json model.dtype_override="fp16" export.output_name=stories110M.pte model.use_kv_cache=True
# Push to device
adb shell mkdir -p /data/local/tmp/llama
adb push stories110M.pte /data/local/tmp/llama
adb push tokenizer.model /data/local/tmp/llamaThe easiest way to run instrumentation tests is using model presets, which automatically download the model and tokenizer files:
# Run with stories model (default, smallest and fastest)
./gradlew connectedCheck -PmodelPreset=stories
# Run with Llama 3.2 1B model
./gradlew connectedCheck -PmodelPreset=llama
# Run with Qwen3 4B model
./gradlew connectedCheck -PmodelPreset=qwen3
# Run with custom model URLs
./gradlew connectedCheck -PmodelPreset=custom \
-PcustomPteUrl=https://example.com/model.pte \
-PcustomTokenizerUrl=https://example.com/tokenizer.model
# Skip model download (use existing files on device)
./gradlew connectedCheck -PmodelPreset=stories -PskipModelDownload=trueAvailable presets:
| Preset | Model | Description |
|---|---|---|
stories |
stories110M | Tiny model for quick testing |
llama |
Llama 3.2 1B | Production-quality Llama model |
qwen3 |
Qwen3 4B | Qwen3 model with INT8/INT4 quantization |
custom |
User-provided | Specify custom URLs for model and tokenizer |
Run a specific test class:
./gradlew connectedCheck -PmodelPreset=stories -Pandroid.testInstrumentationRunnerArguments.class=com.example.executorchllamademo.SanityCheck
./gradlew connectedCheck -PmodelPreset=stories -Pandroid.testInstrumentationRunnerArguments.class=com.example.executorchllamademo.UIWorkflowTestIf you encountered any bugs or issues following this tutorial, please file a bug/issue here on GitHub, or join our Discord here.





