🦾 VLA-Lab

The Missing Toolkit for Vision-Language-Action Model Deployment

Log · Replay · Analyze · Evaluate — All-in-one toolkit for real-world VLA deployment

🚀 Quick Start · 📸 Screenshots · 🎯 Features · 🔧 Installation

---

🎯 Why VLA-Lab?

Deploying VLA models to real robots is hard. You face:

• 🕵️ Black-box inference — Can't see what the model "sees" or why it fails
• ⏱️ Hidden latencies — Transport delays, inference bottlenecks, control loop timing issues
• 📊 Fragmented logging — Every framework logs differently, making cross-model comparison painful
• 🔄 Tedious debugging — Replaying failures requires manual log parsing and visualization

VLA-Lab solves this. A unified logging format + interactive visualization dashboard covering the full workflow from data collection to open-loop evaluation.

---

✨ Features

🔬 Inference Replay Step-by-step playback of policy inference: multi-camera views, 3D end-effector trajectories, full-dimensional state/action curves — pinpoint deployment failures instantly. 📊 Dataset Viewer Interactively browse Zarr-format training data: frame-by-frame analysis, global statistics overview, and workspace distribution heatmaps.

📈 Latency Analysis Decompose transport delay, GPU inference latency, and end-to-end loop time. Time-series plots + statistical distributions + multi-run comparison to identify bottlenecks fast. 🎯 Open-Loop Eval Compare predicted actions against ground truth: MSE / MAE metric summaries, temporal alignment, error heatmaps, and 3D trajectory overlay.

🔧 Supported Frameworks

Framework	Status
Diffusion Policy	✅ Supported
Isaac-GR00T	✅ Supported
Pi 0.5	✅ Supported
DreamZero	✅ Supported
VITA	✅ Supported

VLA-Lab uses a unified logging protocol — adapting a new framework takes only a few lines of glue code.

---

📸 Screenshots

🚀 Get Started — Feature Overview
🔬 Inference Replay Multi-camera views · 3D end-effector trajectory · Full state/action curves	📈 Latency Analysis Transport & inference latency time-series · Statistical distributions · Multi-run comparison
📊 Dataset Frame-by-Frame Analysis Camera views · Robot state · Timeline scrubbing	📊 Dataset Global Overview Episode statistics · Action distributions · Image grid
📊 Workspace Distribution 3D workspace sampling density visualization	🎯 Open-Loop Evaluation MSE/MAE summary · Temporal comparison · Error heatmap · 3D trajectories

---

🔧 Installation

pip install vlalab

Install with full dependencies (including Zarr dataset support):

pip install vlalab[full]

Or install from source:

git clone https://github.com/ky-ji/VLA-Lab.git
cd VLA-Lab
pip install -e .

---

🚀 Quick Start

Minimal Example (3 Lines!)

import vlalab

# Initialize a run
run = vlalab.init(project="pick_and_place", config={"model": "diffusion_policy"})

# Log during inference
vlalab.log({"state": obs["state"], "action": action, "images": {"front": obs["image"]}})

Full Example

import vlalab

# Initialize with detailed config
run = vlalab.init(
    project="pick_and_place",
    config={
        "model": "diffusion_policy",
        "action_horizon": 8,
        "inference_freq": 10,
    },
)

# Access config anywhere
print(f"Action horizon: {run.config.action_horizon}")

# Inference loop
for step in range(100):
    obs = get_observation()
    
    t_start = time.time()
    action = model.predict(obs)
    latency = (time.time() - t_start) * 1000
    
    # Log everything in one call
    vlalab.log({
        "state": obs["state"],
        "action": action,
        "images": {"front": obs["front_cam"], "wrist": obs["wrist_cam"]},
        "inference_latency_ms": latency,
    })

    robot.execute(action)

# Auto-finishes on exit, or call manually
vlalab.finish()

Launch Visualization

# Default web dashboard (FastAPI + Next.js)
vlalab view

Launch Web Dashboard (FastAPI + Next.js)

# 1. Install Python package
pip install -e .

# 2. Install web dependencies once
cd web
npm install
cd ..

# 3. Start FastAPI + Next.js together
vlalab view

Useful variants:

# Start only the API backend
vlalab serve --no-frontend

# Change the frontend port
vlalab view --port 3100

# Override the run directory used by the API
vlalab view --run-dir /path/to/vlalab_runs

The web UI covers the main workflows:

• overview / run list / run detail
• latency comparison
• dataset viewer
• open-loop eval viewer

---

📖 Documentation

Core Concepts

Run — A single deployment session (one experiment, one episode, one evaluation)

Step — A single inference timestep with observations, actions, and timing

Artifacts — Images, point clouds, and other media saved alongside logs

API Reference

vlalab.init() — Initialize a run

run = vlalab.init(
    project: str = "default",     # Project name (creates subdirectory)
    name: str = None,             # Run name (auto-generated if None)
    config: dict = None,          # Config accessible via run.config.key
    dir: str = "./vlalab_runs",   # Base directory (or $VLALAB_DIR)
    tags: list = None,            # Optional tags
    notes: str = None,            # Optional notes
)

vlalab.log() — Log a step

vlalab.log({
    # Robot state
    "state": [...],                    # Full state vector
    "pose": [x, y, z, qx, qy, qz, qw], # Position + quaternion
    "gripper": 0.5,                    # Gripper opening (0-1)
    
    # Actions
    "action": [...],                   # Single action or action chunk
    
    # Images (multi-camera support)
    "images": {
        "front": np.ndarray,           # HWC numpy array
        "wrist": np.ndarray,
    },
    
    # Timing (any *_ms field auto-captured)
    "inference_latency_ms": 32.1,
    "transport_latency_ms": 5.2,
    "custom_metric_ms": 10.0,
})

RunLogger — Advanced API

For fine-grained control over logging:

from vlalab import RunLogger

logger = RunLogger(
    run_dir="runs/experiment_001",
    model_name="diffusion_policy",
    model_path="/path/to/checkpoint.pt",
    task_name="pick_and_place",
    robot_name="franka",
    cameras=[
        {"name": "front", "resolution": [640, 480]},
        {"name": "wrist", "resolution": [320, 240]},
    ],
    inference_freq=10.0,
)

logger.log_step(
    step_idx=0,
    state=[0.5, 0.2, 0.3, 0, 0, 0, 1, 1.0],
    action=[[0.51, 0.21, 0.31, 0, 0, 0, 1, 1.0]],
    images={"front": image_rgb},
    timing={
        "client_send": t1,
        "server_recv": t2,
        "infer_start": t3,
        "infer_end": t4,
    },
)

logger.close()

CLI Commands

# Launch the default FastAPI + Next.js dashboard
vlalab view [--port 3000] [--api-port 8000]

# Launch FastAPI backend and optional Next.js frontend
vlalab serve [--api-port 8000] [--web-port 3000] [--frontend/--no-frontend]

# Convert legacy logs (auto-detects format)
vlalab convert /path/to/old_log.json -o /path/to/output

# Inspect a run
vlalab info /path/to/run_dir

Attention Backend Integration

VLA-Lab now treats attention extraction as a model-provided backend with a stable interface. The built-in default still auto-discovers the existing Isaac-GR00T backend, but new models should register their own backend explicitly:

export VLALAB_ATTENTION_BACKEND=/abs/path/to/backend.py
export VLALAB_ATTENTION_PYTHON=/abs/path/to/python  # optional

Interface contract and migration checklist:

• Attention Backend Guide

---

📁 Run Directory Structure

vlalab_runs/
└── pick_and_place/                 # Project
    └── run_20240115_103000/        # Run
        ├── meta.json               # Metadata (model, task, robot, cameras)
        ├── steps.jsonl             # Step records (one JSON per line)
        └── artifacts/
            └── images/             # Saved images
                ├── step_000000_front.jpg
                ├── step_000000_wrist.jpg
                └── ...

---

🗺️ Roadmap

• Core logging API & unified run format
• Streamlit visualization suite (5 pages)
• Diffusion Policy adapter
• Isaac-GR00T adapter
• Pi 0.5 adapter
• DreamZero adapter
• VITA adapter
• Open-loop evaluation pipeline
• Cloud sync & team collaboration
• Real-time streaming dashboard
• Automatic failure detection
• Integration with robot simulators

---

🤝 Contributing

We welcome contributions!

git clone https://github.com/ky-ji/VLA-Lab.git
cd VLA-Lab
pip install -e ".[dev]"

---

📄 License

MIT License — see LICENSE for details.

---

⭐ Star us on GitHub if VLA-Lab helps your research!

Built with ❤️ for the robotics community