Roboflow is a computer vision platform that provides tools for building, deploying, and managing vision AI models. The Roboflow plugin for Vision Agents enables real-time object detection using both Roboflow’s hosted inference API and local RF-DETR models.
The plugin provides two processors:
- RoboflowCloudDetectionProcessor: Uses Roboflow’s hosted API for inference with pre-trained models from Roboflow Universe
- RoboflowLocalDetectionProcessor: Runs RF-DETR models locally on your device for inference without API calls
Installation
Install the Roboflow plugin with
uv add vision-agents[roboflow]
Quick Start
Using RoboflowCloudDetectionProcessor (Hosted)
The RoboflowCloudDetectionProcessor uses Roboflow’s hosted API for inference with pre-trained models from Roboflow Universe.
from vision_agents.plugins import roboflow
from vision_agents.core import Agent, User
from vision_agents.plugins import gemini, getstream
processor = roboflow.RoboflowCloudDetectionProcessor(
api_key="your_api_key", # or set ROBOFLOW_API_KEY env var
api_url="https://detect.roboflow.com", # or set ROBOFLOW_API_URL env var
model_id="football-players-detection-3zvbc/20",
classes=["player"],
conf_threshold=0.5,
fps=3,
)
agent = Agent(
edge=getstream.Edge(),
agent_user=User(name="Sports Analyst", id="agent"),
instructions="You are a sports analyst that can detect and track players.",
llm=gemini.Realtime(fps=10),
processors=[processor],
)
To initialize without passing in the API key, make sure the ROBOFLOW_API_KEY is available as an environment variable. You can do this either by defining it in a .env file or exporting it directly in your terminal.
Using RoboflowLocalDetectionProcessor (On-Device)
The RoboflowLocalDetectionProcessor runs RF-DETR models locally on your device. This is useful when you need higher throughput or want to avoid API rate limits.
from vision_agents.plugins import roboflow
from vision_agents.core import Agent, User
from vision_agents.plugins import gemini, getstream
processor = roboflow.RoboflowLocalDetectionProcessor(
model_id="rfdetr-seg-preview",
conf_threshold=0.5,
classes=["person"],
fps=3,
)
agent = Agent(
edge=getstream.Edge(),
agent_user=User(name="Vision Assistant", id="agent"),
instructions="You are a helpful vision assistant.",
llm=gemini.Realtime(fps=10),
processors=[processor],
)
Example
Check out our Roboflow example to see a complete working example with a video call agent that uses Roboflow detection.
Configuration
RoboflowCloudDetectionProcessor Parameters
| Name | Type | Default | Description |
|---|
model_id | str | - | Roboflow Universe model ID. Example: "football-players-detection-3zvbc/20" |
api_key | str or None | None | Roboflow API key. If not provided, uses ROBOFLOW_API_KEY env variable. |
api_url | str or None | None | Roboflow API URL. If not provided, uses ROBOFLOW_API_URL env variable. |
conf_threshold | float | 0.5 | Confidence threshold for detections (0-1.0). |
fps | int | 5 | Frame processing rate. |
classes | List[str] or None | None | Optional list of class names to detect. Verify classes are supported by the model. If None, all classes are detected. |
annotate | bool | True | If True, annotate detected objects with boxes and labels. |
dim_background_factor | float | 0.0 | How much to dim the background around detected objects (0-1.0). Only effective when annotate=True. |
client | InferenceHTTPClient or None | None | Optional custom instance of inference_sdk.InferenceHTTPClient. |
RoboflowLocalDetectionProcessor Parameters
| Name | Type | Default | Description |
|---|
model_id | str | "rfdetr-seg-preview" | Identifier of the RF-DETR model. Available: "rfdetr-base", "rfdetr-large", "rfdetr-nano", "rfdetr-small", "rfdetr-medium", "rfdetr-seg-preview". |
conf_threshold | float | 0.5 | Confidence threshold for detections (0-1.0). |
fps | int | 10 | Frame processing rate. |
classes | List[str] or None | None | Optional list of class names to detect. Verify classes are supported by the model. If None, all classes are detected. |
annotate | bool | True | If True, annotate detected objects with boxes and labels. |
dim_background_factor | float | 0.0 | How much to dim the background around detected objects (0-1.0). Only effective when annotate=True. |
model | RFDETRModel or None | None | Optional instance of RFDETRModel to use with custom parameters. |
Cloud vs. Local
Cloud (RoboflowCloudDetectionProcessor)
- Use when: You want access to pre-trained models from Roboflow Universe or custom models you’ve trained on Roboflow
- Pros: Access to thousands of pre-trained models, no local GPU required, easy model updates
- Cons: Requires API key, network latency, potential rate limits
- Best for: Using specialized pre-trained models, development, testing
Local (RoboflowLocalDetectionProcessor)
- Use when: You need higher throughput, lower latency, or want to avoid API dependencies
- Pros: No API costs, no rate limits, lower latency, works offline
- Cons: Requires local compute resources, limited to RF-DETR models
- Best for: Production deployments, high-volume applications, edge computing
Use Cases
- Sports Analytics: Track players, balls, and equipment in real-time
- Retail: Monitor inventory, detect products, track customer movement
- Manufacturing: Quality control, defect detection, equipment monitoring
- Security: Person detection, vehicle tracking, anomaly detection
- Healthcare: Medical imaging analysis, patient monitoring
Links
