OpenFrontier: General Navigation with Visual-Language Grounded Frontiers

RSS 2026

* equal contribution
1ETH Zürich, 2Microsoft Spatial AI Lab, 3University of Bonn

Abstract

Open-world navigation requires robots to make decisions in complex everyday environments while adapting to flexible task requirements. Conventional navigation approaches often rely on dense 3D reconstruction and hand-crafted goal metrics, which limits their generalization across tasks and environments. Recent advances in vision–language navigation (VLN) and vision–language–action (VLA) models enable end-to-end policies conditioned on natural language, but typically require interactive training, large-scale data collection, or task-specific fine-tuning with a mobile agent. We formulate navigation as a sparse subgoal identification and reaching problem and observe that providing visual anchoring targets for high-level semantic priors enables highly efficient goal-conditioned navigation. Based on this insight, we select navigation frontiers as semantic anchors and propose OpenFrontier, a training-free navigation framework that seamlessly integrates diverse vision–language prior models. OpenFrontier enables efficient navigation with a lightweight system design, without dense 3D mapping, policy training, or model fine-tuning. We evaluate OpenFrontier across multiple navigation benchmarks and demonstrate strong zero-shot performance, as well as effective real-world deployment on a mobile robot.

Method

The core idea of OpenFrontier is the visual frontier: a navigation frontier that is detected and reasoned about directly in the 2D image, without dense 3D semantic mapping. Given the current RGB observation, a natural-language goal, and the camera pose, OpenFrontier jointly detects candidate frontiers and evaluates them in image space, then lifts the chosen frontier into 3D to drive the robot. The system runs in a fully zero-shot manner, with no policy training, no fine-tuning, and no dense reconstruction.

OpenFrontier pipeline overview

The key insight is that a single representation, the visual frontier, can serve as the bridge between the sub-problems that open-world navigation usually treats separately:

  • Joint detection and reasoning on the image. Frontiers are detected and clustered on the RGB observation, and each cluster is marked with a set-of-marks prompt so a vision–language model can score, in a single forward pass, how relevant it is to the language goal. Anchoring the reasoning in the image plane avoids brittle 3D spatial reasoning by the VLM.
  • Target identification and task-relevance reasoning. The VLM's relevance score re-weights each frontier's exploration-driven information gain, naturally balancing goal-directed exploitation with general exploration to decide where to go next.
  • 3D navigation-target registration. The selected visual frontier is back-projected into 3D metric space to produce a concrete goal pose, and confirmed object targets are registered as high-priority viewpoint frontiers.
  • Global planning. A lightweight frontier manager maintains the active set of frontiers, updates their utility as the robot moves, and hands the best one to any low-level planner, closing the loop until the goal is reached.

Because every stage communicates through frontiers, each component (the detector, the VLM, and the planner) can be swapped independently, giving OpenFrontier a simple, flexible, and open-context design.

BibTeX

@inproceedings{padilla2026openfrontier,
      title={OpenFrontier: General Navigation with Visual-Language Grounded Frontiers},
      author={Padilla, Esteban and Sun, Boyang and Pollefeys, Marc and Blum, Hermann},
      booktitle={Robotics: Science and Systems (RSS)},
      year={2026}
}