Autonomous navigation systems trained on fixed datasets struggle to generalize when deployed in unfamiliar environments. A model trained on high-infrastructure urban settings may fail to correctly interpret a rural tractor, a costumed pedestrian, or an unusual roadside obstacle — not because the model is poorly designed, but because the object or scene lies outside its training distribution. Rather than attempting to retrain or replace existing navigation systems, this work explores a supplementary pipeline that can be layered on top of them to handle out-of-distribution inputs.
The pipeline operates in two stages. The first stage, unknown object detection, addresses the problem of a model encountering objects it cannot confidently classify. Three approaches were implemented and evaluated on live webcam input: a confidence-thresholded YOLO-World detector that flags low-confidence predictions as unknown, a dual-model consensus pipeline pairing a class-agnostic detector (NanoOWL) with YOLO-World to identify objects that are detected but cannot be matched to a known class, and a closed-vocabulary YOLO baseline that illustrates the core limitation motivating the other approaches. The second stage, multimodal scene interpretation, addresses the downstream question of what to do once an unknown or ambiguous scene is detected. A suite of vision-language models (VLMs) was evaluated on their ability to produce a natural-language scene description and a binary traversability judgment from a single image. Models evaluated include local VLMs run on-device (Qwen2-VL, LLaVA-1.5, Moondream2, InternVL2-2B) and cloud API approaches (GPT-4o-mini, Gemini 2.0 Flash Lite). Local models exhibited inference times of 10–30+ minutes per image on CPU, making them impractical for real-time deployment in their current form. Cloud API models returned results in 2–5 seconds with no local hardware requirements, demonstrating substantially greater feasibility for near-term prototyping.
Shared utilities across the VLM implementations were validated using property-based testing with the Hypothesis library, ensuring consistent traversability parsing, output formatting, and file path construction across all model backends. Taken together, the two subsystems form a modular, model-agnostic pipeline that can flag uncertainty in perception and provide contextual scene understanding — two capabilities that are largely absent from standard autonomous navigation stacks when operating outside their training domain.
