MiniCPM-RobotTrack

MiniCPM-RobotTrack A Compact Vision-Language-Action Policy for Embodied target Tracking Github  | X  | Discord  | MiniCPM-RobotTrack is a compact vision-language-action model built on MiniCPM4-0.5B for target tracking with the following highlights: Language-conditioned target tracking: The model combines natural-language instructions with fused DINOv3 and SigLIP visual features, then directly predicts eight future [x, y, yaw] waypoints for embodied person following. Quality-driven Self-evolving Data: Automated checks and manual review remove abnormal trajectories, incorrect actions, and invalid interactions. DAgger-style model-environment interaction continually adds difficult samples involving target crossings, rapid turns, short occlusions, and multi-person intersections. Efficient On-device Inference: Joint optimization of visual capture, input encoding, model inference, action generation, command transmission, and execution delivers a stable 5+ FPS with approximately 180 ms end-to-end latency on the Unitree Go2's native onboard compute. Outdoor Obstacle-aware Tracking Elevator Tracking Underground Parking Tracking Benchmark Results Inference Example Please ensure `transformers>=4.56, from __future__ import annotations from pathlib import Path import torch from transformers import AutoModel, AutoTokenizer VISION_FEATURE_DIM = 1536 HISTORY_FRAMES = 31 COARSE_TOKENS_PER_FRAME = 4 FINE_TOKENS_CURRENT_FRAME = 64 class MiniCPMRobotTrackInference: """Tokenizer and model wrapper for MiniCPM-RobotTrack inference.""" def __init__( self, checkpoint_path: str | Path = "openbmb/MiniCPM-RobotTrack", device: str | torch.device | None = None, ): if device is None: device = "cuda" if torch.cuda.is_available() else "cpu" self.device = torch.device(device) checkpoint = str(checkpoint_path) self.tokenizer = AutoTokenizer.from_pretrained(checkpoint) if self.tokenizer.pad_token_id is None: self.tokenizer.pad_token = self.tokenizer.eos_token self.model = AutoModel.from_