CVPR 2026 Scenario Mining Challenge

Published: June 01, 2026

Challenge Summary

Autonomous Vehicles (AVs) collect and pseudo-label terabytes of multi-modal data localized to HD maps during normal fleet tests. However, identifying interesting and safety critical scenarios from uncurated data streams is prohibitively time-consuming and error-prone. Although prior works have explored this problem in the context of structured queries and hand-crafted heuristics, we are hosting this challenge to solicit better end-to-end solutions to this important problem.

To this end, our benchmark includes 10,000 planning-centric natural language queries. Challenge participants can use all RGB frames, Lidar sweeps, HD Maps, and track annotations from the AV2 sensor dataset to find relevant actors in each log. Methods will be evaluated at three levels of spatial and temporal granularity. First, methods must determine if an action (defined by a natural language query) occurs in the log. Next, if the action occurs in the log, methods must temporally localize (e.g. find the start and end time) of the action. Lastly, methods must detect and track all objects relevant to the text description. Our primary evaluation metric is HOTA-temporal, a tracking metric that jointly considers detection and association accuracy for referred objects.

Spatio-Temporal Track Winners

Team: HYU Oasis

Members: Jeongwoo Park, Yuseung Na, Seongjae Jeong, Minwon Lee, Kichun Jo

[Dowload Technical Report] [Code]

Temporal Track Winners

Team: MISI

Members: Songliang Cao, Jiele Zhao, Yuru Wang, Hao Li, Daqi Liu, Zehan Zhang, Fangzhen Li, Yu Wang, Yue Zhang, Bing Wang, Guang Chen, Hao Lu, Hangjun Ye

[Dowload Technical Report] [Code]