AlpaJudge: NVIDIA's Learned Judges for Autonomous-Driving Reasoning

AlpaJudge is a family of learned judges for evaluating autonomous-driving reasoning, providing reference-based and reference-free checkpoints. The reference-based checkpoint evaluates questions with a reference answer and a candidate answer, measuring alignment probability in [0, 1]. The reference-free checkpoint evaluates front-wide/front-tele frames, ego meta-action history and future, and a candidate rationale, assessing five rubric probabilities plus aggregated scores. Both variants are available from NVIDIA's GitHub repository and can be downloaded with proper authentication. The system supports usage questions and code issues through specific channels, and it includes security guidelines for vulnerability reporting.

The key findings highlight the importance of evaluating autonomous-driving reasoning through machine learning models, enabling more accurate and reliable assessments of AI systems in autonomous vehicles. The reference-based and reference-free checkpoints offer different evaluation approaches, allowing for flexibility in assessing various aspects of autonomous driving, such as visual input and decision-making processes. These tools are critical for ensuring the safety and reliability of self-driving technologies by providing structured ways to evaluate and improve AI reasoning capabilities.

AI-generated summary of the original article — see the source for the full story.