CommonRoad solution checker verdict reference

Two different things say PASS or FAIL about a run, and they answer two different questions.

	Who asks it	What it means	When you get it
The official checks	commonroad-drivability-checker	did this solution actually solve the planning problem?	when a .verdict.json is loaded next to the solution
drawtonomy’s own checks	drawtonomy, during playback	did this run trip a fail condition you wrote?	always, with no extra file

drawtonomy’s own check is the immediate one you get without installing anything; the checker’s is the authority. Neither overwrites the other: a verdict can add a failure, never remove one. If the checker passed a solution that drawtonomy saw collide, both answers are shown.

The four official checks

drawtonomy-cr verdict runs exactly these, and writes what they said:

Check	Fails when
obstacle_collision	the ego’s footprint overlaps a scenario obstacle. Reported with the time-step range and the obstacle id.
boundary_collision	the ego leaves the drivable road boundary. Reported as SKIP when the triangle package it needs is missing.
goal_reached	the trajectory never satisfies the planning problem’s goal (position, and the speed interval if one was set).
solution_feasible	the recorded trajectory is not reproducible by the vehicle model the solution declares (KS, PM, …): the states cannot be reached with admissible inputs.

solution_feasible is the one drawtonomy cannot reach on its own: a trajectory can be collision-free and still be something no car could drive.

SKIP: a check that could not be judged

A check can also come back SKIP, which is neither a pass nor a failure: the checker was unable to reach an answer, so there is nothing to say about the solution. Today that happens in exactly one case. boundary_collision needs the triangle package to triangulate the road, and triangle is not installed by default because it is not free for commercial use. Install it with the [boundary] extra (pip install "drawtonomy-commonroad[checker,boundary]").

An exception that says anything else, including the ego actually leaving the road, is still a FAIL: a missing package must not read as “your trajectory left the road”.

A SKIP is excluded from the badge’s count. Three checks judged and passed with boundary_collision skipped reads Checker PASS 3/3, not PASS 3/4. The denominator is always the number of checks that actually had an answer, and the numerator counts the word in front of it: PASS n/N is n passed, FAIL m/N is m failed. So Checker FAIL 3/4 means three of the four judged checks came back FAIL.

On the CLI the line names the next step in brackets, and the exit code is still 0:

[SKIP] boundary_collision (pip install triangle)

drawtonomy’s own checks

What runs during playback is your FAIL CONDITIONS, most importantly Collision, which is on by default for every scenario, drawn or imported. A scenario that is meant to collide is a FAIL until you remove that condition. This is a runtime check against the fail conditions only, not a validation of the scenario.

Reading the badge

After a run, the transport row carries a badge. With a verdict loaded, the checker’s answer is folded into its tooltip.

A solution the checker accepted. The tooltip names the replay file, the vehicle model (KS), the state count, and how many checks passed. Checks that passed are counted, not listed.

A FAIL. Each check that did not pass gets its own line: the check name, the obstacle involved and the seconds it happened. When the FAIL comes from the checker, that block is the reason, so the badge does not repeat it as a separate Fail condition: line.

Three things to read off it:

• Checker PASS 4/4 / Checker FAIL 2/4: the summary. The numerator counts the word in front of it: 4 checks passed, 2 checks failed. The same words appear in the toast when the verdict is loaded, so you can find it again after the toast is gone.
• One line per check that did not pass: the check name, the obstacle id and the time range where the checker has them, separated by · (the same shape the timeline’s fail marker uses). Checks that passed are only counted in the summary. A skipped check reads boundary_collision · SKIP and keeps its explanation. The checker’s own boilerplate sentence (“There is a collision between the scenario obstacles and the ego vehicle…”) is dropped; a real diagnosis (“76 of 179 state transitions … position drifts up to 149.6 cm”) is kept, indented under its check.
• The times are seconds, converted from the sidecar’s time steps. A check that failed at a specific time also plants a marker on the timeline at that second; a check with no time (a judgement about the whole trajectory, like a bare solution_feasible) is placed at the end of the replay and reads whole trajectory.

The sidecar: drawtonomy-verdict/1

The verdict is a small JSON file next to the solution. This one is the checker’s answer for a solution that collides:

{
  "schema": "drawtonomy-verdict/1",
  "benchmarkId": "PM1:JB1:ZAM_Untitled202609011139-1_1_T-1:2020a",
  "scenarioId": "ZAM_Untitled202609011139-1_1_T-1",
  "dt": 0.1,
  "tool": { "name": "commonroad-drivability-checker", "version": "2025.4.0" },
  "generatedAt": "2026-09-03T04:52:49Z",
  "checks": [
    { "name": "obstacle_collision", "status": "FAIL",
      "message": "CollisionException: There is a collision between the scenario obstacles and the ego vehicle in planning problem solution 16",
      "timeSteps": [127, 133], "obstacleId": 15 },
    { "name": "boundary_collision", "status": "PASS" },
    { "name": "goal_reached", "status": "PASS" },
    { "name": "solution_feasible", "status": "FAIL",
      "message": "Exception: infeasible for planning problems [16]",
      "vehicleModel": "PM" }
  ]
}

• schema is required. checks[].status is PASS, FAIL or SKIP.
• checks[].name is a free string; the four official names above are the ones drawtonomy knows how to place on the timeline.
• message is the official exception verbatim ({type}: {text}). The display drops the type prefix and shows the text.
• timeSteps is [first, last], inclusive, counted from 0, so seconds are step × dt.

You do not have to use drawtonomy-cr to produce it. It is small enough to write from your own CI job; the judging half is four official calls:

from commonroad.common.file_reader import CommonRoadFileReader
from commonroad.common.solution import CommonRoadSolutionReader
from commonroad_dc.feasibility.solution_checker import (
    obstacle_collision, boundary_collision, goal_reached, solution_feasible)

scenario, pps = CommonRoadFileReader("scenario.xml").open()
solution = CommonRoadSolutionReader.open("solution.xml")
for name, fn in [("obstacle_collision", obstacle_collision),
                 ("boundary_collision", boundary_collision),
                 ("goal_reached", goal_reached)]:
    try:
        fn(scenario, pps, solution); print("PASS", name)
    except Exception as e:
        print("FAIL", name, e)
print(solution_feasible(solution, scenario.dt, pps))

Feasibility says more than pass or fail

The official checker returns a boolean per planning problem, which does not say where or why. So drawtonomy-cr re-runs the official state_transition_feasibility on every pair of consecutive states and reports which transitions the vehicle model cannot reproduce, which limit the reconstructed input hit, and how far the recorded state drifts from the simulated one. Every number is an official API return value. A KS solution from the official corpus scenario ZAM_Tjunction-1_42_T-1 gets:

{ "name": "solution_feasible", "status": "FAIL",
  "message": "14 of 147 state transitions (4.2-6.1 s) need a steering rate beyond the KS limit of 0.4 rad/s; position drifts up to 5.8 cm from the simulated state (tolerance 2 cm), orientation up to 0.019 rad (tolerance 0.03).",
  "timeSteps": [42, 61], "reason": "steering_rate",
  "infeasibleTransitions": 14, "transitions": 147,
  "maxPositionError": 0.0581, "maxOrientationError": 0.0186,
  "steeringRateLimit": 0.4, "accelerationLimit": 11.5, "vehicleModel": "KS" }

reason is whichever limit was hit most often: steering_rate, acceleration, friction_circle, input_bounds or state_deviation. Because timeSteps is present, the fail marker sits at 4.2 s instead of at the end of the run.

Pairing: a verdict belongs to one solution

A verdict is matched to the loaded replay by scenario id: scenarioId, or benchmarkId with the <model>:<cost>: prefix and the :2020a suffix stripped. If it does not match, the verdict is not applied, and one line says why, naming both ids:

other_scenario.verdict.json was not applied: it was computed for scenario
"ZAM_Other-1_1_T-1" but the loaded replay is for "ZAM_CutIn-1_1_T-1".

The replay itself stays loaded. Two related rules:

• A verdict on its own does nothing. With no solution loaded there is nothing to judge; a toast asks you to load the solution first, or drop both together.
• Editing the scene drops the verdict but keeps the replay. The checker’s answer was computed against the scene as it was. The ego keeps following the planner’s trajectory, and a toast says the trace predates the edit. Rerun the planner for an answer about the edited scene.

Exit codes

drawtonomy-cr verdict:

code
0	the sidecar was written. A FAIL verdict is a 0: the failure lives inside the JSON.
3	the checker is not installed (the [checker] extra, Linux x86_64). One line says how to install it.

Next

• Bring your own planner: producing the verdict, and the loop it belongs to.
• Planning trace: what the planner intended, not just what it drove.
• End and fail conditions: drawtonomy’s own side of the PASS / FAIL.