CommonRoad solution checker verdict reference

תוכן זה אינו זמין עדיין בשפה שלך.

Generate a verdict with the SDK command:

drawtonomy-cr verdict scenario.xml solution.xml

When both sides provide a solution or scenario fingerprint, a mismatch rejects the verdict and preserves the existing replay and verdict. Missing fingerprints remain compatible: the verdict loads as identity unchecked, with an amber warning that stays until dismissed and a separate CHECK ? badge. Its tooltip shows the reported checks, which do not affect the RUN result or add FAIL markers to the timeline. A scenario ID alone cannot distinguish another planner run or changed road/obstacle data.

From drawtonomy-commonroad 0.2.0, drawtonomy-cr verdict writes both fingerprints, so its results load as verified. Verdicts written by 0.1.0, or by other tools without fingerprints, still load but on the unchecked path. Re-run the checker for changed inputs; do not manually add a fingerprint to an old result.

The replay side is matched the same way, whether it is a solution or a planning trace. From drawtonomy-commonroad 0.3, a trace carries the solutionFingerprint of the solution its driven states match, so a trace and its verdict pair and load as verified. Traces without a solution fingerprint show as unverified: they load, but the verdict stays on the CHECK ? path.

Two different things say PASS or FAIL about a run, and they answer two different questions.

	Who asks it	What it means	When you get it
The official checks	`commonroad-drivability-checker`	did this solution actually solve the planning problem?	when a `.verdict.json` is loaded next to the solution
drawtonomy’s own checks	drawtonomy, during playback	did this run trip a fail condition you wrote?	always, with no extra file

drawtonomy’s own check is the immediate one you get without installing anything; the checker’s is the authority. Neither overwrites the other: a verdict can add a failure, never remove one. If the checker passed a solution that drawtonomy saw collide, both answers are shown.

The seven official checks

drawtonomy-cr verdict runs exactly these, in this order, and writes what they said:

Check	Fails when
`solved_all_problems`	the file does not carry a solution for every planning problem the scenario states.
`goal_reached`	no state of the trajectory satisfies the planning problem’s goal: its time-step window, its position, and the velocity and orientation intervals if they were set.
`starts_at_correct_state`	the solution’s first state is not the planning problem’s `initialState`. Every attribute is compared, with a tolerance of 0.1 (2.0 for velocity).
`obstacle_collision`	the ego’s footprint overlaps a scenario obstacle. Reported with the time-step range and the obstacle id.
`boundary_collision`	the ego leaves the drivable road boundary. Reported as SKIP when the `triangle` package it needs is missing.
`ego_collision`	two ego vehicles of a cooperative solution collide with each other. On a single-ego solution there is nothing to collide with, so it passes.
`solution_feasible`	the recorded trajectory is not reproducible by the vehicle model the solution declares (KS, PM, …): the states cannot be reached with admissible inputs.

All seven results come from drawtonomy-cr verdict. drawtonomy does not run these checks itself, and its own RUN PASS / FAIL judges only the fail conditions you wrote. Two of the official rules are also ported into the app, with the official tolerances, so you can see them without Docker or Linux:

starts_at_correct_state: a warning when the solution is loaded.
goal_reached: with a CommonRoad scenario open, the RUN badge tooltip gets one line, Goal: reached at step 187 (18.7 s) or Goal: not reached (never entered goal lanelet 2), and the timeline gets a flag at the first step that meets the goal. This is information, not a verdict: it never turns the badge into FAIL, because a minimum risk maneuver that stops short of the goal can be the right answer. It is evaluated against the goal that Export CommonRoad would write right now, so it follows your edits to the goal lane, speed, time and heading. When that goal cannot be evaluated (the export is blocked, for example because no Goal lane is picked), the line and the flag are not shown. A planning trace does not get this line.

A trajectory can also be collision-free and still be something no car could drive; only solution_feasible in the verdict tells you that.

SKIP: a check that could not be judged

A check can also come back SKIP, which is neither a pass nor a failure: the checker was unable to reach an answer, so there is nothing to say about the solution. Today that happens in exactly one case. boundary_collision needs the triangle package to triangulate the road, and triangle is not installed by default because it is not free for commercial use. Install it with the [boundary] extra (pip install "drawtonomy-commonroad[checker,boundary]").

An exception that says anything else, including the ego actually leaving the road, is still a FAIL: a missing package must not read as “your trajectory left the road”.

A SKIP is excluded from the badge’s count. Six checks judged and passed with boundary_collision skipped reads Checker PASS 6/6, not PASS 6/7. The denominator is always the number of checks that actually had an answer, and the numerator counts the word in front of it: PASS n/N is n passed, FAIL m/N is m failed. So Checker FAIL 3/7 means three of the seven judged checks came back FAIL.

On the CLI the line names the next step in brackets, and the exit code is still 0:

[SKIP] boundary_collision (pip install triangle)

drawtonomy’s own checks

What runs during playback is your FAIL CONDITIONS, most importantly Collision, which is on by default for every scenario, drawn or imported. A scenario that is meant to collide is a FAIL until you remove that condition. This is a runtime check against the fail conditions only, not a validation of the scenario.

Reading the badge

After a run, the transport row carries the RUN badge. Only a fingerprint-verified verdict contributes checker results to its tooltip, RUN evaluation or FAIL timeline markers.

An unverified verdict appears on a separate amber CHECK ? badge. Its tooltip contains the checks reported in the file and the reason identity could not be verified. These reported checks do not alter RUN or create FAIL markers. Verdicts written without fingerprints (drawtonomy-commonroad 0.1.0, or other tools) follow this unchecked path; drawtonomy’s own runtime checks remain independent.

The screenshots and detailed PASS/FAIL descriptions below illustrate the verified-verdict path. The screenshots predate fingerprint pairing and the seven-check release, so they show 4/4 and 2/4; a verified sidecar with all seven checks has denominator 7. They do not depict the current unchecked badge.

A solution the checker accepted. The tooltip names the replay file, the vehicle model (KS), the state count, and how many checks passed. Checks that passed are counted, not listed.

A FAIL. Each check that did not pass gets its own line: the check name, the obstacle involved and the seconds it happened. When the FAIL comes from the checker, that block is the reason, so the badge does not repeat it as a separate Fail condition: line.

Three things to read off it:

Checker PASS 7/7 / Checker FAIL 2/7: the summary. The denominator is the number of checks that were judged, so it is 7 for a full sidecar and smaller when one was skipped or the sidecar simply did not carry it — the screenshots above, written before the seven-check release, read 4/4 and 2/4 for that reason. The numerator counts the word in front of it: 7 checks passed, 2 checks failed. The same words appear in the toast when the verdict is loaded, so you can find it again after the toast is gone.
One line per check that did not pass: the check name, the obstacle id and the time range where the checker has them, separated by · (the same shape the timeline’s fail marker uses). Checks that passed are only counted in the summary. A skipped check reads boundary_collision · SKIP and keeps its explanation. The checker’s own boilerplate sentence (“There is a collision between the scenario obstacles and the ego vehicle…”) is dropped; a real diagnosis (“76 of 179 state transitions … position drifts up to 149.6 cm”) is kept, indented under its check.
The times are seconds, converted from the sidecar’s time steps. A check that failed at a specific time also plants a marker on the timeline at that second; a check with no time (a judgement about the whole trajectory, like a bare solution_feasible) is placed at the end of the replay and reads whole trajectory.

The sidecar: `drawtonomy-verdict/1`

The verdict is a small JSON file next to the solution. This one is the checker’s answer for a solution that collides:

{
  "schema": "drawtonomy-verdict/1",
  "solutionFingerprint": "sha256:aacc4bad344c5b62adf139dd9889319cc3236f28d38f756445965b0862c09b50",
  "scenarioFingerprint": "sha256:914a4300443dcf4bc585c7e43769983b4932c22bf69be37ac02bd086c1d5ab33",
  "benchmarkId": "PM1:JB1:ZAM_Untitled202609011139-1_1_T-1:2020a",
  "scenarioId": "ZAM_Untitled202609011139-1_1_T-1",
  "dt": 0.1,
  "tool": { "name": "commonroad-drivability-checker", "version": "2025.4.0" },
  "generatedAt": "2026-09-03T04:52:49Z",
  "checks": [
    { "name": "obstacle_collision", "status": "FAIL",
      "message": "CollisionException: There is a collision between the scenario obstacles and the ego vehicle in planning problem solution 16",
      "timeSteps": [127, 133], "obstacleId": 15 },
    { "name": "boundary_collision", "status": "PASS" },
    { "name": "goal_reached", "status": "PASS" },
    { "name": "solution_feasible", "status": "FAIL",
      "message": "Exception: infeasible for planning problems [16]",
      "vehicleModel": "PM" }
  ]
}

schema is required. checks[].status is PASS, FAIL or SKIP.
checks[].name is a free string; the seven official names above are the ones drawtonomy knows how to place on the timeline.
message is the official exception verbatim ({type}: {text}). The display drops the type prefix and shows the text.
timeSteps is [first, last], inclusive, counted from 0, so seconds are step × dt.

You do not have to use drawtonomy-cr to produce it. It is small enough to write from your own CI job; the judging half is seven official calls. Watch the argument order: it is not the same for all of them.

from commonroad.common.file_reader import CommonRoadFileReader
from commonroad.common.solution import CommonRoadSolutionReader
from commonroad_dc.feasibility.solution_checker import (
    solved_all_problems, goal_reached, starts_at_correct_state,
    obstacle_collision, boundary_collision, ego_collision, solution_feasible)
from commonroad_dc.feasibility.vehicle_dynamics import VehicleDynamics

scenario, pps = CommonRoadFileReader("scenario.xml").open()
solution = CommonRoadSolutionReader.open("solution.xml")
checks = [
    ("solved_all_problems", lambda: solved_all_problems(pps, solution)),
    ("goal_reached", lambda: goal_reached(scenario, pps, solution)),
    ("starts_at_correct_state", lambda: starts_at_correct_state(solution, pps)),
    ("obstacle_collision", lambda: obstacle_collision(scenario, pps, solution)),
    ("boundary_collision", lambda: boundary_collision(scenario, pps, solution)),
    ("ego_collision", lambda: ego_collision(scenario, pps, solution)),
]
for name, fn in checks:
    try:
        fn(); print("PASS", name)
    except Exception as e:
        print("FAIL", name, e)

vehicle_dynamics = VehicleDynamics.from_model(
    solution.vehicle_model, solution.vehicle_type)
print(solution_feasible(solution, scenario.dt, vehicle_dynamics))

valid_solution(scenario, pps, solution) runs all seven in one call and raises on the first one that fails; the loop above is the same judging, kept apart so every check gets its own line in the sidecar.

Feasibility says more than pass or fail

The official checker returns a boolean per planning problem, which does not say where or why. So drawtonomy-cr re-runs the official state_transition_feasibility on every pair of consecutive states and reports which transitions the vehicle model cannot reproduce, which limit the reconstructed input hit, and how far the recorded state drifts from the simulated one. Every number is an official API return value. A KS solution from the official corpus scenario ZAM_Tjunction-1_42_T-1 gets:

{ "name": "solution_feasible", "status": "FAIL",
  "message": "14 of 147 state transitions (4.2-6.1 s) need a steering rate beyond the KS limit of 0.4 rad/s; position drifts up to 5.8 cm from the simulated state (tolerance 2 cm), orientation up to 0.019 rad (tolerance 0.03).",
  "timeSteps": [42, 61], "reason": "steering_rate",
  "infeasibleTransitions": 14, "transitions": 147,
  "maxPositionError": 0.0581, "maxOrientationError": 0.0186,
  "steeringRateLimit": 0.4, "accelerationLimit": 11.5, "vehicleModel": "KS" }

reason is whichever limit was hit most often: steering_rate, acceleration, friction_circle, input_bounds or state_deviation. Because timeSteps is present, the fail marker sits at 4.2 s instead of at the end of the run.

Pairing: a verdict belongs to one solution

solutionFingerprint and scenarioFingerprint are optional sha256: values followed by 64 lowercase hex digits. Hash the UTF-8 text after removing a leading BOM and normalizing CRLF/CR to LF. Compare a fingerprint only when both sides provide it: a mismatch rejects the verdict; absence marks identity as unchecked and keeps the legacy loading path. The amber warning requires manual dismissal. A separate CHECK ? badge explains the missing verification and shows the reported checks in its tooltip; these checks do not affect RUN or the FAIL timeline. Editing the scene clears the currently attached verdict and invalidates the original scenario fingerprint; reloading without that identity remains unchecked.

A verdict is matched using scenario id, with SHA-256 fingerprints compared when available on both sides: scenarioId, or benchmarkId with the <model>:<cost>: prefix and the :2020a suffix stripped. If it does not match, the verdict is not applied, and one line says why, naming both ids:

other_scenario.verdict.json was not applied: it was computed for scenario
"ZAM_Other-1_1_T-1" but the loaded replay is for "ZAM_CutIn-1_1_T-1".

The replay itself stays loaded. Two related rules:

A verdict on its own does nothing. With no solution loaded there is nothing to judge; a toast asks you to load the solution first, or drop both together.
Editing the scene drops the verdict but keeps the replay. The checker’s answer was computed against the scene as it was. The ego keeps following the planner’s trajectory, and a toast says the trace predates the edit. Rerun the planner for an answer about the edited scene.

Exit codes

drawtonomy-cr verdict:

code
`0`	the sidecar was written. A FAIL verdict is a 0: the failure lives inside the JSON.
`3`	the checker is not installed (the `[checker]` extra, Linux x86_64). One line says how to install it.

Bring your own planner: producing the verdict, and the loop it belongs to.
Planning trace: what the planner intended, not just what it drove.
End and fail conditions: drawtonomy’s own side of the PASS / FAIL.