Text-to-CAD benchmark method

Prompt classes

Each class is scored separately so a pretty bracket cannot hide a failed assembly.

Simple prismatic

Plates, blocks, bushings and L-brackets with stated envelope and hole positions.

Standard elements

ISO metric fasteners, washers and catalogue envelopes such as a 6204 bearing.

Manufacturing features

Pockets, fillets, threads and wall thickness named with the intended process.

Assembly

Two or more separate solids that must mate; a single fused body is a fail when separate parts were required.

Deliberately underspecified

Prompts that omit a driving dimension. The run must not invent a load case or a standard the user did not name. Assumption is allowed; silent fabrication of extra features is a fail.

Pass / fail

A trial passes only if every gate below holds. Invalid or undelivered output counts as a loss, not as “excluded”.

1. A STEP file downloads and opens as a solid in a mechanical CAD system.
2. Body count matches the class (one solid for simple parts; two or more separate bodies for assembly).
3. Bounding box and volume sit inside the published band for that prompt.
4. Stated driving dimensions are present as editable parameters or as measured geometry within the band.
5. No extra functional members the prompt did not ask for (invented steps, sleeves, or holes).
6. Runtime, credit cost and the exact prompt string are recorded with the file.

Prompt set

Download the v1 prompt table (CSV). Expected volume bands are engineering envelopes, not shop-floor tolerances. Re-run the same rows under the same mode before comparing tools.

How we will publish results

When a public run is posted it will include every row: pass, fail, and timeout. Screenshots and STEP/STL will be linked per prompt. Competitor comparisons, if any, will use this same table and the same conditions — not a different prompt for each product.

Published runs

No complete public run has been published.