CAD-Assistant: Tool-Augmented VLLMs as Generic CAD Task Solvers

Dimitrios Mallis, Ahmet Serdar Karadeniz, Sebastian Cavada, Danila Rukhovich, Niki Foteinopoulou, Kseniya Cherenkova, Anis Kacem, Djamila Aouada

This repository will contain the official codebase for our ICCV 2025 paper.

📄 CAD-Assistant Paper | 🌐 CAD-Assistant Project Page | 📼 CAD-Assistant Video

News:

• 🚨 The 3D CAD question answering and the solid_recognizer tool is now released.

🚀 Overview

CAD-Assistant is a tool-augmented VLLM framework for AI-assisted CAD. Our framework generates FreeCAD code that is executed within CAD software directly. CAD-Assistant can process multimodal inputs, including textual queries, sketches, drawn commands and 3D scans.

This release includes the CAD-Assistant framework implementation intergrated with FreeCAD and augmented with a 2D CAD sketch renderering tool and a 3D solid recognizer tool. Evaluation is performed on both the 2D and 3D partition of the SGP-Bench. The code also allows for easy intergration of novel tools.

🛠️ Installation

Configuration

Edit config.json and add your OpenAI API key:

  "openai_api_key": "your-openai-api-key-here",

Docker Installation

The easiest way to get started is using Docker, which includes all dependencies including FreeCAD and Python OCC:

1. Clone the repository

   git clone <your-repo-url>
   cd CAD-Assistant

2. Build and run with Docker

   docker build -t cad-assistant .
   docker run -it -v $(pwd):/app cad-assistant

🎯 Evaluation for CAD Question Answering

This repository includes evaluation on both the 2D and 3D partition of the SGP-bench dataset. We provide preprocessed SGP-bench test samples in the sgp_bench_samples/ directory, were CAD sketches or CAD models are preloaded to FreeCAD .FCStd files. The directory includes the 1700 FreeCAD .FCStd test sketch files (700 for the 2D partitiona and 1000 for the 3D partition) organized directories (2D_0001/, 2D_0002/, 3D_0001/, etc.)

Quick Test (5 samples, sequential, 3D partition)

python run_cad_evaluation.py --subject 3D --limit 5 --debug

Full Evaluation (all samples, parallel, 2D partition)

python run_cad_evaluation.py --parallel --subject 2D --workers 4

After running evaluation, a JSON results file will appear in the root directory and intermediate logs per sample will be saved in the .logs/ folder.

To perform full evaluation within the docker container you can use the following command. Simply substitute the eval partition (subject) and the number of workers you prefer.

  docker run -d --rm \
    --name cad-eval-3d \
    -v $(pwd):/app \
    -w /app \
    -e DISPLAY=:99 \
    cad-assistant \
    bash -c "python run_cad_evaluation.py --subject 3D --parallel --workers 4 --output cad_evaluation_results_3d.json"

🔧 CAD-Specific Tools

• The sketch_recognizer tool

Current release includes the 2D CAD sketch renderering tool sketch_recognizer. The code implementation of the sketch_recognizer is adapted from Vitruvion.

Is Construction Flag: The sketch rendering function loads construction information from 2D_****_isconstruction.json since FreeCAD doesn't support an is construction flag stored in the .FCStd file format.

• The solid_recognizer tool

Current release also includes the 3D CAD solid renderering tool solid_recognizer. The code implementation of the solid_recognizer is using Python OCC.

Extending CAD-Assistant with New Tools and FreeCAD Commands

The CAD-Assistant framework is easily extendable with new tools and FreeCAD functions.

Adding New Tools: To add custom tools to the CAD-Assistant framework:

1. Create your tool: Add both the tool code ([cad-tool-name]_code.py) and tool documentation (cad-tool_doc.txt) in the cad_assistant/tools/ directory.
2. Register the tool: Add your tool to the tool registry in config.json

Adding New FreeCAD Commands: To extend the available FreeCAD commands simply extend the cad_assistant/prompts/freecad_doc.j2 file with your new FreeCAD command documentation

🛡️ License

This project is licensed under an Attribution-NonCommercial License.

📑 Citation

If you find this work useful for your research, please cite our paper:

@article{Mallis2024CADAssistantTV,
        title={CAD-Assistant: Tool-Augmented VLLMs as Generic CAD Task Solvers},
        author={Dimitrios Mallis and Ahmet Serdar Karadeniz and Sebastian Cavada and Danila Rukhovich and Niki Maria Foteinopoulou and Kseniya Cherenkova and Anis Kacem and Djamila Aouada},
        journal={International Conference on Computer Vision (ICCV)},
        year={2025},
      }