Skip to content

About

No description, website, or topics provided.

Resources

Stars

15 stars

Watchers

0 watching

Forks

Latest commit

 

History

5 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Towards Realistic Project-Level Code Generation via Multi-Agent Collaboration and Semantic Architecture Modeling

ProjectGen is a multi-agent framework that decomposes projects into architecture design, skeleton generation, and code filling stages with iterative refinement and memory-based context management.

ProjectGen

Code Structure Overview

The core implementation resides in the src/ directory, which contains three major components: the multi-agent system, the memory management module, and a set of workflow and utility scripts. The directory structure is shown below:

src/
├── agents/
│   ├── architecture_agent.py
│   ├── arch_judge_agent.py
│   ├── skeleton_agent.py
│   ├── skeleton_judge_agent.py
│   ├── code_agent.py
│   ├── code_judge_agent.py
│   └── test.py
│
├── memory_manager/
│   ├── arch_memory.py
│   ├── skeleton_memory.py
│   ├── code_memory.py
│   └── __init__.py
│
├── build_dependency_graph.py
├── extract_api.py
├── logger.py
├── main.py
├── prompts.py
├── utils.py
└── workflow.py
  • agents/: This directory contains the core agents used in the three-stage generation workflow. Each stage consists of a generation agent and a judging agent, forming a generate–evaluate–refine loop.

  • memory_manager/: The memory modules preserve intra-stage semantic information, enabling agents to efficiently access relevant context from previous iterations.

  • build_dependency_graph.py: Constructs file-level dependency graphs to support ordering.

  • extract_api.py: Extracts function signatures from generated code to support iteration.

  • logger.py: A unified logging module for debugging, tracing agent outputs, and monitoring workflow execution.

  • main.py: The main entry point of the system.

  • prompts.py: Contains prompt templates used by all agents.

  • utils.py: Provides general-purpose utility functions.

  • workflow.py: Defines the overall multi-agent generation workflow.

Installation

conda create -n projectgen python=3.9
conda activate projectgen
pip install -r requirements.txt

Usage

Open src/utils.py and fill in your OpenAI API key:

os.environ["OPENAI_API_KEY"] = "your_openai_api_key_here"

Navigate to the src directory and run the main script:

cd src
python main.py --dataset=CodeProjectEval

The outputs will be stored in the CodeProjectEval_outputs/ folder.

CodeProjectEval

To better reflect real-world project scenarios and support evaluation through executable test cases, we construct a new project-level code generation dataset, CodeProjectEval, which consists of 18 Python repositories covering a wide range of topics.

We further extended the dataset to cover the Java language, incorporating a total of 12 repositories.

Detailed Information

Repository (Python) #FILE #LOC Complexity #Check Tests (cov.) #Unit Tests (cov.) #PRD tokens
bplustree 8 1,509 2.29 8 (82%) 356 (98%) 1,339
cookiecutter 18 2,805 3.42 7 (55%) 375 (99%) 2,100
csvs-to-sqlite 3 816 5.83 10 (81%) 25 (88%) 1,841
deprecated 3 597 4.08 26 (80%) 176 (95%) 953
djangorestframework-simplejwt 31 2,014 2.09 8 (63%) 191 (93%) 1,614
flask 24 9,314 2.71 25 (52%) 482 (91%) 2,913
imapclient 17 3,531 2.81 9 (40%) 267 (80%) 3,810
parsel 5 1,128 2.60 5 (65%) 250 (95%) 1,522
portalocker 9 1,958 2.84 10 (58%) 71 (94%) 1,990
pyjwt 12 2,690 3.01 10 (53%) 294 (94%) 382
python-hl7 11 2,434 2.98 10 (56%) 100 (87%) 2,292
rsa 14 2,949 2.40 6 (73%) 100 (87%) 3,318
simpy 12 2,184 2.01 7 (60%) 149 (90%) 2,147
tinydb 10 2,170 1.76 10 (58%) 204 (95%) 947
trailscraper 13 890 2.01 4 (65%) 93 (92%) 3,415
voluptuous 7 3,100 2.55 11 (55%) 161 (90%) 1,221
xmnlp 24 1,504 3.47 8 (65%) 23 (81%) 3,105
zxcvbn 8 1,402 5.69 6 (81%) 31 (84%) 2,399
Avg. 12.7 2,388.6 3.03 10 (63.4%) 186 (90.7%) 2,067
Mid. 11.5 2,092 2.76 8.5 (61.5%) 168.5 (91.5%) 2,100
Repository (Java) # Files # LOC Complexity # Check Test(cov.) # Unit Test(cov.) # PRD tokens
compile-testing 24 4571 267 10(50.91%) 305(82.52%) 1,919
fuzzy-matcher 19 1422 117 8(70.44%) 76(92.04%) 2,192
geo 9 1243 114 7(61.99%) 88(99.46%) 1,124
geohash-java 13 1331 193 6(64.41%) 61(92.71%) 1,750
GoogleAuth 10 1689 84 2(61.80%) 9(71.12%) 1,826
hmm-lib 5 913 96 2(41.35%) 9(83.97%) 1,908
InMemoryJavaCompiler 6 256 13 1(66.28%) 6(88.37%) 2,059
java-emoji-converter 4 528 51 1(50.22%) 4(62.33%) 2,337
jetbrick-template-2x 111 9558 1306 8(45.20%) 152(69.57%) 2,308
JSON-java 26 10536 874 10(43.45%) 741(89.37%) 2,158
password4j 29 7080 515 7(65.74%) 240(96.14%) 2,146
SerialKiller 2 174 10 4(57.63%) 15(93.22%) 1380
Avg. 21.5 3275.1 303.1 5.5(56.6%) 142.2(85.1%) 1,925
Mid. 11.5 1376.5 115.5 6.5(59.7%) 68.5(88.9%) 1,989

Each repository is supplemented with:

  • docs/PRD.md: provides detailed descriptions of a software system’s functional and non-functional requirements, guiding subsequent design and development.
  • docs/UML_pyreverse.md: UML class diagram and package diagram generated by Pyreverse.
  • docs/architecture_design.md: the directory tree of the repository and descriptions for each source file, accompanied by summaries of the classes and functions they contain.
  • check_tests/: to provide initial verification during the code generation process.
  • unit_tests/: executed upon completion of code generation to evaluate the overall quality and functional correctness of the generated projects.

Test Scripts

  • Similarity-based Evaluation (SketchBLEU)

    We adopt SketchBLEU, a similarity-based metric originally introduced in the CodeS framework. This metric evaluates the structural similarity between the generated code and the reference implementation. To calculate SketchBLEU:

    cd datasets/evaluation
    python calc_sketchbleu.py
    
  • Unit Test–based Evaluation

    For functional correctness evaluation, each repository contains a set of unit tests. To run the unit tests:

    cd datasets/evaluation
    python calc_passrate.py
    

About

No description, website, or topics provided.

Resources

Stars

15 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages