Dyna-Mind

Dyna-Mind: Learning to Simulate from Experience for Better AI Agents

We present ReSim and Dyna-GRPO for building simulation-capable agents for long-horizon interactive tasks. ReSim teaches agents to reason with grounded simulations constructed from search trees expanded through real environment interaction, while Dyna-GRPO further improves policy quality with online reinforcement learning over both outcome rewards and intermediate interaction feedback.

Dyna-Mind is designed to improve agent reasoning, planning, and action in environments where next-step intuition alone is insufficient. The repository currently focuses on the three benchmarks used in the paper: Sokoban, ALFWorld, and AndroidWorld.

📢 Updates

• 2026/04/22: initial repo release
• 2026/01/26: Accepted by ICLR 2026!

Installation

Install veRL

conda create -n dyna-mind python==3.10
conda activate dyna-mind

pip3 install -e .

pip3 install vllm==0.9.0
pip3 install flash-attn==2.7.3 --no-build-isolation
pip3 install transformers==4.51.0
pip3 install flashinfer-python
pip3 install antlr4-python3-runtime==4.9.3

Install Supported Environments

Install ALFworld

pip3 install gymnasium==0.29.1
pip3 install stable-baselines3==2.6.0
pip install alfworld
pip install opencv-python==4.12.0.88

Download PDDL & Game files and pre-trained MaskRCNN detector (will be stored in ~/.cache/alfworld/):

alfworld-download -f

Use --extra to download pre-trained checkpoints and seq2seq data.

Play a Textworld game:

alfworld-play-tw

Install Sokoban

pip install matplotlib==3.10.3
pip install gym==0.26.2
pip install gym_sokoban==0.0.6

Install AndroidWorld

Note: the following DOES NOT need to be installed in the same machine as the one running the training script. To communicate with the androidworld instances, we only need to know a url and a port number to access the androidworld docker instance.

Install androidworld from https://github.com/google-research/android_world and build the androidworld docker image

docker build -t android_world:latest .

Make sure the following runs:

docker run --privileged -p xxxx:5000 -it android_world:latest

For batch training / inference, we use the manager server at android_manager_server.py to coordinate a bank of AndroidWorld docker instances.

The script assumes:

• the AndroidWorld containers are already running
• container public ports start at 55200 and increase contiguously (55200, 55201, ...)
• the manager server itself listens on port 55008

Example: if you want 8 environments, first start 8 docker containers:

docker run --privileged -p 55200:5000 -d android_world:latest
docker run --privileged -p 55201:5000 -d android_world:latest
docker run --privileged -p 55202:5000 -d android_world:latest
docker run --privileged -p 55203:5000 -d android_world:latest
docker run --privileged -p 55204:5000 -d android_world:latest
docker run --privileged -p 55205:5000 -d android_world:latest
docker run --privileged -p 55206:5000 -d android_world:latest
docker run --privileged -p 55207:5000 -d android_world:latest

Then start the manager server:

python agent_system/environments/env_package/androidworld/server/android_manager_server.py \
  --n_envs 8 \
  --reset_concurrency 4 \
  --step_concurrency 8 \
  --close_concurrency 8

For a larger setup, increase --n_envs and launch the corresponding number of docker containers on ports 55200 + i. The current script examples use:

• env_urls to point directly to the AndroidWorld docker instances
• manager_vm_urls to point to the manager server, typically http://127.0.0.1:55008

If you run the AndroidWorld containers on another machine, update the host part of env_urls and manager_vm_urls in the AndroidWorld scripts under examples/dyna_grpo and inference/androidworld accordingly.

Run

The main entry scripts follow the two-stage setup in Dyna-Mind: ReSim for stage 1 and Dyna-GRPO for stage 2. ReSim is the simulation-grounding stage, and Dyna-GRPO is the online RL stage built on top of those simulation-capable policies.

ReSim

Use the scripts under inference/*/resim:

bash inference/sokoban/resim/run_sokoban-text.sh
bash inference/alfworld/resim/run_alfworld-text.sh
bash inference/androidworld/resim/run_androidworld.sh

Dyna-GRPO

Use the scripts under examples/dyna_grpo:

bash examples/dyna_grpo/run_sokoban_dynagrpo.sh
bash examples/dyna_grpo/run_alfworld_dynagrpo.sh
bash examples/dyna_grpo/run_androidworld_dynagrpo.sh

The corresponding GRPO baseline scripts are:

bash examples/dyna_grpo/run_sokoban_grpo.sh
bash examples/dyna_grpo/run_alfworld_grpo.sh
bash examples/dyna_grpo/run_androidworld_grpo.sh

Citation

@misc{yu2025dynamindlearningsimulateexperience,
      title={Dyna-Mind: Learning to Simulate from Experience for Better AI Agents}, 
      author={Xiao Yu and Baolin Peng and Michel Galley and Hao Cheng and Qianhui Wu and Janardhan Kulkarni and Suman Nath and Zhou Yu and Jianfeng Gao},
      year={2025},
      eprint={2510.09577},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2510.09577}, 
}

Acknowledgement: this repo is built on top of https://github.com/langfengq/verl-agent.