ReSpAct: Harmonizing Reasoning, Speaking, and Acting

ReSpAct: Harmonizing Reasoning, Speaking, and Acting Towards Building Large Language Model-Based Conversational AI Agents
Vardhan Dongre, Xiaocheng Yang, Emre Can Acikgoz, Suvodip Dey, Gokhan Tur, Dilek Hakkani-Tür

This repository contains code for ReSpAct. If you find this work useful in your research, please cite:

@misc{dongre2024respactharmonizingreasoningspeaking,
      title={ReSpAct: Harmonizing Reasoning, Speaking, and Acting Towards Building Large Language Model-Based Conversational AI Agents}, 
      author={Vardhan Dongre and Xiaocheng Yang and Emre Can Acikgoz and Suvodip Dey and Gokhan Tur and Dilek Hakkani-Tür},
      year={2024},
      eprint={2411.00927},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2411.00927}, 
}

Quickstart

Create a virtual environment (recommended)

conda create -n respact python=3.12
conda activate respact
pip install -r requirements.txt

🤖 Alfworld

⚒️ Environment Setup

Warning

If you are using MacOS with an arm-based system, it is recommended to use

CONDA_SUBDIR=osx-64 conda create -n alfworld python=3.12
conda activate alfworld

Install with pip (python3.9+):

pip install alfworld

Download PDDL & Game files and pre-trained MaskRCNN detector:

export ALFWORLD_DATA=<storage_path>
alfworld-download

Run ReSpAct Experiments

export OPENAI_API_KEY=<key>

For Azure:

export AZURE_OPENAI_ENDPOINT=<endpoint>
export AZURE_OPENAI_KEY=<key>

The experiments are controlled via two yaml files, base_config.yaml to control the alfworld games and experiment_configs.yaml for the respact experimentation.

python main.py

🛒 WebShop

⚒️ Environment Setup

To setup, do the following:

1. Install Python 3.8.13
2. Install Java
3. Download the source code:

> git clone https://github.com/princeton-nlp/webshop.git webshop

4. Create a virtual environment using Anaconda and activate it

> conda create -n webshop python=3.8.13
> conda activate webshop

5. Install requirements into the webshop virtual environment via the setup.sh script

> ./setup.sh [-d small|all]

The setup script performs several actions in the following order:

• Installs Python dependencies listed in requirements.txt
• Downloads product and instruction data for populating WebShop
• Downloads spaCy en_core_web_lg model
• Construct search engine index from product, instruction data
• Downloads 50 randomly chosen trajectories generated by MTurk workers The -d flag argument allows you to specify whether you would like to pull the entire product + instruction data set (-d all) or a subset of 1000 random products (-d small).

6. By default the WebShop only loads 1,000 products for a faster environment preview. To load all products, change web_agent_site/utils.py:

# DEFAULT_ATTR_PATH = join(BASE_DIR, '../data/items_ins_v2_1000.json')
# DEFAULT_FILE_PATH = join(BASE_DIR, '../data/items_shuffle_1000.json')
DEFAULT_ATTR_PATH = join(BASE_DIR, '../data/items_ins_v2.json')
DEFAULT_FILE_PATH = join(BASE_DIR, '../data/items_shuffle.json')

🛠️ Usage

The WebShop environment can be rendered in two modes - html and simple - each of which offer a different observation space. The simple mode strips away the extraneous meta-data that the html mode includes to make model training and evaluation easier.

Webpage Environment (html mode)

Launch the WebShop webpage:

> ./run_dev.sh

The site should then be viewable in the browser. Go to http://localhost:3000/ABC, where you should land on the search home page with a random instruction.

Run ReSpAct Experiments

export OPENAI_API_KEY=<key>

For Azure:

export AZURE_OPENAI_ENDPOINT=<endpoint>
export AZURE_OPENAI_KEY=<key>

🧙‍♂️ MultiWOZ

🚀 Setup

Our code is implemented in Python. To setup, do the following:

1. Go to the multiwoz folder:

> cd ./multiwoz

2. Create a virtual environment using Anaconda and activate it:

> conda create -n multiwoz python=3.10
> conda activate multiwoz

3. Install requirements into the multiwoz virtual environment:

> pip install -r requirements.txt --no-dependencies

4. Download the source code:

> git clone "https://github.com/smartyfh/MultiWOZ2.4.git"

5. Preprocess the dataset:

> cd "./MultiWOZ2.4"
> python create_data.py
> cd ".."

🛠️ Usage

Offline Evaluation

For offline (traditional) evaluation, please follow the following instructions:

1. Open the offline_evaluate.ipynb file.
2. In the Data Process section, change the sample variable. If it is an integer, then that many of dialogues will be sampled from the test set. If it is None, then the whole test set will be used.
3. In the Client Config section, change the client configuration to use your own one.
4. In the Evaluation section, change the save_dir to the folder you prefer.
5. (Optional) Feel free to use your own prompt and pass it to prompt in the evaluator object.
6. (Igore the first two blocks if you have downloaded and preprocessed the dataset.) Run the offline_evaluate.ipynb file.
7. (Optional) Clean the generated output:

> python clean.py --file_name <generated_file_name> --new_file_name <cleaned_file_name>

8. Format the generated output:

> python eval_postprocess.py --file_name <file_name> --new_file_name <formatted_file_name>

9. Use MultiWOZ Helper to obtain the scores.

Online Evaluation

For offline (traditional) evaluation, please follow the following instructions:

1. Open the online_evaluate.ipynb file.
2. In the Data Process section, change the sample variable. If it is an integer, then that many of dialogues will be sampled from the test set. If it is None, then the whole test set will be used.
3. In the Client Config section, change the client_config for the assistant and the user_client_config for the user simulator to use your own ones.
4. In the Evaluation section, change the save_dir to the folder you prefer.
5. (Optional) Feel free to use your own prompt and pass it to prompt in the evaluator object.
6. (Igore the first two blocks if you have downloaded and preprocessed the dataset.) Run the online_evaluate.ipynb file.
7. (Optional) Clean the generated output:

> python clean.py --file_name <generated_file_name> --new_file_name <cleaned_file_name>

8. Format the generated output:

> python eval_postprocess_autotod.py --file_name <file_name> --new_file_name <formatted_file_name>

9. Use AutoTOD to obtain the scores.

Interactive Mode

ReAct.ipynb provides an example for interactive mode. It supports both plain text interaction and gradio interaction.