Evaluating Causal Reasoning of LLMs

Commands

Setup

If the data.zip was downloaded separately, unzip it and place it inside the main project folder, at the same level as the src folder.

conda create --name causal-plan-reasoning python=3.8
conda activate causal-plan-reasoning
pip install -r requirements.txt
cd src/

Create a config.py file within src/ and add the following code with your filled in OpenAI API key:

config = {
    "OPENAI_API_KEY": "sk-<your-api-key>",
    "GEMINI_API_KEY": "<your-api-key>",
}

Question Creation from Processed FlowGraph Corpus

python flowgraph_question_creation.py --input_file ../data/processed_flowgraph/train.jsonl --output_dir ../data/generated_questions/train_must_why/ --prompt_name must_why_prompt
python flowgraph_question_creation.py --input_file ../data/processed_flowgraph/val.jsonl --output_dir ../data/generated_questions/val_must_why/ --prompt_name must_why_prompt
python flowgraph_question_creation.py --input_file ../data/processed_flowgraph/test.jsonl --output_dir ../data/generated_questions/test_must_why/ --prompt_name must_why_prompt

This creates the following question types:

• dependent_real - Given the relation Step B depends on Step A (A->B) and Step A is presented before Step B in plan text
• nondependent_real - Given the relation Step B does not depend on Step A and Step A is presented before Step B in plan text
• nondependent_switched_real - Given the relation Step B does not depend on Step A and Step A is presented before Step B in plan text, with a twist that the relevant steps are swapped in the plan itself

Answer Generation

See the scripts in src/ folder for commands to generate answers from various models