↕️ ir_axioms

Axiomatic constraints for information retrieval and retrieval-augmented generation.

ir_axioms is a Python framework for experimenting with axioms in information retrieval in a declarative way. It includes reference implementations of many commonly used traditional retrieval axioms as well as axioms for retrieval-augmented generation. The library is well integrated with the PyTerrier framework and supports Pyserini retrieval indices too. Use-cases of ir_axioms include search-result re-ranking, analyses of (neural) retrieval systems, and analyses of generated RAG answers.

Note: If you're looking out for ir_axioms<1.0, please go here.

Publications

Read more about the ir_axioms framework and the new RAG axioms in these publications:

• Alexander Bondarenko, Maik Fröbe, Jan Heinrich Reimer, Benno Stein, Michael Völske, and Matthias Hagen. Axiomatic Retrieval Experimentation with ir_axioms.
• Jan Heinrich Merker, Maik Fröbe, Benno Stein, Martin Potthast, and Matthias Hagen. Axioms for Retrieval-Augmented Generation.

Installation

1. Install the package from PyPI:

   pip install ir_axioms>=1.0

2. Install spaCy model:

   python -m spacy download en_core_web_sm

Optional Dependencies

Tag	Description
pyserini	Use index statistics, document contents, and tokenization from Pyserini/Anserini indices.
pyterrier	Use index statistics, document contents, and tokenization from PyTerrier/Terrier indices. Apply KwikSort re-ranking to PyTerrier pipelines. Run axiomatic analyses.
keybert	Use KeyBERT for aspect extraction (must be enabled manually).
sbert	Use Sentence Transformers for sentence similarity (enabled automatically).

Note: To reproduce our results from ICTIR'25, please use these extras: pyterrier,keybert,sbert

Usage

Run the CLI with:

ir_axioms --help

Development

1. Install Python 3.11 or later.

2. Create and activate a virtual environment:

   python3 -m venv venv/
   source venv/bin/activate

3. Install project dependencies:

   pip install -e .[tests]

4. Install spaCy model:

   python -m spacy download en_core_web_sm

5. After having implemented a new feature, please check the code format, inspect common LINT errors, and run all unit tests with the following commands:

   ruff check .                   # Code format and LINT
   mypy .                         # Static typing
   bandit -c pyproject.toml -r .  # Security
   pytest .                       # Unit tests

Contribute

If you have found an important feature missing from our tool, please suggest it by creating an issue. We also gratefully accept pull requests!

If you are unsure about anything, post an issue or contact us:

• [email protected]

We are happy to help!

License

This repository is released under the MIT license. Files in the data/ directory are exempt from this license.