Skip to content

README.md

Latest commit

 

History

History
297 lines (213 loc) · 9.05 KB

README.md

File metadata and controls

297 lines (213 loc) · 9.05 KB
Raw

maxminddb-rust

A Rust-backed Python module for reading MaxMind DB files.

It mirrors the public API of the official maxminddb package and keeps the same programming model with the maxminddb_rust import name.

Performance

This project is intended to provide performance comparable to the maxminddb C extension while keeping API compatibility.

Performance depends on the database, lookup pattern, and hardware. Run the benchmark scripts in benchmarks/ against your own databases to measure expected throughput in your environment.

The reader is thread-safe and can be shared across threads. On standard CPython builds, CPU-bound lookups from Python threads remain constrained by the GIL. Free-threaded CPython 3.14 wheels allow lookup threads to run concurrently; scaling depends on the database and workload.

Features

API Compatibility

This package provides API compatibility with the official maxminddb Python module.

Supported:

  • Reader class with get(), get_with_prefix_len(), metadata(), and close() methods
  • open_database() function
  • Context manager support (with statement)
  • MODE_* constants (MODE_AUTO, MODE_MMAP, MODE_MMAP_EXT, MODE_FILE, MODE_MEMORY, and MODE_FD)
  • InvalidDatabaseError exception
  • Metadata class with all attributes and computed properties
  • Support for string IP addresses and ipaddress.IPv4Address/IPv6Address objects
  • closed attribute
  • Iterator support (__iter__) for iterating all database records

Extensions (not in the original package):

  • get_many() for batch lookups
  • get_path() for retrieving a specific field from a record (for example, ('country', 'iso_code')) without decoding the entire record
  • get_many_path() for retrieving a specific field for many IP addresses

Installation

From PyPI

pip install maxminddb-rust

From Source

git clone https://github.com/oschwald/maxminddb-rust-python.git
cd maxminddb-rust-python
uv run --with maturin maturin develop --release

Usage

This module follows the same API as maxminddb, with a different import name:

import ipaddress

import maxminddb_rust

# Open the database and release resources automatically when done.
with maxminddb_rust.open_database("/var/lib/GeoIP/GeoIP2-City.mmdb") as reader:
    result = reader.get("8.8.8.8")
    print(result)

    result, prefix_len = reader.get_with_prefix_len("8.8.8.8")
    print(f"Result: {result}, Prefix: {prefix_len}")

    ip = ipaddress.IPv4Address("8.8.8.8")
    result = reader.get(ip)
    print(result)

    metadata = reader.metadata()
    print(f"Database type: {metadata.database_type}")
    print(f"Node count: {metadata.node_count}")

Batch Lookup (Extension)

get_many() is an extension that is not available in the original maxminddb module:

import maxminddb_rust

with maxminddb_rust.open_database("/var/lib/GeoIP/GeoIP2-City.mmdb") as reader:
    ips = ["8.8.8.8", "1.1.1.1", "208.67.222.222"]
    results = reader.get_many(ips)

    for ip, result in zip(ips, results):
        print(f"{ip}: {result}")

Selective Field Lookup (Extension)

get_path() retrieves a specific field from a record:

import maxminddb_rust

with maxminddb_rust.open_database("/var/lib/GeoIP/GeoIP2-City.mmdb") as reader:
    # Retrieve a specific field without decoding the entire record.
    # Path elements can be strings (map keys) or integers (array indices).
    iso_code = reader.get_path("8.8.8.8", ("country", "iso_code"))
    print(f"ISO Code: {iso_code}")

    subdivision = reader.get_path("8.8.8.8", ("subdivisions", 0, "iso_code"))
    print(f"Subdivision: {subdivision}")

    ips = ["8.8.8.8", "1.1.1.1", "208.67.222.222"]
    iso_codes = reader.get_many_path(ips, ("country", "iso_code"))
    print(iso_codes)

Iterator Support

Iterate over all networks in the database:

import maxminddb_rust

with maxminddb_rust.open_database(
    "/var/lib/GeoIP/GeoLite2-Country.mmdb"
) as reader:
    for network, data in reader:
        print(f"{network}: {data['country']['iso_code']}")

Database Modes

Choose between memory-mapped files (default) and read-file modes:

import maxminddb_rust

# MODE_AUTO: currently resolves to MODE_MMAP.
reader = maxminddb_rust.open_database(
    "/var/lib/GeoIP/GeoIP2-City.mmdb", mode=maxminddb_rust.MODE_AUTO
)

# MODE_MMAP: explicitly use memory-mapped files.
reader = maxminddb_rust.open_database(
    "/var/lib/GeoIP/GeoIP2-City.mmdb", mode=maxminddb_rust.MODE_MMAP
)

# MODE_MMAP_EXT: accepted for compatibility; same Rust mmap reader as MODE_MMAP.
reader = maxminddb_rust.open_database(
    "/var/lib/GeoIP/GeoIP2-City.mmdb", mode=maxminddb_rust.MODE_MMAP_EXT
)

# MODE_MEMORY: load the database file into memory.
reader = maxminddb_rust.open_database(
    "/var/lib/GeoIP/GeoIP2-City.mmdb", mode=maxminddb_rust.MODE_MEMORY
)

# MODE_FILE: compatibility mode that also reads the file into memory.
reader = maxminddb_rust.open_database(
    "/var/lib/GeoIP/GeoIP2-City.mmdb", mode=maxminddb_rust.MODE_FILE
)

# MODE_FD: read from a file-like object into memory.
with open("/var/lib/GeoIP/GeoIP2-City.mmdb", "rb") as database:
    reader = maxminddb_rust.open_database(database, mode=maxminddb_rust.MODE_FD)

MODE_FD follows the official package's pure Python behavior: pass a readable binary object and the reader calls read() from its current position. Raw integer OS file descriptors are not accepted directly.

MODE_MMAP_EXT is accepted for official-package compatibility. This package does not have a separate C extension backend, so it uses the same Rust memory-mapped reader as MODE_MMAP.

Examples

The examples/ directory contains complete working examples:

Run any example:

uv run python examples/basic_usage.py
uv run python examples/batch_processing.py

Documentation

  • API documentation: classes and methods include docstrings. Use help():

    import maxminddb_rust
help(maxminddb_rust.open_database)
help(maxminddb_rust.Reader.get)

  • Type hints: full type stub file (maxminddb_rust.pyi) is included for IDE autocomplete and type checking

  • Changelog: see CHANGELOG.md for version history and release notes

  • Migration Guide: see MIGRATION.md for migrating from the official maxminddb package

Benchmarking

Benchmark scripts are consolidated in the benchmarks/ directory. They default to commonly installed databases under /var/lib/GeoIP, preferring GeoLite2 City when available. Pass --file /path/to/database.mmdb to benchmark a different database. The comprehensive benchmark accepts --file more than once.

Run the included benchmarks after building from source:

# Single lookup benchmark
uv run python benchmarks/benchmark.py --count 250000

# Comprehensive benchmark across multiple databases
uv run python benchmarks/benchmark_comprehensive.py --count 250000

# Batch lookup benchmark
uv run python benchmarks/benchmark_batch.py --batch-size 100

# Threaded lookup benchmark (shared Reader across Python threads, default DB set)
uv run python benchmarks/benchmark_parallel.py --count 500000 --workers 1,2,4,8

# get() vs get_path() benchmark
uv run python benchmarks/benchmark_path.py --count 250000

# Compare benchmark throughput between two git refs
uv run python benchmarks/compare_refs.py --baseline-ref origin/main --candidate-ref HEAD

# CI-friendly comparison with JSON output and a 5% regression threshold
uv run python benchmarks/compare_refs.py --json-output bench.json --max-regression-pct 5

# Force successful lookups, including when using a sparse test database
uv run python benchmarks/compare_refs.py --workload database-hits

# Path cache profiling: cached tuple, new tuple per call, list path per call
uv run python benchmarks/compare_refs.py --case get_path --case get_path_new_tuple --case get_path_list

Testing

This project includes Python tests, Rust unit tests, and adapted upstream compatibility tests from MaxMind-DB-Reader-python.

# Initialize test data submodule (first time only)
git submodule update --init --recursive

# Run Python tests.
uv run pytest

# Run Rust unit tests.
cargo test --all-targets --all-features

# Run configured linters and formatters.
uv run precious lint .

For upstream test compatibility and syncing instructions, see tests/maxmind/README.md.

License

ISC License. See LICENSE for details.

Contributing

See CONTRIBUTING.md for development setup, code quality guidelines, and pull request procedures.