BPF Conformance

This project measures the conformance of a BPF runtime to the ISA. To measure conformance the BPF runtime under test is built into a plugin process that does:

1. Accept both BPF byte code an initial memory.
2. Execute the byte code.
3. Return the value in %r0 at the end of the execution.

eBPF runtime implementations that are currently measured using this project

1. Linux Kernel via libbpf
2. uBPF
3. eBPF for Windows / bpf2c
4. rbpf
5. Prevail Verifier
6. bpftime / llvmbpf, The result is here

Note: Linux Kernel is treated as the authorative eBPF implementation.

Prerequisites

Ubuntu

Install build dependencies:

sudo apt-get install -y \
    build-essential \
    cmake \
    libboost-dev \
    libboost-filesystem-dev \
    libboost-program-options-dev \
    libelf-dev \
    lcov

For the libbpf_plugin (required to run tests against the Linux kernel), you also need libbpf. The system package (libbpf-dev) may be outdated; building from source is recommended. This mirrors the CI workflow in .github/scripts/build-libbpf.sh:

# Clone libbpf v1.3.0 (pinned to commit for reproducibility)
git clone https://github.com/libbpf/libbpf.git
cd libbpf
git checkout 20c0a9e3d7e7d4aeb283eae982543c9cacc29477  # v1.3.0
cd src
make
sudo PREFIX=/usr LIBDIR=/usr/lib/x86_64-linux-gnu/ make install

macOS

brew install cmake ninja ccache boost

Note: The libbpf_plugin is Linux-only. On macOS, you can build the conformance runner and use other plugins.

Windows

On Windows, Boost is automatically fetched via NuGet during CMake configuration. No manual dependency installation is required beyond having Visual Studio with C++ support.

Building

Run cmake -S . -B build to configure the project, then run cmake --build build to build the project.

Usage

Options:
  --help                         Print help messages
  --test_file_path arg           Path to test file
  --test_file_directory arg      Path to test file directory
  --plugin_path arg              Path to plugin
  --plugin_options arg           Options to pass to plugin
  --list_instructions arg        List instructions used and not used in tests
  --list_used_instructions arg   List instructions used in tests
  --list_unused_instructions arg List instructions not used in tests
  --debug (true|false)           Print debug information
  --xdp_prolog (true|false)      XDP prolog
  --elf (true|false)             ELF format
  --cpu_version arg              CPU version
  --include_regex arg            Include regex
  --exclude_regex arg            Exclude regex

--xdp_prolog true should only be used with the libbpf_plugin.

Using a published package

Select the desired version from bpf_conformance

Assume the package is named: "ghcr.io/alan-jowett/bpf_conformance:main"

docker run --privileged -it --rm ghcr.io/alan-jowett/bpf_conformance:main src/bpf_conformance_runner --test_file_directory tests --plugin_path libbpf_plugin/libbpf_plugin --cpu_version v3

Running the test

Linux (test require Linux kernel BPF support):

cmake --build build --target test --

Note: The libbpf_plugin requires root or BPF permissions.

Using bpf_conformance as a static lib

The BPF Conformance tests can also be used as a static library as part of another tests.

1. Include include/bpf_conformance.h
2. Link against libbpf_conformance.a and boost_filesystem (depending on platform).
3. Invoke bpf_conformance, passing it a list of test files.

Interpreting results

On completion of the test the bpf_conformance tools prints the list of tests that passes/failed and a summary count.

sudo build/src/bpf_conformance --test_file_directory tests --plugin_path build/libbpf_plugin/libbpf_plugin
Test results:
PASS: "tests/add.data"
PASS: "tests/add64.data"
PASS: "tests/alu-arith.data"
PASS: "tests/alu-bit.data"
PASS: "tests/alu64-arith.data"
PASS: "tests/alu64-bit.data"
PASS: "tests/arsh-reg.data"
PASS: "tests/arsh.data"
PASS: "tests/arsh32-high-shift.data"
PASS: "tests/arsh64.data"
PASS: "tests/be16-high.data"
PASS: "tests/be16.data"
PASS: "tests/be32-high.data"
PASS: "tests/be32.data"
PASS: "tests/be64.data"
PASS: "tests/call_unwind_fail.data"
PASS: "tests/div-by-zero-reg.data"
PASS: "tests/div32-high-divisor.data"
PASS: "tests/div32-imm.data"
PASS: "tests/div32-reg.data"
PASS: "tests/div64-by-zero-reg.data"
PASS: "tests/div64-imm.data"
PASS: "tests/div64-reg.data"
PASS: "tests/exit-not-last.data"
PASS: "tests/exit.data"
PASS: "tests/jeq-imm.data"
PASS: "tests/jeq-reg.data"
PASS: "tests/jge-imm.data"
PASS: "tests/jgt-imm.data"
PASS: "tests/jgt-reg.data"
PASS: "tests/jit-bounce.data"
PASS: "tests/jle-imm.data"
PASS: "tests/jle-reg.data"
PASS: "tests/jlt-imm.data"
PASS: "tests/jlt-reg.data"
PASS: "tests/jne-reg.data"
PASS: "tests/jset-imm.data"
PASS: "tests/jset-reg.data"
PASS: "tests/jsge-imm.data"
PASS: "tests/jsge-reg.data"
PASS: "tests/jsgt-imm.data"
PASS: "tests/jsgt-reg.data"
PASS: "tests/jsle-imm.data"
PASS: "tests/jsle-reg.data"
PASS: "tests/jslt-imm.data"
PASS: "tests/jslt-reg.data"
PASS: "tests/lddw.data"
PASS: "tests/lddw2.data"
PASS: "tests/ldxb-all.data"
PASS: "tests/ldxb.data"
PASS: "tests/ldxdw.data"
PASS: "tests/ldxh-all.data"
PASS: "tests/ldxh-all2.data"
PASS: "tests/ldxh-same-reg.data"
PASS: "tests/ldxh.data"
PASS: "tests/ldxw-all.data"
PASS: "tests/ldxw.data"
PASS: "tests/le16.data"
PASS: "tests/le32.data"
PASS: "tests/le64.data"
PASS: "tests/lsh-reg.data"
PASS: "tests/mem-len.data"
PASS: "tests/mod-by-zero-reg.data"
PASS: "tests/mod.data"
PASS: "tests/mod32.data"
PASS: "tests/mod64-by-zero-reg.data"
PASS: "tests/mod64.data"
PASS: "tests/mov.data"
PASS: "tests/mul32-imm.data"
PASS: "tests/mul32-reg-overflow.data"
PASS: "tests/mul32-reg.data"
PASS: "tests/mul64-imm.data"
PASS: "tests/mul64-reg.data"
PASS: "tests/neg.data"
PASS: "tests/neg64.data"
PASS: "tests/prime.data"
PASS: "tests/rsh-reg.data"
PASS: "tests/rsh32.data"
PASS: "tests/stack.data"
PASS: "tests/stb.data"
PASS: "tests/stdw.data"
PASS: "tests/sth.data"
PASS: "tests/stw.data"
PASS: "tests/stxb-all.data"
PASS: "tests/stxb-all2.data"
PASS: "tests/stxb-chain.data"
PASS: "tests/stxb.data"
PASS: "tests/stxdw.data"
PASS: "tests/stxh.data"
PASS: "tests/stxw.data"
PASS: "tests/subnet.data"
Passed 91 out of 91 tests.

bpf_conformance_runner plugins

bpf_conformance_runner invokes plugins using command-line execution, and uses stdin, stdout, and stderr to exchange additional information as follows:

<plugin name> [<base16 memory bytes>] [<plugin options>] [--elf]

where:

• <plugin name>: the plugin executable name
• <base16 memory bytes>: if present, the contents of the mem section of a test data file
• <plugin options>: any additional options to pass to the plugin
• --elf: if present, indicates that the data passed to stdin will be in ELF format

The program is then passed to the plugin via stdin, either as raw bytecode or (if --elf is specified) in ELF format. The plugin must then either:

• compute a successful result and output the final contents of r0 in hex (either with or without a leading "0x") to stdout and exit with a status of 0, OR
• output an error message to stderr and exit with a non-zero status.

Additional plugins can be created based on the above specification.