BPF in GCC

The BPF virtual machine is being used ever more widely in the kernel, but it has not been a target for GCC until recently. BPF is currently generated using the LLVM compiler suite. Jose E. Marchesi gave a pair of presentations as part of the GNU Tools track at the 2020 Linux Plumbers Conference (LPC) that provided attendees with a look at the BPF for GCC project, which started around a year ago. It has made some significant progress, but there is, of course, more to do.

There are three phases envisioned for the project. The first is to add the BPF target to the GNU toolchain. Next up is to ensure that the generated programs pass the kernel's verifier, so that they can be loaded into the kernel. That will also require effort to keep it working, Marchesi said, because the BPF world moves extremely fast. The last phase is to provide additional tools for BPF developers, beyond just a compiler and assembler, such as debuggers and simulators.

Binutils support for BPF has been upstream since August 2019, while the GCC BPF backend was added in September 2019. In August 2020, just a few weeks before he gave the talk, support for BPF in GDB was added along with a simulator for it. The simulator is intended to be used with a board file for the DejaGnu test framework in order to run the GCC test suite for the BPF backend.

The binutils support is complete at this point, he said, as is the GCC backend. The GDB support is basic; it only handles loading a BPF program, single stepping, setting breakpoints, examining BPF registers, and listing the assembler code. Similarly, the simulator support is basic; it is integrated with GDB and most of the BPF instructions are supported so programs can run on it. There are plans to add various kinds of kernel contexts to the simulator so that different BPF program types can be run; for example, a program to be attached to a kprobe would be provided with a kprobe context in the simulator. The simulator also needs support for more kernel helper functions beyond just printk().

In addition to support for standard BPF, the project has been adding support for "experimental BPF" (or xBPF, though he said "the name is not important"). It adds features that get around the limitations imposed by the kernel verifier so that the full GCC test suite can be run. There are thousands of GCC tests that will not even build for BPF because they need features the language does not support, such as indirect calls and functions that need to do their own save and restore of the register values. Beyond just the compiler tests, xBPF extensions are needed for debugging BPF in GDB using the DWARF format; that will allow getting backtraces from BPF programs, for example.

With those pieces in place, Marchesi said, the project has turned to support for BPF type format (BTF), which is the debugging format used by BPF. It is similar to Compact C type format (CTF), which has been added to the GNU toolchain; CTF and BTF share a common ancestor. When the -g option to GCC is used to request debugging information, it should generate BTF and not DWARF for BPF programs. BTF is integral to the "compile once, run everywhere" (CO-RE) plan for being able to run BPF programs on multiple kernel versions.

The basic problem is that when BPF is compiled, it uses a set of kernel headers that describe various kernel data structures for that particular version, which may be different from those on the kernel where the program is run. Until relatively recently, that was solved by distributing the BPF as C code along with the Clang compiler to build the BPF on the system where it was going to be run. Now, the compiler generates BPF along with BTF information about the data structures and struct members being used so that the BPF loader can fix up those references when it loads the program on a different kernel version. "It is amazing that it works", but it seems to work well, he said.

By looking at the LLVM BPF backend, Marchesi learned that there is a class for handling debugging information as part of the intermediate representation (IR). Adding a new debugging format is a matter of extending that class to support it; currently LLVM supports DWARF, CodeView, and BTF. The situation for GCC is completely different. There is something called debug_hooks, but it is called from many different places within the compiler: from the frontend, backends, link-time optimization (LTO) stage, and so on. There are several different formats that can be generated via those debug_hooks, many of which are legacy (e.g. VMS dbg, DBX); his team recently wanted to add CTF support as well.

Initially, the plan was to simply extend the current scheme, which is "very old", he said, for CTF, but the feedback from the GCC maintainers was that the legacy format support should not hold things back. It was suggested that using the DWARF support was the right path forward for adding new formats, so CTF and BTF will be added via that mechanism. Once it is working, older formats can be ported to the new scheme and, eventually, the older mechanism can be removed. One thing that needs to be considered in all of this is that the new debugging formats (e.g. BTF, CTF) are compact, while DWARF is large and comprehensive; when only generating one of the more compact formats, the overhead of using the DWARF path may be problematic.

There was a fair amount of overlap between the two talks (first talk: raw video [YouTube] and slides [PDF]; second talk: raw video [YouTube] and slides [PDF]), but the second was targeted at engaging the LLVM and BPF developers at LPC. That effort was not particularly successful, as attendees from those projects were apparently busy elsewhere at the conference. But the issues raised will need to be resolved at some point. This thread on the BPF mailing list indicates that there may be some resistance to the xBPF plan, however.

There were three separate items that Marchesi raised in that second talk. The first is with regard to the declaration of BPF helper functions, which are auto-generated into bpf_helpers.h. The existing declarations look like this:

    static __u32 (* bpf_get_prandom_u32)( void ) = ( void *) 7;

    static __u32 (* bpf_get_prandom_u32)( void )
        __attribute__ (( kernel_helper (7)));

The BPF FAQ says that there is no signed division instruction because it would be rarely used. He wondered, though, why it would be a frequently asked question if signed division was so infrequently used. In any case, its lack is a big problem for the GCC test suite, which has lots of tests that use signed division. So support for four instructions (sdiv, smod, and their 32-bit versions) was added to xBPF. He asked if the BPF developers would reconsider signed division. In the chat, Lorenz Bauer said that adding the support would be tricky due to the kernel's support for just-in-time (JIT) compilation of BPF.

Adding those instructions to xBPF brings up another problem area, Marchesi said. Instructions have to be assigned to opcodes, which are a finite resource. If BPF expands in the kernel, it could use opcodes that xBPF has already used. For example, the signed division instructions are currently using the last two available opcodes in each of the ALU and ALU64 instruction classes. He wondered if there could be a range of opcodes that were set aside for extensions. Many instruction classes are running out of opcodes, but there are 23 in LD and 28 in ST that are available, so perhaps some space could be found there. Those questions will be taken to the mailing lists, Marchesi said, to try to resolve them that way.

Having a second toolchain that supports BPF will clearly be a benefit; both GCC and LLVM have gotten better over the years due to their "competition". It would seem that the approach taken by the GCC hackers is different, at least from a testing standpoint, than that taken by LLVM; those areas will need to be worked out before too long. Beyond that, though, the GCC BPF simulator and GDB support will bring new tools to the table for BPF developers.