Building fmt as a module is currently failing (both GCC and Clang). I used the following:

cmake -DFMT_MODULE=1 -DFMT_TEST=0 ..

I've bisected the issue to the following commit: 680db66. The commit immediately before works fine.

PS: Reverting the commit also works.

PS2: Apparently you're not supposed to build fmt directly as a module, but to use it with add_subdirectory. Regardless, I get similar errors.