C++'s __FILE__ macro expands to the file's path. Whether that's a relative path or an absolute one depends on what's given to the compiler. For example,
will expand test.cpp's __FILE__ into ../../test.cpp, whereas
will expand it into /whatever/absolute/path/test.cpp.
In system_lib.py, we use both of them depending on the paths.
In CircleCI, tests that depend on build-libs, which is a part of build-linux, which uses Ninja, will be built with absolute paths when code includes __FILE__. This includes all core, other, and browser tests. Other tests (test-mac-arm64, test-windows, ...) don't use build-libs so they use build_objects and thus will include relative paths.
So, this will produce different builds for different CircleCI tests. deterministic_paths parameter, which uses -ffile-prefix-map in system_lib.py makes all relative paths the same string and all absolute paths the same string, but does not make an absolute path and a relative path the same. So even if we make CircleCI always set deterministic_paths, the problem remains:
|
if self.deterministic_paths: |
|
source_dir = utils.path_from_root() |
|
if batch_inputs: |
|
relative_source_dir = os.path.relpath(source_dir, build_dir) |
|
cflags += [f'-ffile-prefix-map={relative_source_dir}/='] |
|
cflags += [f'-ffile-prefix-map={source_dir}=/emsdk/emscripten', |
|
'-fdebug-compilation-dir=/emsdk/emscripten'] |
This problem was brought into attention because the new LLVM 19's libc++abi adds more usage of __FILE__ and one of them ended up in other.test_minimal_runtime_code_size_hello_embind, causing the code sizes to be different between different CircleCI tests. We can make this usage an empty string in release mode (which Zig did) but this does not fundamentally fix the problem that __FILE__ can end up in other code size tests. Currently we seem to use __FILE__ in several places in libraries:
aheejin@aheejin:~/emscripten/system/lib$ grep __FILE__ * -R
compiler-rt/lib/sanitizer_common/sanitizer_internal_defs.h: __sanitizer::CheckFailed(__FILE__, __LINE__, \
compiler-rt/lib/builtins/int_util.h:#define compilerrt_abort() __compilerrt_abort_impl(__FILE__, __LINE__, __func__)
libc/musl/include/assert.h:#define assert(x) ((void)((x) || (__assert_fail(#x, __FILE__, __LINE__, __func__),0)))
libcxx/include/__assert: : _LIBCPP_ASSERTION_HANDLER(__FILE__ ":" _LIBCPP_TOSTRING(__LINE__) ": assertion " _LIBCPP_TOSTRING( \
libcxx/src/verbose_abort.cpp: __assert2(__FILE__, __LINE__, __func__, buffer);
libcxxabi/src/abort_message.h: ::abort_message("%s:%d: %s", __FILE__, __LINE__, __msg); \
libcxxabi/src/abort_message.cpp: __assert2(__FILE__, __LINE__, __func__, buffer);
mimalloc/include/mimalloc/types.h:#define mi_assert(expr) ((expr) ? (void)0 : _mi_assert_fail(#expr,__FILE__,__LINE__,__func__))
wasmfs/support.h: wasmfs::handle_unreachable(msg, __FILE__, __LINE__)
Not sure what is the best way to proceed:
- Make
__FILE__ an empty string in release mode via adding -D__FILE__="" in system_lib.py.
- Make
__FILE__ an empty string when the environment variable CIRCLECI is set, which is set in all CIrcleCI tests. But then we have to make sure to clear cache and set CIRCLECI when rebaselining code size tests on our local machine, which is a pain, like
$ ./emcc --clear-cache
$ CIRCLECI=1 ./tools/maint/rebaseline_tests.py
- ???
I personally prefer 1, which can be as simple as #23196.
C++'s
__FILE__macro expands to the file's path. Whether that's a relative path or an absolute one depends on what's given to the compiler. For example,$ clang++ ../../test.cppwill expand
test.cpp's__FILE__into../../test.cpp, whereas$ clang++ ~/test.cppwill expand it into
/whatever/absolute/path/test.cpp.In
system_lib.py, we use both of them depending on the paths.build_objects) andbatch_inputsis true, which is the default:emscripten/tools/system_libs.py
Lines 536 to 539 in f615920
build_objects) andbatch_inputsis false (Add switch to disable batching of inputs when building system libs #20929)create_ninja_file). I don't think we can use relative paths in Ninja files.In CircleCI, tests that depend on
build-libs, which is a part ofbuild-linux, which uses Ninja, will be built with absolute paths when code includes__FILE__. This includes all core, other, and browser tests. Other tests (test-mac-arm64,test-windows, ...) don't usebuild-libsso they usebuild_objectsand thus will include relative paths.So, this will produce different builds for different CircleCI tests.
deterministic_pathsparameter, which uses-ffile-prefix-mapinsystem_lib.pymakes all relative paths the same string and all absolute paths the same string, but does not make an absolute path and a relative path the same. So even if we make CircleCI always setdeterministic_paths, the problem remains:emscripten/tools/system_libs.py
Lines 495 to 501 in f615920
This problem was brought into attention because the new LLVM 19's libc++abi adds more usage of
__FILE__and one of them ended up inother.test_minimal_runtime_code_size_hello_embind, causing the code sizes to be different between different CircleCI tests. We can make this usage an empty string in release mode (which Zig did) but this does not fundamentally fix the problem that__FILE__can end up in other code size tests. Currently we seem to use__FILE__in several places in libraries:Not sure what is the best way to proceed:
__FILE__an empty string in release mode via adding-D__FILE__=""insystem_lib.py.__FILE__an empty string when the environment variableCIRCLECIis set, which is set in all CIrcleCI tests. But then we have to make sure to clear cache and setCIRCLECIwhen rebaselining code size tests on our local machine, which is a pain, likeI personally prefer 1, which can be as simple as #23196.