qa: Facilitate debugging bitcoind inside functional tests #31723

hodlinator · 2025-01-23T13:18:06Z

Facilitates debugging specific instances of bitcoind in the context of Python tests, inside of an editor/IDE/debugger.

Makes bitcoind spin during startup, waiting for a debugger to be attached, so that breakpoints during process init can be debugged.

Example usage

Generate build files with -DENABLE_WAIT_FOR_DEBUGGER=ON & build.

cmake -B build -DCMAKE_BUILD_TYPE=Debug -DENABLE_WAIT_FOR_DEBUGGER=ON && cmake --build build -j$(nproc)

Set breakpoints in the C++ code in your editor/IDE/debugger.

Start a Python functional test with a debugger command such as:

LLDB inside an already started Tmux pane (append -o continue to the LLDB args to have the debugger continue automatically):
```
./build/test/functional/feature_loadblock.py --debug_runs 2 --debug_cmd "tmux split-window -h lldb -p \$PID\$"
```

Sublime Text Debugger, allows a running session to attach:

./build/test/functional/feature_loadblock.py --debug_runs 2 --debug_cmd "subl --command \"debugger {\\\"action\\\": \\\"start\\\", \\\"configuration\\\": {\\\"name\\\": \\\"foo\\\", \\\"type\\\": \\\"lldb\\\", \\\"request\\\": \\\"attach\\\", \\\"program\\\": \\\"dummyprogram\\\", \\\"pid\\\": \\\"\$PID\$\\\"}}\""

Don't specify --debug_cmd ...\$PID\$..., instead have the test framework log which bitcoind PID you should manually attach to:
```
./build/test/functional/feature_loadblock.py --debug_runs 2
```

Experience how debugger hits breakpoints as the functional test executes RPCs.

Using -debug_runs <RUN_INDEX> often requires doing a dry-run of a functional test with -ldebug and examining the log to see which run index is the one we want to debug:

2025-...Z TestFramework.node0 (DEBUG): bitcoind started (run #1, node #0), waiting for RPC to come up

Alternative usage

Tweak functional test code to specify wait_for_debugger when (re)starting a node:

-        self.restart_node(0)
+        self.restart_node(0, wait_for_debugger=True)

While it can be messy to modify files while moving between commits, and cumbersome inside of loops, it might be more straightforward than -debug_runs <RUN_INDEX> in a given case.

Needed before merge

Testing on Mac to confirm -waitfordebugger on the C++ side behaves as expected.

DrahtBot · 2025-01-23T13:18:09Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/31723.

Reviews

See the guideline for information on the review process.

Type	Reviewers
Concept ACK	l0rinc

If your review is incorrectly listed, please copy-paste  into the comment that the bot should ignore.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#33974 (cmake: Check dependencies after build option interaction by hebasto)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

src/bitcoind.cpp

fanquake · 2025-01-23T13:47:36Z

Makes bitcoind spin during startup, waiting for a debugger to be attached.

Can you explain more? If you want to run under a debugger, then why not just start bitcoind under a debugger; why do we need to add this option?

hodlinator · 2025-01-23T13:55:34Z

Can you explain more? If you want to run under a debugger, then why not just start bitcoind under a debugger; why do we need to add this option?

Clarified PR description text and added example usage. But I'm curious to hear other ways of doing the same thing if there are any.

maflcko · 2025-01-23T14:00:28Z

I think if you want to debug the start-up sequence specifically, it may be easier to do it outside the tests? Then anything else, if needed, can happen without patching Bitcoin Core source code itself.

hodlinator · 2025-01-23T14:19:09Z

I think if you want to debug the start-up sequence specifically, it may be easier to do it outside the tests? Then anything else, if needed, can happen without patching Bitcoin Core source code itself.

This is not about debugging the start-up sequence specifically, but rather debugging from an early stage. Maybe one has set 5 breakpoints, some of which are hit during bitcoind init, and some which are hit by re-running a Python test which calls a set of RPCs.

fanquake · 2025-01-23T14:24:07Z

Ideally we wouldn't be adding debug code, to production binaries, to facilitate using an IDE.

Can you configure your IDE to do something like the equivalent of using gdb/lldb, and process attach --name bitcoind --waitfor? So it just grabs bitcoind once it spins up?

hodlinator · 2025-01-23T14:37:44Z

Ideally we wouldn't be adding debug code, to production binaries, to facilitate using an IDE.

Agree it should not be in production binaries, should be #ifdefed away. Added as another reason for draft status.

Can you configure your IDE to do something like the equivalent of using gdb/lldb, and process attach --name bitcoind --waitfor? So it just grabs bitcoind once it spins up?

I've never seen something like that in an IDE/editor, but I'll have a look, might exist a debugger console which allows that kind of thing.

This PR does however allow for attaching to a specific node instance of bitcoind if a Python tests runs several nodes at once. (I'm considering changing the index to refer to how many bitcoind-launches one has done, some tests just relaunch node 0 a bunch of times, and one might be interested in debugging it on the 3rd execution Edit: this is now implemented).

yuvicc · 2025-01-27T13:46:05Z

Agree with @maflcko @fanquake, not sure if we want to add that to the production code.

hodlinator · 2025-01-27T19:15:45Z

Added tentative #ifdef solution in latest push. CMake part could probably be improved since now I think the #define leaks into the CMake project's public interface, which should not be necessary.

hodlinator · 2025-02-13T08:05:32Z

Added CMake option WAIT_FOR_DEBUGGER.
Changed the Python logic from debugging specific nodes, to debugging specific node runs/executions, which is more specific.
Use prctl() on Linux to allow any process to attach to bitcoind. This removes the necessity to disable kernel.yama.ptrace_scope on some systems, which may be seen as decreasing security, but is compiled out unless WAIT_FOR_DEBUGGER=ON.
(Confirmed PR works on Windows).
(Rebased).

maflcko · 2025-02-13T11:45:28Z

I think if you want to debug the start-up sequence specifically, it may be easier to do it outside the tests? Then anything else, if needed, can happen without patching Bitcoin Core source code itself.

This is not about debugging the start-up sequence specifically, but rather debugging from an early stage. Maybe one has set 5 breakpoints, some of which are hit during bitcoind init, and some which are hit by re-running a Python test which calls a set of RPCs.

I'd still say it is easier to debug the init sequence separately than to start a new build with a new build flag for the edge case where someone wants to debug both in an IDE and also the init sequence , but no strong opinion.

hodlinator · 2025-02-13T15:49:27Z

fanquake:

Can you configure your IDE to do something like the equivalent of using gdb/lldb, and process attach --name bitcoind --waitfor? So it just grabs bitcoind once it spins up?

I've never seen something like that in an IDE/editor, but I'll have a look, might exist a debugger console which allows that kind of thing.

The only debugging console I get with the IDE states "No Active Debug Session" when I try to input commands. :(

maflcko:

This is not about debugging the start-up sequence specifically, but rather debugging from an early stage. Maybe one has set 5 breakpoints, some of which are hit during bitcoind init, and some which are hit by re-running a Python test which calls a set of RPCs.

I'd still say it is easier to debug the init sequence separately than to start a new build with a new build flag for the edge case where someone wants to debug both in an IDE and also the init sequence , but no strong opinion.

Maybe part of this is down to me not getting used to using a Python debugger, breaking in Python at appropriate times and then attaching a C++ debugger to the new process. Being able to specify which bitcoind-executions one wants to debug still seems like a smoother workflow when testing the same scenario repeatedly.

I recently used a version of this PR when digging into #31734. Being able to step through the code (with a panel for inspecting local variables and watches), as the Python test framework is issuing RPCs, in the safety of one's IDE, is nice. Knowing that bitcoind has not passed through one's breakpoints (unless there's some rare static initialization going on) helps build certainty about what unfamiliar code is involved in.

Granted, using --waitfor straight in gdb/lldb would give definite certainty what code runs. But the IDE GUI would not be there. I know gdb has some kind of TUI, and there are probably commands for saving and restoring state. It just feels like operating the computer with my feet. Happy to be proven wrong though. Please shill your setups. :)

hodlinator

Latest push adds a functional test and breaks apart commits somewhat.

src/bitcoind.cpp

hodlinator · 2025-09-30T19:41:15Z

CMakeLists.txt

@@ -120,6 +120,7 @@ cmake_dependent_option(BUILD_WALLET_TOOL "Build bitcoin-wallet tool." ${BUILD_TE
 option(REDUCE_EXPORTS "Attempt to reduce exported symbols in the resulting executables." OFF)
 option(WERROR "Treat compiler warnings as errors." OFF)
 option(WITH_CCACHE "Attempt to use ccache for compiling." ON)
+option(WAIT_FOR_DEBUGGER "Support for waiting during startup for debugger to be attached." OFF)


To me there's a silent, implied, "Enables ..." or "Provides ..." at the beginning of the description for these options that would get repetitive if spelled out, which may make "support" more natural.

Note that simply enabling the option does not pause startup, one also has to start with -waitfordebugger.

Curious to hear what other reviewers think.

src/bitcoind.cpp

src/init.cpp

hodlinator · 2025-10-24T06:12:39Z

Latest push adds 9d1de2e8b3c73edc44bb3f7578ec96c2b91c2dd8 to support modifying a functional test to make a specific node (re)start trigger attaching of a debugger. This may be a more natural alternative when humans read through functional test code and want to select a specific node instance to debug.

(It also removes a header which caused a build failure on Alpine/musl, and adds helpful details in 2 commit messages).

There has been some feedback to see if we could avoid changing the functional test framework and move most of this feature to /contrib/, so I might try to figure out alternative approaches.

DrahtBot · 2025-12-15T11:03:17Z

🚧 At least one of the CI tasks failed.
_{Task lint: https://github.com/bitcoin/bitcoin/actions/runs/20229750611/job/58069642887}
_{LLM reason (✨ experimental): CI failure due to Python lint errors reported by ruff (unused import and an f-string issue) in the lint step.}

Hints

Try to run the tests locally, according to the documentation. However, a CI failure may still
happen due to a number of reasons, for example:

Possibly due to a silent merge conflict (the changes in this pull request being
incompatible with the current code in the target branch). If so, make sure to rebase on the latest
commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the
affected test.
An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

hodlinator · 2025-12-15T11:05:44Z

Changes compared to f352bd9d246b20579b9b4ccec366bb694df5e051:

feature_debugging.py - Added stderr output upon failure when we expect to be able to attach. Switched to subprocess.run(..., text=True) in spirit of fab085c.
Renamed CMake option WAIT_FOR_DEBUGGER -> ENABLE_WAIT_FOR_DEBUGGER to align convention with other options (also renamed constexpr bool in bitcoind.cpp from WAIT_FOR_DEBUGGER_ENABLE -> WAIT_FOR_DEBUGGER_SUPPORT to disambiguate).
Added config.ini support for ENABLE_WAIT_FOR_DEBUGGER and implemented is_wait_for_debugger_compiled() instead of custom parsing of CMakeCache.txt in feature_debugging.py.
Added comment in test_node.py explaining that functional test will continue executing even though a debugger has not been attached yet.
Switched from mentioning Gnome Console in commit message and developer-notes.md to using Tmux as an example.
Switched example test from feature_abortnode.py to feature_loadblock.py since the former causes a node to crash.
Made --debug_cmd without --debug_runs not enable it for all runs, but instead require modifying test code to specify wait_for_debugger=True or use --debug_runs.

Verifies that debugging bitcoind is not allowed by default on Linux, at least in sane environments.

Makes bitcoind spin during startup, waiting for a debugger to be attached. Once a debugger is detected, execution will continue.

Useful for debugging one or several bitcoind nodes running in the context of a functional test. You can run a test with -ldebug once first to get output with run#s to see which you are interested in.

Enables automated attaching of debuggers. The special identifier "$PID$" is substituted by the actual bitcoind process id number. Attaching with LLDB in Tmux pane: $ ./build/test/functional/feature_loadblock.py --debug_runs 2 --debug_cmd "tmux split-window -h lldb -p \$PID\$" Attaching with Sublime Text Debugger: $ ./build/test/functional/feature_loadblock.py --debug_runs 2 --debug_cmd "subl --command \"debugger {\\\"action\\\": \\\"start\\\", \\\"configuration\\\": {\\\"name\\\": \\\"foo\\\", \\\"type\\\": \\\"lldb\\\", \\\"request\\\": \\\"attach\\\", \\\"program\\\": \\\"dummyprogram\\\", \\\"pid\\\": \\\"\$PID\$\\\"}}\""

Enables tweaking functional tests to start debugging a specific node run/execution.

hodlinator marked this pull request as draft January 23, 2025 13:18

hodlinator mentioned this pull request Jan 23, 2025

test: Remove --noshutdown flag, Tidy startup failures #31620

Merged

maflcko reviewed Jan 23, 2025

View reviewed changes

src/bitcoind.cpp Outdated Show resolved Hide resolved

This was referenced Jan 23, 2025

wallet: Disable creating and loading legacy wallets #31250

Merged

qa: Verify clean shutdown on startup failure #30660

Merged

Remove the legacy wallet and BDB dependency #28710

Merged

multiprocess: Add bitcoin wrapper executable #31375

Merged

hodlinator force-pushed the 2024/06/wait_for_debugger branch from 7d181da to 7dd8609 Compare January 27, 2025 19:11

DrahtBot mentioned this pull request Jan 28, 2025

build: set build type and per-build-type flags as early as possible #31711

Merged

DrahtBot added the Needs rebase label Feb 12, 2025

hodlinator force-pushed the 2024/06/wait_for_debugger branch from 7dd8609 to a1da3bf Compare February 12, 2025 21:58

DrahtBot removed the Needs rebase label Feb 12, 2025

hodlinator changed the title ~~qa debug: Add --debugnode/-waitfordebugger [DRAFT]~~ qa debug: Add --debug_runs/-waitfordebugger [DRAFT] Feb 13, 2025

This was referenced Feb 14, 2025

test, refactor: Add TestNode.binaries to hold binary paths #31866

Merged

scripted-diff: Type-safe settings retrieval #31260

Open

DrahtBot added the Needs rebase label Mar 20, 2025

hodlinator force-pushed the 2024/06/wait_for_debugger branch from a1da3bf to 191dfc4 Compare March 20, 2025 14:51

DrahtBot removed the Needs rebase label Mar 20, 2025

hodlinator force-pushed the 2024/06/wait_for_debugger branch from 8c99eab to 56439ef Compare October 1, 2025 07:52

hodlinator commented Oct 1, 2025

View reviewed changes

hodlinator force-pushed the 2024/06/wait_for_debugger branch 3 times, most recently from e785b47 to d1a50b1 Compare October 1, 2025 12:28

DrahtBot mentioned this pull request Oct 16, 2025

test: [move-only] binary utils to utils.py #33633

Merged

DrahtBot added the Needs rebase label Oct 21, 2025

hodlinator force-pushed the 2024/06/wait_for_debugger branch from d1a50b1 to 57df47a Compare October 22, 2025 20:42

DrahtBot removed the Needs rebase label Oct 22, 2025

hodlinator force-pushed the 2024/06/wait_for_debugger branch from 57df47a to 9d1de2e Compare October 22, 2025 21:54

DrahtBot added the Needs rebase label Oct 29, 2025

hodlinator force-pushed the 2024/06/wait_for_debugger branch from 9d1de2e to f352bd9 Compare October 29, 2025 18:58

DrahtBot removed the Needs rebase label Oct 29, 2025

DrahtBot mentioned this pull request Nov 12, 2025

kernel, logging: Pass Logger instances to kernel objects #30342

Draft

DrahtBot mentioned this pull request Dec 2, 2025

cmake: Check dependencies after build option interaction #33974

Open

hodlinator force-pushed the 2024/06/wait_for_debugger branch 2 times, most recently from fb9d0e6 to f021644 Compare December 15, 2025 11:02

DrahtBot added the CI failed label Dec 15, 2025

DrahtBot removed the CI failed label Dec 15, 2025

hodlinator added 5 commits December 15, 2025 18:58

qa: Add feature_debugging.py

f338af4

Verifies that debugging bitcoind is not allowed by default on Linux, at least in sane environments.

node: Add -waitfordebugger

aafdbd7

Makes bitcoind spin during startup, waiting for a debugger to be attached. Once a debugger is detected, execution will continue.

qa: Add --debug_runs

f8572e2

Useful for debugging one or several bitcoind nodes running in the context of a functional test. You can run a test with -ldebug once first to get output with run#s to see which you are interested in.

qa: Add wait_for_debugger param for node (re)starts

42db717

Enables tweaking functional tests to start debugging a specific node run/execution.

hodlinator changed the title ~~node: Add --debug_runs/-waitfordebugger + --debug_cmd~~ qa: Facilitate debugging bitcoind inside functional tests Dec 18, 2025

DrahtBot added the Tests label Dec 18, 2025

hodlinator force-pushed the 2024/06/wait_for_debugger branch from f021644 to 42db717 Compare December 18, 2025 15:18

qa: Facilitate debugging bitcoind inside functional tests #31723

Are you sure you want to change the base?

qa: Facilitate debugging bitcoind inside functional tests #31723

Conversation

hodlinator commented Jan 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Example usage

Alternative usage

Needed before merge

Uh oh!

DrahtBot commented Jan 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Coverage & Benchmarks

Reviews

Conflicts

Uh oh!

Uh oh!

fanquake commented Jan 23, 2025

Uh oh!

hodlinator commented Jan 23, 2025

Uh oh!

maflcko commented Jan 23, 2025

Uh oh!

hodlinator commented Jan 23, 2025

Uh oh!

fanquake commented Jan 23, 2025

Uh oh!

hodlinator commented Jan 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yuvicc commented Jan 27, 2025

Uh oh!

hodlinator commented Jan 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hodlinator commented Feb 13, 2025

Uh oh!

maflcko commented Feb 13, 2025

Uh oh!

hodlinator commented Feb 13, 2025

Uh oh!

hodlinator left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hodlinator Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hodlinator commented Oct 24, 2025

Uh oh!

DrahtBot commented Dec 15, 2025

Uh oh!

hodlinator commented Dec 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

hodlinator commented Jan 23, 2025 •

edited

Loading

DrahtBot commented Jan 23, 2025 •

edited

Loading

hodlinator commented Jan 23, 2025 •

edited

Loading

hodlinator commented Jan 27, 2025 •

edited

Loading