the bot run every night I could ack and merge those PRs too

Last night's broke my local CI but passes the Github CI. This has happened multiple times. These cannot be merged without testing.

Fair, I probably should have remembered that. I was like, its rusfmt what could possibly go wrong ...

Last night's broke my local CI but passes the Github CI. This has happened multiple times. These cannot be merged without testing.

Something seems really meh if that's the case. Just saying. Maybe good time to reconsider weight of arguments as the conditions changed.

This comment is point 4 below I believe (and is the reason I posted below).

My local CI box is not being utilized well. I need to refactor a bunch of stuff. It is difficult with family obligations. I should be able to double its throughput by just improving its logic for re-scheduling merge commits after things get merged, detecting when Tobin merges stuff, avoiding PRs "falling between the cracks" by getting into weird acked-but-not-merge-tested states, etc.

FWIW over the last couple weeks I was able to double the throughput already by just telling it to prioritize non-merge commits once there's already one waiting to be signed, and now I am slowly grinding through notifications (I am reducing the count by 10-20 per week lol).

But part of what's helping me get through this is that I get to a 3-4 day old notification and find that it now needs a rebase ... and usually it's because of the API files or something dumb like this. So one thing that would greatly improve this is a rebase bot that would comment and label stuff when it needs a rebase. (And if the conflict is purely in the API files it seems like the bot could also add an extra commit that'd fix the conflicts.)

An easy non-github-integrated way would be for all devs to run through their PRs as job 1 of their day and check what needs rebasing?

Probably end of day too, or anytime a bunch of purple shows up in notifications?

I have big plans for my local CI setup. When we've moved to Forgejo and I have a full API that doesn't take a minute for every call and flake out constantly, my system can probably start posting these comments.

It will be a little while -- I also need to migrate my state machine logic from taskwarrior to postgres and come up with a tighter set of state transitions. But it helps a lot to have LLMs at my disposal now.

My local CI box is not being utilized well. I need to refactor a bunch of stuff. It is difficult with family obligations. I should

To add to this -- in January I rewrote the system to use Taskwarrior as a "database" in place of sqlite, and to test one commit at a time, and to query Github much more frequently and in more detail (e.g. to count ACKs). This gave me a huge boost in flexibility (with TW you can just add arbitrary fields to a running system and it'll just roll with it, no database migration needed) and legibility (I have full logs of every change to every task) but all three of these changes came with a pretty big perf hit.

@tcharding no, because we're trying to make the tooling happy I'd like to run the full matrix on stuff like that (also, in this specific case it's your own PR so you can't merge it without an ACK).

Fair, cheers.

Note that I've gave up on following rust-bitcoin development in detail a while ago, so discount and adjust for individual situation.

I feel you, this project is hard to follow, especially if you're following many other projects, or just a few other high-noise ones. Although, I really enjoy when you comment, since I think the cross-pollination is refreshing.

If we don't want to have formatting at all, okay, I invite you to go read the 100s of 000s of words that we've wasted on this already.

I've read most of them, and responded in most of them (probably much to the annoyance of project maintainers). However, it seems to boil down to a small number of people (like one so far I can think) who can't run fmt on their local machines because of the OS, or at least, that's how this all started as I recollect. I feel like just putting up a CI warning on broken fmt and not a hard no merge policy would suffice and be an acceptable compromise. I dunno, feels like I got crickets the last time I said this, sorry to re-hash it.

Along those lines, now that Kix is gone (again) the red API tape is back. This causes a ton of merge conflicts and false positives. On the other side, I've never found it to be useful, and I mostly just get annoyed when it breaks a PR. I dunno, maybe there are some examples though where it's caught a fatal bug. It could be useful when 1.0 is shipped and then the API should not be as much in flux..

Be more lax on what gets into the master/staging branch. Defer Andrew scanning things to after the critical section. Reverts and follow-up fixes are OK. No one cares if every commit was perfect. Only the tagged release needs to be.

Kix was always beating on the should be ready to release from master at anytime drum. Personally, I feel like this is the correct goal to have. Ideally, the CI should be beefy enough to test every commit. And for other things that just take a very long time, like fuzzing, maybe a nightly cron job. Now that we are in the sort of process of moving off Github, now would be a great time to see if we can just make the CI runners to the things that Andrew needs magic box for?

The ones that are tricky put in the CLAUDE.md (symlink AGENTS.md) and ask contributors to have LLMs scan their changes and flag these

LOL. Some contributors won't run cargo +nightly fmt, asking them to run an LLM would be overboard crazy IIUC. Most of the suggestions sound interesting, however personally I just can't see this one. Maybe if I could run slim-model someday locally would I be more on the LLM bandwagon. How long before you can't even code projects without paying some mega corporation for the privilege to do so, or give them all your datas. After all, isn't this a project where people contribute for the good of the cause, pro-decentralization and user pro-privacy for the people. Both of which are anti-theoretical to the current state of AI.

Anyway, welcome back from your long vacation @dpc and hopefully not to distract too much from your backlog (or Andrew's backlog too for that matter).

Probably multiple of such things compounding.

I guess it could be combinational explosion of (some of):

• Building stuff in release mode (for fuzzing?). Not critical, could be run daily? CI can check that fuzzing targets compile.
• Building combinations of Cargo.lock (min dep versions?)? Not really all that important. Can be fixed once before release.
• Doing whole CI per every commit? Multiplies the time of the main path taken every day by many people and already slow by 3 to 8 every time, just to save 2 minutes of inconvenience 5 times a year when bisect/backport/revert is actually needed.
• Building combinations of feature flags. Not all that important. Can do for only most commonly broken combinations, or via probabilistic sampling. Otherwise can be fixed in post.
• Caching not fully utilized leading to needless rebuilds of stuff.

Again - I don't want to say "disable everything" and I'm just speculating. But if the CI runs being too heavy is a bottleneck, maybe something to think about because any of these can take a 10 minute CI and turn it into a >1h CI.

But this, times 3 different compilers (nightly stable and MSRV), times two lockfiles, times release/debug (though for rust-bitcoin I turned off that flag because I couldn't handle another doubling) ... it adds up.

Well, yeah. While I was writing it you already described it. You're great at math, you know how multiplication works, even on elliptic curves not just integers like me. 🤣 . Do you really really need to ensure all of that stuff in the CI and have to wait for it? How often do these actually break, and how much of a big deal it is vs every * it creates?

In Fedimint we test all features, no features, defaults features, and only in the MQ, not PR itself. The rest is taken care of before the release. I still complained about it and was thinking it is unnecessary.

Same with toolchains. And especially if it is toolchains times lockfiles. That would be such a waste. What's the chance that this actually breaks in one combination only, and how much of a big deal is it?

And then if all of that is multiplied by 5x avg commits... 💀

I am building about 5000 derivations per commit (not including any dependencies etc; those are cached).

BTW. I never did a proper benchmark, but I'm of the opinion that crane's approach is actually better than rust2nix / cargo2nix . A crane build of Fedimint workspace (kind of) creates 2 derivations: one with the whole target with all dependencies built, and second one on top that builds the project. 2 vs hundreds/thousands. And thanks to that crane can zstd compress the whole target directory in both cases, incrementally w.r.t to each other yielding huge compression rates which then helps a ton with caching, which having each dependency individually can't do. cargo2nix approach lures people with the promise of incremental build of every dependency being so "optimized", while in reality dependencies change very rarely anyway. If anyone ever has any thoughts/meassurements contradicting it, please ping me. :)

Yes, you make good points that I certainly don't need to be doing toolchains times lockfiles. I'm not certain that I've ever seen a per-lockfile breakage, except when editing the lockfiles themselves and only on the MSRV compiler. (On the other hand, I see compiler-specific issues in the main codebase all the time.)

Interesting claims about crane and compression! I also struggle with disk space (but there I suspect that crate2nix's approach is better, even with compression, because it avoids so much duplication)

2 vs hundreds/thousands

It's more like 2 per project, vs less than a hundred across all my projects. I don't think they're dramatically different. Though I suppose if I were dealing with typical Rust projects with hundreds of dependencies this would be a big improvement.

With crate2nix I also only need to rebuild dependencies when my compiler changes, and then only once (not once per project).

I have also never done a proper benchmark of any of this. But I'm skeptical.

Since this conversation I have made two major changes:

• I reduced the CI matrix from ~1 hour to less than 30 minutes per commit, by eliminating some of the lockfile/compiler combinations and some of the feature/compiler combinations
• I rewrote the whole CI system so that it supports stacking merges, meaning that if I queue up 5 PRs then go to bed, it will make a branch off master with 5 merges. Then I just have to sign all the commits and push the whole stack. (Previously it would make 5 separate merges, meaning that I would sign and push one, then the others would get thrown away and recreated and retested off the new master. This wasted a ton of time and required I be available to sign-and-push many times throughout the day.)

This latter thing is only just implemented and I'm still working out the kinks, but we should see visibly higher CI velocity this week.

Plus... aren't we using Github already? :D

@dpc you may have missed that we are a substantial part of the migration off of Github into a self-hosted Forgejo instance. rust-secp is at https://gitea.bitcoin.ninja/rust-bitcoin/rust-secp256k1

There are some growing pains (in particular there's a whole new API that I need to implement with my local-ci tool, which does a fair bit more than just running these massive test matrices ... it also posts and counts ACKs and tracks my reviews, deals with release tags and signing merge commits, and I would like it to do more).

The main push for the movement has been the massive amount of LLM slop being posted by bot accounts and other negative-value "contributors".

As for data isolation is recommended and practical so I'm not too worried about that part.

If your simply looking for filesystem isolation, can I suggest chroot instead of full blown container environment. Anyway, my comment was more to the point of surveillance capitalism as much as giving AI free reign on a box.

s/your/you're

FWIW I am hoping to get cargo-rbmt's use-ability to the point where local and github are about the same, and I think it is getting close. It is kinda confusing to me that the Github CI doesn't just run on every commit, seems like it should be able to?

Yeah, but I personally am not doing any more work patching or learning GitHub actions since GitHub is nearly dead to us. After we move to forge we should definitely do this IMO.

Just to note, you can add fetch-depth: 0 in the checkout actions step and that gives you access to all the commits within the PR. You then can just loop over the commits and run some check. Its my understanding that you can't run the entire suite per commit unless you force each one to have the fetch depth flag and run them all as loops essentially.

I dug into it a little and came to your same conclusion @benalleng. I am attempting to codify the pattern in the cargo-rbmt tool over here: https://github.com/rust-bitcoin/rust-bitcoin-maintainer-tools/pull/99

some kind of web UI so you can view API diffs even without it being committed.

No, no no. If I want to see how the API changed over the history of the project I want to use git (actually, I use jj because git doesn't track enough stuff). I have no interest in relitigating this.

I just noticed one super idiotic thing - they include overridden trait methods but don't include default non-overriden methods, so diffs are artificially bigger.

Yes, there are a couple situations like this where the API files have unreasonably large diffs. There are many more cases where the API file diff matches the actual size of the API change, and in these cases they're super useful.

If I want to see how the API changed over the history of the project I want to use git (actually, I use jj because git doesn't track enough stuff). I have no interest in relitigating this.

OK but I presume you care about what was merged and not some temporary flukes, right? If so, how about having a bot commit it to a separate repository?

There are many more cases where the API file diff matches the actual size of the API change, and in these cases they're super useful.

To be clear, I do find it useful, I just don't like that it's at the cost of getting in the way of development.

No, no no. If I want to see how the API changed over the history of the project I want to use git (actually, I use jj because git doesn't track enough stuff). I have no interest in relitigating this.

Couldn't the API text files be created by simply walking the git history and running cargo-public-api on the sequence of commits (EG [(HEAD..HEAD~1), (HEAD~1..HEAD~2, ..]? I'm still baffled why we need to write into the Git repo what could be generated from data in the Git repo? And not that I would care, except the annoying Git rebase that takes place which adds round trips, not to mention the increased data requirements to store and parse the repo locally, not to mention the very large diffs, and trying to remember yet another script to run before submitting a patch..

Yeah, lemme just spend another $20000 on compute hardware to produce a nondeterministic version of what I already have.

Yeah, lemme just spend another $20000 on compute hardware to produce a nondeterministic version of what I already have.

It would be deterministic, and I could probably do it on my piece of crap laptop in a few day. I think we are just tying to find ways to reduce the round tripping here in this thread so we can have a 1.0; therefore I think this is a legitimate discussion point.