Note: this issue is blocked on #2545 being merged (and will possibly become inapplicable if we end up splitting that code into a separate crate)

[note to self] we should make this required for merging PRs. Will split off a separate issue for this

Originally posted by @szokeasaurusrex in #2545 (comment)