Skip to content

Commit f43f67b

Browse files
chore: check for circular dependencies with cargo publish (apache#26582)
## Which issue does this PR close? - Related to apache#25088, apache#25148 ## Rationale for this change Normal `cargo` commands reject circular dependencies between packages, but they allow cycles through dev-dependencies. This has bitten us in the past, which is why the `dev/depcheck` tool and its associated CI job were added to detect dev-dependency cycles. Running `dev/depcheck` in CI is relatively expensive, in part because it depends on the `cargo` crate, which has a large dependency tree. Building and running `depcheck` takes ~130 seconds on a 4 vCPU container, and also requires an `ubuntu-latest` runner. Cargo 1.90 added support for multiple packages in a single `cargo publish`, which does check for dev-dependency cycles. It follows that we can use a wrapper script over `cargo publish --workspace --dry-run` to replace `depcheck`. This is more efficient and can run on `ubuntu-slim`. Moreover, it avoids the need to maintain a separate tool and ensures that the CI check is consistent with how packages are actually published. (We should probably also switch to using multi-package `cargo publish` to make releases, but I'll defer that for now.) One downside is that error reporting when a cycle is found is less helpful, but I don't think that is a major concern. ## What changes are included in this PR? * Delete `dev/depcheck` * Change `ci/scripts/check_circular_dependencies.sh` to call `cargo publish --dry-run --workspace` * The "Circular Dependency Check" now runs on `ubuntu-slim` ## What is the testing strategy for this PR? CI / tooling only. Manual local testing: - Small test workspace with a dev-dependency cycle: `cargo test` passes, and `cargo package` / `cargo publish --workspace --dry-run` fail. - Same workspace without the cycle: packaging passes. - Same workspace with a dev-dependency that has a path but no version: packaging passes, because Cargo drops that dependency. - DataFusion main: the dry run passes in 15 seconds and skips the 11 `publish = false` crates. - DataFusion with a dev-dependency cycle added: `cargo check --tests` passes, and the dry run fails. - DataFusion with a normal-dependency cycle added: `cargo check` fails by itself. - DataFusion bumped to unpublished versions: the dry run passes. - Bumped versions plus the dev-dependency cycle: the dry run fails. ## Are there any user-facing changes? No. Co-authored-by: Kumar Ujjawal <[email protected]>
1 parent 2717e25 commit f43f67b

10 files changed

Lines changed: 35 additions & 4367 deletions

File tree

‎.github/dependabot.yml‎

Lines changed: 0 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -62,14 +62,6 @@ updates:
6262
update-types:
6363
- "minor"
6464
- "patch"
65-
# dev/depcheck is a standalone crate excluded from the workspace, so it is
66-
# not covered by the workspace entry above
67-
- package-ecosystem: cargo
68-
directory: "/dev/depcheck"
69-
schedule:
70-
interval: weekly
71-
target-branch: main
72-
labels: [auto-dependencies]
7365
# the wasm test app has its own npm lockfile
7466
- package-ecosystem: npm
7567
directory: "/datafusion/wasmtest/datafusion-wasm-app"

‎.github/workflows/rust.yml‎

Lines changed: 4 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -980,21 +980,15 @@ jobs:
980980
working-directory: datafusion/proto
981981
run: cargo msrv --output-format json --log-target stdout verify
982982

983-
# Check for dependency cycles between DataFusion crates with `dev/depcheck`.
983+
# Check for dependency cycles between DataFusion crates, which would block
984+
# publishing them to crates.io, with a publishing dry run.
984985
depcheck:
985986
name: Circular Dependency Check
986987
if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') && !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}
987-
runs-on: ubuntu-latest
988-
container:
989-
image: amd64/rust
988+
runs-on: ubuntu-slim
990989
steps:
991990
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
992-
with:
993-
submodules: true
994-
fetch-depth: 1
995991
- name: Setup Rust toolchain
996-
uses: ./.github/actions/setup-builder
997-
with:
998-
rust-version: stable
992+
uses: ./.github/actions/setup-rustup
999993
- name: Check dependencies
1000994
run: bash ci/scripts/check_circular_dependencies.sh

‎Cargo.toml‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -69,7 +69,6 @@ members = [
6969
"datafusion/doc",
7070
"xtask",
7171
]
72-
exclude = ["dev/depcheck"]
7372
resolver = "2"
7473

7574
[workspace.package]

‎ci/scripts/check_circular_dependencies.sh‎

Lines changed: 23 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -17,18 +17,34 @@
1717
# specific language governing permissions and limitations
1818
# under the License.
1919

20-
# Checks for circular dependencies between DataFusion crates with
21-
# `dev/depcheck`, the same way the "Circular Dependency Check" job does.
20+
# Checks for circular dependencies between DataFusion crates, the same way the
21+
# "Circular Dependency Check" job does.
22+
#
23+
# A cycle prevents publishing the crates to crates.io. A crate can be
24+
# published only after every DataFusion crate it depends on, including through
25+
# dev-dependencies, optional dependencies, and dependencies for other targets,
26+
# so no crate in a cycle can be published first. Cargo refuses to build a
27+
# cycle of normal dependencies, but it allows one that passes through a
28+
# dev-dependency, because a dev-dependency is built only for the crate's own
29+
# tests, examples, and benchmarks. Such a cycle builds and passes tests, and
30+
# fails only at release time.
31+
#
32+
# The check is a publishing dry run. `cargo publish --workspace` packages every
33+
# crate that does not set `publish = false`, in an order where each crate comes
34+
# after the DataFusion crates it depends on, and fails if there is no such
35+
# order. `--dry-run` stops before uploading, and `--no-verify` skips building
36+
# the packages. The dry run also fails on any other problem that would stop the
37+
# crates from being published, such as a dependency without a version.
38+
# `--allow-dirty` lets the check run on a working tree with uncommitted
39+
# changes.
2240

2341
set -euo pipefail
2442

2543
SCRIPT_NAME="$(basename "${BASH_SOURCE[0]}")"
2644
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
2745
ROOT_DIR="$(cd "${SCRIPT_DIR}/../.." && pwd)"
2846

29-
# depcheck is outside the root workspace and finds the root manifest from its
30-
# own directory, so run Cargo from there.
31-
cd "${ROOT_DIR}/dev/depcheck"
47+
cd "${ROOT_DIR}"
3248

33-
echo "[${SCRIPT_NAME}] \`cargo run --locked\` in dev/depcheck"
34-
cargo run --locked
49+
echo "[${SCRIPT_NAME}] \`cargo publish --workspace --dry-run --no-verify --allow-dirty\`"
50+
cargo publish --workspace --dry-run --no-verify --allow-dirty

0 commit comments

Comments
 (0)