Skip to content

Commit 879c8fb

Browse files
authored
dev: include examples README checks in the local lint suite (apache#25466)
## Which issue does this PR close? - Part of apache#21048. ## Rationale for this change `dev/rust_lint.sh` does not run the examples README check from the `examples-docs-check` job. A contributor finds a stale `datafusion-examples/README.md` only in CI. The script also writes to a fixed `README-NEW.md` and a shared file under `/tmp`, and its update hint is a pipeline that does not work. ## What changes are included in this PR? - `ci/scripts/check_examples_docs.sh` writes its generated README and diff into a scratch directory created per run and removes it on exit, including on interruption. A `diff` error is reported as a comparison failure, not as a stale README. - The script gets `--write` and `--allow-dirty`, with the same clean-tree rule as `rust_fmt.sh`. `--write` replaces the README with the generated one. Without it, the script only reports and tells the contributor to rerun with `--write`. - `dev/rust_lint.sh` runs the script as a write step, so `./dev/rust_lint.sh --write` updates the README with the other formatters. - `docs/source/contributor-guide/testing.md` documents the check and the update command. The generator command, the Prettier version, and the comparison do not change. CI calls the script the same way as before. ## What is the testing strategy for this PR? - In a clean clone: a fresh README passes, a committed stale README fails with the diff, `--write` restores it byte for byte, `--write` on a dirty tree is refused, and `--write --allow-dirty` runs. No lockfile changed. - A fixture with stub `cargo` and `npx` covered generator and formatter failures in both modes, a `diff` error, a missing `npx`, SIGINT and SIGTERM cleanup, concurrent runs, and a run from a subdirectory. The README was untouched in every failure case. - A fixture with stubbed steps covered the runner: flags are forwarded in both write modes and absent in check mode, and a failure stops later steps. The full `./dev/rust_lint.sh` passes. ## Are there any user-facing changes? No. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
1 parent 64871d9 commit 879c8fb

3 files changed

Lines changed: 114 additions & 42 deletions

File tree

‎ci/scripts/check_examples_docs.sh‎

Lines changed: 100 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -17,61 +17,119 @@
1717
# specific language governing permissions and limitations
1818
# under the License.
1919

20-
# Generates documentation for DataFusion examples using the Rust-based
21-
# documentation generator and verifies that the committed README.md
22-
# is up to date.
23-
#
24-
# The README is generated from documentation comments in:
25-
# datafusion-examples/examples/<group>/main.rs
26-
#
27-
# This script is intended to be run in CI to ensure that example
28-
# documentation stays in sync with the code.
29-
#
30-
# To update the README locally, run this script and replace README.md
31-
# with the generated output.
20+
# Generates the examples README with the Rust generator and checks that the
21+
# committed `datafusion-examples/README.md` matches it, the same way the
22+
# "check example README is up-to-date" job does. With `--write`, replaces the
23+
# README with the generated one.
3224

3325
set -euo pipefail
3426

35-
ROOT_DIR="$(git rev-parse --show-toplevel)"
27+
SCRIPT_NAME="$(basename "${BASH_SOURCE[0]}")"
28+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
29+
ROOT_DIR="$(cd "${SCRIPT_DIR}/../.." && pwd)"
30+
31+
source "${SCRIPT_DIR}/utils/tool_versions.sh"
32+
source "${SCRIPT_DIR}/utils/git.sh"
33+
34+
EXAMPLES_DIR="${ROOT_DIR}/datafusion-examples"
35+
README="${EXAMPLES_DIR}/README.md"
36+
37+
MODE="check"
38+
ALLOW_DIRTY=0
39+
40+
usage() {
41+
cat >&2 <<USAGE
42+
Usage: $0 [--write] [--allow-dirty]
43+
44+
Checks that datafusion-examples/README.md matches the output of the examples-docs generator.
45+
--write Replace the README with the generated one (requires a clean git worktree, no uncommitted changes).
46+
--allow-dirty Allow \`--write\` to run even when the git worktree has uncommitted changes.
47+
USAGE
48+
exit 1
49+
}
50+
51+
while [[ $# -gt 0 ]]; do
52+
case "$1" in
53+
--write)
54+
MODE="write"
55+
;;
56+
--allow-dirty)
57+
ALLOW_DIRTY=1
58+
;;
59+
-h|--help)
60+
usage
61+
;;
62+
*)
63+
usage
64+
;;
65+
esac
66+
shift
67+
done
3668

37-
# Load centralized tool versions
38-
source "${ROOT_DIR}/ci/scripts/utils/tool_versions.sh"
69+
if [[ "$MODE" == "write" && $ALLOW_DIRTY -eq 0 ]]; then
70+
require_clean_work_tree "$SCRIPT_NAME" || exit 1
71+
fi
72+
73+
if ! command -v npx >/dev/null 2>&1; then
74+
echo "[${SCRIPT_NAME}] npx is required to run the prettier check. Install Node.js (e.g., brew install node) and re-run." >&2
75+
exit 1
76+
fi
77+
78+
# The scratch directory sits beneath the examples directory so Prettier finds
79+
# the same configuration as for the committed README.
80+
SCRATCH_DIR="$(mktemp -d "${EXAMPLES_DIR}/.examples-docs-check.XXXXXX")"
81+
trap 'rm -rf "${SCRATCH_DIR}"' EXIT
82+
trap 'exit 130' INT
83+
trap 'exit 143' TERM
84+
README_NEW="${SCRATCH_DIR}/README.md"
85+
README_DIFF="${SCRATCH_DIR}/README.diff"
3986

40-
EXAMPLES_DIR="$ROOT_DIR/datafusion-examples"
41-
README="$EXAMPLES_DIR/README.md"
42-
README_NEW="$EXAMPLES_DIR/README-NEW.md"
87+
cd "${ROOT_DIR}"
4388

4489
echo "▶ Generating examples README (Rust generator)…"
4590
cargo run --quiet \
46-
--manifest-path "$EXAMPLES_DIR/Cargo.toml" \
91+
--manifest-path "${EXAMPLES_DIR}/Cargo.toml" \
4792
--bin examples-docs \
48-
> "$README_NEW"
93+
> "${README_NEW}"
4994

5095
echo "▶ Formatting generated README with prettier ${PRETTIER_VERSION}…"
5196
npx "prettier@${PRETTIER_VERSION}" \
5297
--parser markdown \
53-
--write "$README_NEW"
98+
--write "${README_NEW}"
5499

55-
echo "▶ Comparing generated README with committed version…"
56-
57-
if ! diff -u "$README" "$README_NEW" > /tmp/examples-readme.diff; then
58-
echo ""
59-
echo "❌ Examples README is out of date."
60-
echo ""
61-
echo "The examples documentation is generated automatically from:"
62-
echo " - datafusion-examples/examples/<group>/main.rs"
63-
echo ""
64-
echo "To update the README locally, run:"
65-
echo ""
66-
echo " cargo run --bin examples-docs \\"
67-
echo " | npx prettier@${PRETTIER_VERSION} --parser markdown --write \\"
68-
echo " > datafusion-examples/README.md"
69-
echo ""
70-
echo "Diff:"
71-
echo "------------------------------------------------------------"
72-
cat /tmp/examples-readme.diff
73-
echo "------------------------------------------------------------"
74-
exit 1
100+
if [[ "$MODE" == "write" ]]; then
101+
cp "${README_NEW}" "${README}"
102+
echo "✅ Examples README updated."
103+
exit 0
75104
fi
76105

77-
echo "✅ Examples README is up-to-date."
106+
echo "▶ Comparing generated README with committed version…"
107+
diff_status=0
108+
diff -u "${README}" "${README_NEW}" > "${README_DIFF}" || diff_status=$?
109+
110+
case "${diff_status}" in
111+
0)
112+
echo "✅ Examples README is up-to-date."
113+
;;
114+
1)
115+
echo ""
116+
echo "❌ Examples README is out of date."
117+
echo ""
118+
echo "The examples documentation is generated automatically from:"
119+
echo " - datafusion-examples/examples/<group>/main.rs"
120+
echo ""
121+
echo "To update the README, run:"
122+
echo ""
123+
echo " ./ci/scripts/check_examples_docs.sh --write"
124+
echo ""
125+
echo "Diff:"
126+
echo "------------------------------------------------------------"
127+
cat "${README_DIFF}"
128+
echo "------------------------------------------------------------"
129+
exit 1
130+
;;
131+
*)
132+
echo "❌ diff exited with status ${diff_status} while comparing the README; no comparison result." >&2
133+
exit "${diff_status}"
134+
;;
135+
esac

‎dev/rust_lint.sh‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -129,6 +129,7 @@ declare -a WRITE_STEPS=(
129129
"ci/scripts/license_header.sh|true"
130130
"ci/scripts/typos_check.sh|true"
131131
"ci/scripts/doc_prettier_check.sh|true"
132+
"ci/scripts/check_examples_docs.sh|true"
132133
)
133134

134135
declare -a READONLY_STEPS=(

‎docs/source/contributor-guide/testing.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -275,6 +275,19 @@ To run either check on its own:
275275
[`dev/depcheck`]: https://github.com/apache/datafusion/tree/main/dev/depcheck
276276
[cargo-machete]: https://github.com/bnjbvr/cargo-machete
277277

278+
## Examples README Check
279+
280+
`datafusion-examples/README.md` is generated from the documentation comments in
281+
`datafusion-examples/examples/<group>/main.rs`. `ci/scripts/check_examples_docs.sh`
282+
regenerates it and fails if the committed file differs. `./dev/rust_lint.sh`
283+
runs it and needs `cargo` and `npx`. To run the check on its own, or to update
284+
the README:
285+
286+
```shell
287+
./ci/scripts/check_examples_docs.sh
288+
./ci/scripts/check_examples_docs.sh --write
289+
```
290+
278291
## Benchmarks
279292

280293
### Criterion Benchmarks

0 commit comments

Comments
 (0)