Skip to content

Commit 81bbae3

Browse files
authored
Improve agents related toolings/docs (#9264)
## Rationale for this change 1. Remove the PR review skill, all agents have this functionality these days. 2. Minimize the top level `AGENTS.md`, point to other more specific ones 3. Move benchmark guidelines to `STYLE.md` 4. Create AGENTS.md files for `vortex-python` and `docs/`, hopefully making the process around them clearer and better documented. --------- Signed-off-by: Adam Gutglick <[email protected]>
1 parent be332fc commit 81bbae3

15 files changed

Lines changed: 213 additions & 231 deletions

File tree

‎.agents/skills/pr-review/SKILL.md‎

Lines changed: 0 additions & 93 deletions
This file was deleted.

‎.github/workflows/ci.yml‎

Lines changed: 2 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -126,14 +126,9 @@ jobs:
126126
pytest --benchmark-disable test/test_hf_datasets.py
127127
working-directory: vortex-python/
128128

129-
- name: Doctest - PyVortex
129+
- name: Check Python documentation
130130
run: |
131-
uv run --all-packages make doctest
132-
working-directory: docs/
133-
134-
- name: Ensure docs build - PyVortex
135-
run: |
136-
uv run --all-packages make html
131+
uv run --all-packages make check
137132
working-directory: docs/
138133

139134
# Keep this last: setup-benchmark.sh confines everything except the wrapped

‎.github/workflows/fuzzer-fix-automation.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -313,7 +313,7 @@ jobs:
313313
- After writing tests, post the COMPLETE test code inline
314314
- This ensures your work is visible and reviewable even if you hit the turn limit
315315
- Keep fixes minimal - only fix the specific bug
316-
- Follow CLAUDE.md code style guidelines
316+
- Follow `AGENTS.md` and any applicable nested `AGENTS.md` guidelines
317317
- **Use `--dev` flag** for faster builds: `cargo +nightly fuzz run --dev --sanitizer=none <target> <crash_file> -- -rss_limit_mb=0`
318318
319319
## Fixability Guidelines

‎AGENTS.md‎

Lines changed: 38 additions & 95 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,6 @@ Guidance for Claude, Codex and other coding agents working in the Vortex reposit
44

55
## Task Routing
66

7-
- When asked to review a PR, especially via `/pr-review`, use the
8-
`.agents/skills/pr-review` skill.
97
- When asked a question about the PR or codebase, especially via `/query`, use the
108
`.agents/skills/query` skill.
119
- When asked to investigate a CI failure, especially via `/ci-failure-analysis`, use the
@@ -34,6 +32,14 @@ documentation in `docs/`, and benchmark tooling in `vortex-bench/` and `benchmar
3432
and execution integrations.
3533
- `vortex-python` contains Python bindings. RST-flavored project docs live in `docs/`.
3634

35+
## Scoped Guidance
36+
37+
Before changing files in a subtree, read the closest nested `AGENTS.md`. In particular:
38+
39+
- `.github/AGENTS.md` covers workflows and other GitHub configuration.
40+
- `docs/AGENTS.md` covers Sphinx documentation.
41+
- `vortex-python/AGENTS.md` covers Python and PyO3 binding work.
42+
3743
## Build
3844

3945
Prefer narrow crate builds while iterating:
@@ -57,39 +63,16 @@ Run tests for the crate or binding you touched before broader checks:
5763
cargo nextest run -p <crate-name>
5864
```
5965

60-
if cargo-nextest is not available you can install it with
61-
```bash
62-
cargo install --locked cargo-nextest
63-
```
64-
65-
Examples:
66-
67-
```bash
68-
cargo nextest run -p vortex-array
69-
make -C docs doctest
70-
uv run --all-packages pytest vortex-python/test
71-
cargo test --doc
72-
```
73-
74-
Run docs doctests from the docs directory with `make -C docs doctest` so the correct Sphinx
75-
Makefile target is used.
76-
77-
If you touch documentation run doc tests via `cargo test --doc`.
78-
79-
For Python binding changes under `vortex-python/`, run the narrow Python checks that match the
80-
files touched before broader test suites. Useful checks include:
66+
If cargo-nextest is not available, you can install it with:
8167

8268
```bash
83-
python -m py_compile <changed-python-files>
84-
uv run --all-packages --reinstall-package vortex-data pytest <changed-python-tests>
69+
cargo install --locked cargo-nextest
8570
```
8671

87-
If Python docstrings, `docs/api/python/`, or Sphinx configuration change, also run the docs checks
88-
from a clean Sphinx environment:
72+
For Rust doc comments or crate documentation, run doctests for the affected crate:
8973

9074
```bash
91-
uv run --all-packages make -C docs clean html
92-
uv run --all-packages make -C docs clean doctest
75+
cargo test --doc -p <crate-name>
9376
```
9477

9578
## Linting, Formatting, and Generated Files
@@ -100,30 +83,29 @@ with no Rust/API behavior impact. For docs/config-only changes, validate formatt
10083
or with a targeted doc/config command, and verify symlink or path changes with `ls`, `find`, and
10184
`git status`.
10285

103-
For Python binding changes under `vortex-python/`, run the relevant Python lint and type checks:
86+
For Rust code, public API, feature flag, or generated-file changes, run these before stopping:
10487

10588
```bash
106-
uv run basedpyright vortex-python
107-
uv run ruff check <changed-python-files>
89+
cargo +nightly fmt --all
90+
cargo clippy --all-targets --all-features
10891
```
10992

110-
If PyO3 Rust files in `vortex-python/src/` change, include `cargo +nightly fmt --check -p
111-
vortex-python`. Always finish Python binding work with `git diff --check`.
112-
113-
For Rust code, public API, feature flag, or generated-file changes, run these before stopping:
93+
For changed C++ and CUDA source or header files covered by CI (`.cpp`, `.hpp`, `.cu`, `.cuh`, and
94+
`.h` files under `lang/cpp`, `vortex-cuda`, `vortex-duckdb`, and `vortex-ffi`), format with the
95+
repository's `.clang-format` configuration and verify the result:
11496

11597
```bash
116-
cargo +nightly fmt --all
117-
cargo clippy --all-targets --all-features
98+
clang-format --style=file -i <changed-files>
99+
clang-format --dry-run --Werror --style=file <changed-files>
118100
```
119101

102+
Pass only the files you changed; CI excludes vendored or generated CUDA and Arrow headers from its
103+
repository-wide check.
104+
120105
Notes:
121106

122107
- For `.github/` changes, follow `.github/AGENTS.md` and run
123108
`yamllint --strict -c .yamllint.yaml` on changed workflow files.
124-
- You can try
125-
`cargo fix --lib --allow-dirty --allow-staged && cargo clippy --fix --lib --allow-dirty --allow-staged`
126-
to fix minor Rust diagnostics automatically when working on Rust code.
127109
- If cargo fails with exactly `sccache: error: Operation not permitted`, rerun that command
128110
with `RUSTC_WRAPPER=` so rustc runs directly. Only do this for that exact error.
129111

@@ -139,62 +121,16 @@ Notes:
139121

140122
## Rust Code Style
141123

142-
- Prefer `impl AsRef<T>` to `&T` for public interfaces where practical, for example
143-
`impl AsRef<Path>`.
144-
- Avoid `unsafe` unless it is necessary. Prefer zero-cost safe abstractions, or cheap
145-
non-zero-cost safe abstractions, over hand-written unsafe code.
146-
- Every new public API definition must have a doc comment. Examples are useful but not required.
147-
- Use `vortex_err!` to create a `VortexError` with a format string.
148-
- Use `vortex_bail!` to create and immediately return a `VortexError` as a `VortexResult<T>`.
149-
- Keep imports at the top of the module. The only exception is a `#[cfg(test)]` test module,
150-
where imports should be at the top of that module.
151-
- Prefer imports over qualified identifiers when the name is used repeatedly.
152-
- Avoid function-scoped imports unless required or unless fully qualifying both sides would be
153-
exceptionally verbose.
124+
- Follow `STYLE.md` for Rust formatting, documentation, API, error-handling, import, and safety
125+
conventions.
154126
- Only write comments that explain non-obvious logic or important context. Do not comment
155127
self-explanatory code.
156128
- Keep public APIs small and consistent with neighboring crates.
157129

158-
## Performance: avoid hidden-cost accessors in hot loops
159-
160-
The most common performance trap in this codebase is calling a *per-element accessor that
161-
hides non-trivial work* inside an `O(n)` loop, instead of doing the work once over the whole
162-
chunk. The call site looks like a cheap getter, but each call re-pays a cost that is constant
163-
(or amortizable) across the loop, making the loop `O(n · k)`.
164-
165-
Watch for these accessors used inside `for i in 0..n { ... }`:
166-
167-
| Per-element accessor (in a loop) | Hidden cost | Bulk replacement |
168-
| --- | --- | --- |
169-
| `Validity::is_valid(i)` / `is_null(i)` | for array-backed validity, **allocates an `ExecutionCtx` and runs a scalar lookup per call** | `validity.execute_mask(len, ctx)?` once, then `Mask::value(i)` |
170-
| `array.scalar_at(i)` / `array.execute_scalar(i, ctx)` | per-element execution through the compute stack | canonicalize once (`execute::<PrimitiveArray>` / `as_slice`) then index |
171-
| `BitBuffer::value(i)` / `Mask::value(i)` accumulated into a count | recomputes the byte address each call; defeats popcount | `true_count()`, `BitBuffer::count_range(start, end)`, `set_indices()` |
172-
| `BitIterator::next()` to accumulate a running rank/prefix count | bit-at-a-time | `count_range` over each gap (SIMD popcount) |
173-
| re-deriving a value inside the loop (e.g. `self.validity()?` each iteration) | re-runs the derivation `n` times | hoist it above the loop |
174-
175-
Decide per site — bulk is not always the answer:
176-
177-
- **Sequential / contiguous access** over an accessor that hides amortizable work → hoist and
178-
go bulk (materialize once, then index or iterate the chunk).
179-
- **Gather over arbitrary indices** → you cannot amortize a per-element *decode*, but you can
180-
still materialize the backing buffer once (e.g. `execute_mask`) and then do cheap `O(1)`
181-
random reads, avoiding per-call context/allocation.
182-
- **The accessor is already genuinely `O(1)`** (e.g. reading an already-materialized `Mask`/
183-
slice, or a native bitmap) → leave it; bulk would not help.
184-
185-
Even after materializing into a `Mask`/`BitBuffer`, **do not loop with `value(i)` per
186-
element** to act on each set bit — the per-element branch dominates. Iterate a `u64` word
187-
at a time with all-set / all-unset fast paths: use [`BitBuffer::for_each_set_index`]. It
188-
beats `for i in 0..len { if buf.value(i) {..} }` by 2-45x (more at low density) and beats
189-
collecting `set_indices()` by ~2x at mid/high density, while self-adapting from sparse to
190-
dense — see `vortex-mask/benches/mask_iteration.rs`. Reach for the cached `indices()` /
191-
`slices()` representations when you need them more than once; otherwise `for_each_set_index`
192-
needs no materialization.
193-
194-
When you touch such a loop, back the change with a benchmark (see
195-
`vortex-array/benches/validity_is_valid.rs` for the `is_valid` case,
196-
`vortex-mask/benches/valid_counts.rs` for the popcount case, and
197-
`vortex-mask/benches/mask_iteration.rs` for the per-element-vs-word-vs-sparse comparison).
130+
## Performance
131+
132+
Avoid hidden-cost per-element accessors in hot loops, follow the performance guidance in
133+
`STYLE.md`, and benchmark changes to hot paths.
198134

199135
## Tests
200136

@@ -225,14 +161,21 @@ Check new and modified lines against this list before finishing:
225161
intended semantics.
226162
- Silently reducing the scope of an approved plan when implementation is harder than expected.
227163
- Calling a hidden-cost per-element accessor (`Validity::is_valid`, `scalar_at`, `BitBuffer::
228-
value` accumulation) inside a hot loop instead of materializing once — see
229-
"Performance: avoid hidden-cost accessors in hot loops".
164+
value` accumulation) inside a hot loop instead of materializing once.
230165

231166
## Summaries
232167

233168
When summarizing work, write valid Markdown that can be copied into GitHub. Include the checks
234169
you ran and call out any checks you could not run.
235170

171+
## Branches
172+
173+
When creating a branch on the user's behalf, prefix its name with the user's established branch
174+
prefix, not the agent's name (for example, do not use `codex/` or `claude/`). Infer the user's
175+
prefix from their existing branches when possible. Otherwise, use available identity context such
176+
as `whoami`, Git configuration, or other information the user has provided. If the evidence is
177+
ambiguous, ask the user instead of inventing a prefix.
178+
236179
## Commits
237180

238181
All commits must be signed off by the committers in this form:

‎CONTRIBUTING.md‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,75 @@ The contribution process is outlined below:
3434
- Disclose LLM usage as described in [AI Assistance](#ai-assistance).
3535
- CI requires approval from external committers.
3636

37+
## Development Workflows
38+
39+
The repository uses [`uv`](https://docs.astral.sh/uv/) to manage its Python workspace. From the
40+
repository root, create or update the development environment with:
41+
42+
```bash
43+
uv sync --all-packages
44+
```
45+
46+
### Python bindings
47+
48+
`vortex-data` is a mixed Python and Rust package. `uv` manages its Python environment, and
49+
[Maturin](https://www.maturin.rs/) compiles the PyO3 code in `vortex-python/src/` into the
50+
`vortex._lib` extension.
51+
52+
Python-only changes do not require an explicit extension rebuild. Run the affected tests directly:
53+
54+
```bash
55+
uv run --all-packages pytest <changed-python-tests>
56+
```
57+
58+
After changing the PyO3 Rust code, `vortex-python/Cargo.toml`, or relevant Cargo features, rebuild
59+
the extension before testing. For a single build-and-test cycle, ask `uv` to reinstall the local
60+
package, which invokes its Maturin build backend:
61+
62+
```bash
63+
uv run --all-packages --reinstall-package vortex-data pytest <changed-python-tests>
64+
```
65+
66+
For repeated Rust edits, install the extension into the development environment explicitly:
67+
68+
```bash
69+
(cd vortex-python && uv run maturin develop)
70+
uv run --all-packages pytest <changed-python-tests>
71+
```
72+
73+
Rerun `maturin develop` after each Rust change. Pass Cargo features through Maturin when needed:
74+
75+
```bash
76+
(cd vortex-python && uv run maturin develop --features <feature>)
77+
```
78+
79+
Use one rebuild path per test cycle. Before handing off broad Python binding changes, run the full
80+
Python check:
81+
82+
```bash
83+
./vortex-python/check.sh
84+
```
85+
86+
This synchronizes the workspace, rebuilds the extension, runs Python formatting, linting, type
87+
checks and tests, and validates the documentation.
88+
89+
### Documentation
90+
91+
Use `docs/Makefile` as the canonical interface for Sphinx. From the repository root, run the full
92+
clean build and doctest flow with:
93+
94+
```bash
95+
uv run --all-packages make -C docs check
96+
```
97+
98+
For live-reloading development, run:
99+
100+
```bash
101+
make -C docs serve
102+
```
103+
104+
Use `make -C docs help` to list focused targets, and finish documentation changes with `check`.
105+
37106
## Governance
38107

39108
Vortex is an independent open-source project and not controlled by any single company. The Vortex Project is a sub-project of the Linux Foundation Projects. As such, the governance is subject to the terms of the [Technical Charter](https://vortex.dev/charter.pdf).

0 commit comments

Comments
 (0)