Skip to content

Commit 95c000d

Browse files
committed
# Conflicts: # tests/integration/test_semantic_router.py
2 parents a433882 + 31d2d0a commit 95c000d

39 files changed

Lines changed: 2109 additions & 345 deletions
Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
---
2+
name: pre-pr-review
3+
description: Review a RedisVL branch before a human sees it, fix what holds up, and write a Reviewer Brief into the PR description so the approver knows where to look. Use it whenever you're about to open a PR, mark one ready for review, or push substantive changes to one, and when you're handling Bugbot or reviewer comments, even if nobody asked for a review by name.
4+
compatibility: Works with any coding agent. Needs git, make, pytest and Python 3. Optional - gh, to read and update the PR; the public Redis Docs MCP at https://redis.io/mcp, for the Redis behaviour lens.
5+
---
6+
7+
# Pre-PR review
8+
9+
A human approves every merge to `main`. This review runs before they see the PR, so the problems it catches never reach them. Its output, the Reviewer Brief, gives them a head start: where to begin reading, where the change could still fail, and what nobody has checked.
10+
11+
The approver can only rely on the brief if it's accurate, so report only what you can tie to the code, and never claim a check you didn't run.
12+
13+
## 1. Scope the change
14+
15+
- Diff the work against its merge base with `main`: `git diff $(git merge-base origin/main HEAD)` covers commits and uncommitted changes, and `git status` shows untracked files to read as well.
16+
- Read each changed file in full, not just the hunks. Bugs often sit in the unchanged lines a hunk depends on.
17+
- For every changed public function, method or class, read its callers and its sync or async twin (`SearchIndex` and `AsyncSearchIndex`, and so on). Drift between twins is one of this project's most common defects.
18+
- In a stack of PRs you may review the whole stack at once, but keep track of which PR introduced each line. Each PR gets its own brief.
19+
20+
## 2. Choose lenses
21+
22+
Run all six lenses by default. Each catches problems the others miss, and running all of them is the most thorough option.
23+
24+
A lens that finds nothing relevant in the diff has still done its job; say nothing about it beyond the footer. If you need to save tokens, the "Most useful when" column says where each lens pays off. Correctness always runs. The footer of the brief lists the lenses that ran, so the reviewer can see what was covered.
25+
26+
| Lens | File | Most useful when |
27+
|---|---|---|
28+
| Correctness | `references/correctness.md` | Always |
29+
| Security | `references/security.md` | The diff touches `redisvl/mcp/`, `redisvl/query/filter.py`, `redisvl/utils/token_escaper.py`, code that builds query strings or `SCAN` patterns, destructive methods (`delete`, `clear`, drop), `redisvl/redis/connection.py`, `.github/`, or dependencies |
30+
| Design | `references/design.md` | Feature work, new or changed public API or MCP surface, work on deprecated APIs, new vectorizer, reranker or provider modules |
31+
| Tests | `references/tests.md` | Tests are added or changed, or the change sits in an area CI can't exercise (cluster, live provider APIs) |
32+
| Docs | `references/docs.md` | `docs/`, notebooks, docstrings or comments change, or public behaviour changes |
33+
| Redis behaviour | `references/redis.md` | The diff changes how Redis commands or queries are built, relies on a server default, limit or reply shape, or states how Redis behaves |
34+
35+
## 3. Review, one lens at a time
36+
37+
Run one pass per lens. Start each pass from the diff and that lens's file, not from an earlier pass's conclusions, so that one pass's blind spots don't become the next one's.
38+
39+
- If your agent can start isolated sub-tasks (subagents, sub-sessions, parallel workers), give each lens its own, and run them in parallel.
40+
- If it can't, run the lenses one after another, and write each pass's findings to a scratch file before starting the next.
41+
42+
Write each finding in this shape, so the challenger can check it quickly:
43+
44+
```
45+
[lens] path/to/file.py:LINE: claim, in plain words. Failure: the input or state that goes wrong, and what happens. Fix: the smallest change that resolves it.
46+
```
47+
48+
Skip formatting, import order and typing nits, which CI and `make lint` already catch.
49+
50+
## 4. Challenge the findings
51+
52+
Run a separate challenger pass, as described in `references/challenger.md`. Review passes produce false positives, and every one that survives costs someone time. Keep findings that score 7 or more out of 10 and have a file, a line and a concrete failure or cost.
53+
54+
Run a second challenge round only if the first round's fixes changed the code materially.
55+
56+
## 5. Fix and re-check
57+
58+
- Fix the kept findings that are within the PR's scope, at the root cause. A real problem outside scope goes to the author as a question, not into the diff.
59+
- Add the narrowest regression test that fails without the fix, or note why none applies. A test you've seen fail without the fix is strong evidence for the brief's "Beyond CI" line.
60+
- Re-run whatever the fixes touched: `make format`, `make check-types`, and the affected tests (`make test ARGS="tests/unit/test_x.py"`, or `pytest` on the specific file).
61+
- Commit the fixes locally. The author pushes.
62+
63+
## 6. Settle open questions with the author
64+
65+
A PR should reach the reviewer with no open questions. Before the PR is opened or marked ready, and before each later rewrite of the brief, take everything the review couldn't settle to the author and get an answer: findings outside the PR's scope, rejections you're unsure about, and trade-offs. You work for the author, and the reviewer shouldn't have to answer questions the author never saw.
66+
67+
Once the author has decided, record a deliberate trade-off that a reviewer might question in the PR's Notes section, as a decision with its reason. Notes say what was decided and why; they don't ask the reviewer anything.
68+
69+
If you can't reach the author, for example because you're running unattended, don't open the PR or mark it ready. Leave it as a draft, or unopened, and put the questions in your report to the author, never in the PR.
70+
71+
## 7. Write the Reviewer Brief
72+
73+
Write the brief following `references/reviewer-brief.md`. The reviewer sees the change as it is now, so the brief points at the current code, not at the problems you fixed.
74+
75+
To put it in the PR description, save the brief to a file and run:
76+
77+
```
78+
python3 .agents/skills/pre-pr-review/scripts/update_brief.py <pr-number> <brief-file>
79+
```
80+
81+
The script replaces only the `## Reviewer Brief` section and keeps the rest of the description, including Bugbot's summary. Add `--dry-run` to print the result without changing the PR. If the PR doesn't exist yet, put the brief under `## Reviewer Brief` in the description you open it with. Without `gh`, give the brief to the author to paste.
82+
83+
## Responding to review
84+
85+
When Bugbot or a person comments on the PR:
86+
87+
1. **List what's still open:**
88+
```
89+
python3 .agents/skills/pre-pr-review/scripts/open_feedback.py <pr-number>
90+
```
91+
It prints unresolved, non-outdated review threads, review summaries and top-level comments, each with its author and location.
92+
2. **Verify each point against the current code.** A reviewer's wording, human or bot, is a claim to check, not a verdict.
93+
3. **Decide its outcome:**
94+
- `fixed in <sha>`
95+
- `already fixed`
96+
- `invalid`, with the evidence
97+
- `duplicate of <item>`
98+
- `needs a decision`: ask the author, and reply once they've decided
99+
4. **Check the fix before saying so.** Re-read the exact location in the fixed tree, and search for other instances of the same pattern, before you reply "fixed".
100+
5. **Reply on each thread with its outcome, and bring the brief up to date.** Fixes that change behaviour need a fresh review of the new commits and a rewritten brief, because the reviewer will rely on it. Trivial fixes need only a "Since the last brief" line. Anything that needs a decision goes to the author.
101+
102+
Ask the author before posting replies on the PR, unless they've already said you can.
Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
{
2+
"skill_name": "pre-pr-review",
3+
"notes": "Fill in any <placeholder> with a current PR before running, since PR state changes. Run each prompt with and without the skill, from a checkout of the PR's head commit. Compare the Reviewer Briefs against the expectations. Never let an eval run push, edit a PR description or post comments: use update_brief.py --dry-run.",
4+
"evals": [
5+
{
6+
"id": 1,
7+
"prompt": "I'm about to mark PR 777 (the OpenTelemetry spans one) ready for review. Run the pre-PR review on it and give me the Reviewer Brief, but don't touch the PR.",
8+
"expected_output": "A brief that a reviewer who never saw the review can act on.",
9+
"expectations": [
10+
"\"Start here\" names one file or function from the diff and says why it matters.",
11+
"No section lists bugs that were fixed during the review, finding counts, open questions or lens names outside the footer.",
12+
"\"Where it could fail\" has at most three items, each with a file and line, or a clearly named behaviour, and a concrete way it could go wrong.",
13+
"\"Not verified\" names live provider APIs or cluster mode.",
14+
"\"Beyond CI\" contains only checks CI does not run, or is omitted.",
15+
"The footer names the reviewed commit, the agent and model, and the lenses that ran.",
16+
"The visible brief is under about 200 words.",
17+
"Any question the review could not settle is put to the user before the brief is written, and none appears in the brief or elsewhere in the PR."
18+
]
19+
},
20+
{
21+
"id": 2,
22+
"prompt": "Quick one: I fixed a typo in a docstring in redisvl/query/filter.py and a broken link in docs/user_guide/how_to_guides/observability.md. Do the pre-PR review thing and write the brief.",
23+
"expected_output": "A very short brief that scales to the change.",
24+
"expectations": [
25+
"The brief is under about 60 words.",
26+
"Nothing is invented to fill \"Where it could fail\".",
27+
"The review does not raise findings about code the change did not touch."
28+
]
29+
},
30+
{
31+
"id": 3,
32+
"prompt": "Bugbot left comments on PR <number of an open PR with unresolved Bugbot threads>. Go through them using the repo's review process and tell me what you'd reply to each. Don't post anything.",
33+
"expected_output": "Each open point verified against the code and given exactly one outcome.",
34+
"expectations": [
35+
"The agent lists the open feedback with scripts/open_feedback.py or an equivalent that respects resolved and outdated threads.",
36+
"Each point gets exactly one of: fixed in <sha>, already fixed, invalid with evidence, duplicate, needs a decision.",
37+
"Points that are already resolved on the PR are not re-litigated.",
38+
"Nothing is posted to the PR."
39+
]
40+
}
41+
]
42+
}
Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# Challenger pass
2+
3+
The challenger's job is to remove false positives before they cost anyone time. It starts from the code and the list of findings, not from the reasoning that produced them. If your agent supports isolated sub-tasks, run the challenger as one, giving it only the diff and the findings list. If it doesn't, put the lens passes out of mind: re-open only the diff and the scratch file of findings, and judge each finding from the code alone.
4+
5+
For each finding:
6+
7+
1. Open the file at the cited line and read enough context to judge the claim.
8+
2. Try to build the failure scenario the finding describes. If you can, write it down as an input and the result, or as a short test.
9+
3. Look for anything that already prevents it: a guard elsewhere, a caller that never passes that input, a test that covers it.
10+
4. Score it.
11+
12+
## Scale
13+
14+
| Score | Meaning |
15+
|---|---|
16+
| 9–10 | Demonstrated: the failure is reproduced or follows directly from the code |
17+
| 7–8 | Very likely: the failure follows from the code as written, and nothing prevents it |
18+
| 4–6 | Plausible but unproven: it depends on inputs or states you couldn't confirm can occur |
19+
| 1–3 | Wrong, already handled, out of scope, or a style preference |
20+
21+
Keep findings that score 7 or more and have a file, a line and a concrete failure or cost. Reject the rest, each with a one-line reason in your scratch notes. If you reject a finding a reasonable person might side with, raise it with the author rather than settling it silently.
22+
23+
The challenger may also add a finding it noticed while checking. A finding it adds gets the same scoring, done in a fresh pass.
Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
# Correctness lens
2+
3+
Find changes that make the code behave wrongly for some input, state or environment the project supports. This lens runs on every PR.
4+
5+
## Look for
6+
7+
- **Line-level mistakes:** wrong or inverted conditions, off-by-one errors, a missing `await`, swallowed or over-broad exceptions, mutable default arguments, late-binding closures in loops.
8+
- **Removed behaviour:** for every deleted or rewritten line, name the invariant it enforced, such as a validation, an escape, a guard or a cleanup. Then find where the new code re-establishes it. If you can't find that place, report it.
9+
- **Broken contracts:** callers that rely on a changed return shape, a new exception, a new precondition or a changed default. Search for every caller, including tests, docs and notebooks.
10+
- **The sync or async twin:** a fix to `SearchIndex` that `AsyncSearchIndex` lacks, or the other way round. The same applies to every sync/async pair in `redisvl/extensions/` and to the async vectorizer and reranker methods.
11+
- **Wrappers and proxies:** a wrapper that drops arguments, ignores a return value, or forgets to close or release what it opened.
12+
- **Setup and teardown asymmetry:** something opened, locked or registered on one path and not released on another, especially on error paths.
13+
14+
## RedisVL specifics
15+
16+
- **Dependency floors.** Check every call into an optional dependency against the oldest version `pyproject.toml` allows. A new attribute or keyword that only exists in a later release crashes users on the floor version. This is the most common high-severity finding in this repository.
17+
- **Redis replies.** The code has to handle RESP2 and RESP3 reply shapes, and both hash and JSON storage.
18+
- **Result limits.** Default result counts (`num_results`, `LIMIT`, `MAXSEARCHRESULTS`) silently truncate. Check any read that is meant to return everything.
19+
- **Cluster mode.** Multi-key operations, `SCAN` and pipelines behave differently on a cluster, and partial failures are possible. CI never runs cluster tests, so reason about these cases from the code. If you can't settle one that way, put it under "Not verified" in the Reviewer Brief.
20+
- **Client ownership.** Who creates, owns and closes a Redis client, and what happens on reconnect, especially for async clients reused across event loops.
21+
22+
## Depth
23+
24+
For diffs under 100 lines, and for chore, CI or refactor changes, do a single pass over the diff and its direct callers. For anything larger, also trace each changed public function through its callers and its twin.
Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
# Design lens
2+
3+
Ask whether the change is a lasting solution at the right layer, or a patch that the next change will have to undo. In this repository, this lens more often produces a decision than a defect: for example, "remove the deprecated API instead of patching it".
4+
5+
## Look for
6+
7+
- **The wrong layer.** A special case bolted onto a caller when the problem lives in a shared builder, client or base class. Or a fix in one subclass that its siblings also need.
8+
- **Patching a footgun.** Effort spent making a deprecated or error-prone API safer, when finishing its deprecation or removing it would be simpler. Use `deprecated_argument` and `deprecated_function` from `redisvl.utils.utils` for the path out.
9+
- **Public API shape.** New or changed signatures, defaults and return types, and whether they stay consistent across sync and async classes and with sibling classes. One example is a new vectorizer that doesn't follow its siblings' constructor and method conventions.
10+
- **Compatibility.** Changes to public behaviour or defaults with no deprecation path. Compatibility branches left behind after a version floor was raised.
11+
- **New dependencies.** A new required dependency, or an optional one imported at module level instead of through `lazy_import`.
12+
- **Error semantics.** What the caller sees on partial failure, retry or reconnect, and whether that matches how the rest of the package behaves.
13+
14+
## How to report
15+
16+
Name the cost of leaving it as it is, such as "every new provider repeats this branch", and the smallest change that would remove that cost. If the better design is clearly out of scope, report it as an open item for the author to decide, not as something to fix in this PR.
17+
18+
Don't propose new abstractions for a single use. Three similar lines are cheaper than a premature helper.
Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Docs lens
2+
3+
Check that docstrings, comments, guides and notebooks say what the code actually does, and nothing more. Removing lines is a valid finding.
4+
5+
## Look for
6+
7+
- **Claims that contradict the code:** a docstring describing an old default, a comment explaining a branch that no longer exists, a guide calling a role "least privilege" when it grants more than it needs.
8+
- **Stale guides and notebooks:** pages under `docs/` and notebooks under `docs/user_guide/` that call a changed API, rely on a changed default, or show output that would now differ. Check every usage the diff makes stale, not just the files it touches.
9+
- **Bloat:** comments that restate the code, docstrings longer than the function they describe, and the same explanation repeated in several places when one should link to the other. A comment should explain why, or warn about something non-obvious.
10+
- **Missing docs for new public API:** a new public class, function or parameter with no docstring, or with a docstring that doesn't state its defaults and what it raises.
11+
- **Sphinx breakage:** wrong `currentmodule` or cross-reference targets, and new modules missing from `docs/api/`.
12+
- **The PR description:** check that it names the change's most consequential behaviour change. If public behaviour changes, check that the description has a Release Notes line a user could act on.
13+
14+
## How to report
15+
16+
Quote the claim that is wrong, or the lines that should go, in a few words, and give the file and line. Don't rewrite prose for style. Report only text that misleads, is out of date, or wastes the reader's time.

0 commit comments

Comments
 (0)