Skip to content

Commit 6f2ab56

Browse files
Jefffreyalambclaudembutrovich
authored andcommitted
Update contributor guidelines regarding AI spam, reviews, code (apache#24510)
From the discussion here: - apache#24413 Updating our contributing guidelines, specifically to target cases where we see spam PRs from AI origins especially from new contributors. Aiming to set some rules/guidelines around this, and also hopefully make any agents involved at least reconsider before spamming - closes apache#24413 - related to apache#18095 - closes apache#25317 --------- Co-authored-by: Andrew Lamb <[email protected]> Co-authored-by: Claude Fable 5.1 <[email protected]> Co-authored-by: Matt Butrovich <[email protected]>
1 parent b2366a2 commit 6f2ab56

5 files changed

Lines changed: 160 additions & 39 deletions

File tree

‎AGENTS.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,12 +4,19 @@
44

55
- [Quick Start Setup](docs/source/contributor-guide/development_environment.md#quick-start)
66
- [Testing Quick Start](docs/source/contributor-guide/testing.md#testing-quick-start)
7+
- [Open Contribution and Assigning tickets](docs/source/contributor-guide/index.md#open-contribution-and-assigning-tickets)
78
- [Choosing What Kind of Test to Write](docs/source/contributor-guide/testing.md#choosing-what-kind-of-test-to-write)
89
- [Before Submitting a PR](docs/source/contributor-guide/index.md#before-submitting-a-pr)
910
- [Reviewing Pull Requests](docs/source/contributor-guide/pr_review.md)
1011
- [Contributor Guide](docs/source/contributor-guide/index.md)
1112
- [Architecture Guide](docs/source/contributor-guide/architecture.md)
1213

14+
## Before Starting Work
15+
16+
Before you start work on an issue, you MUST follow the instructions in
17+
[Open Contribution and Assigning tickets](docs/source/contributor-guide/index.md#open-contribution-and-assigning-tickets). You must ensure duplicate work is not being created.
18+
Do not create multiple PRs if this is your first time contributing.
19+
1320
## Before Committing
1421

1522
Before committing any changes, you MUST follow the instructions in
Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
<!---
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
10+
http://www.apache.org/licenses/LICENSE-2.0
11+
12+
Unless required by applicable law or agreed to in writing,
13+
software distributed under the License is distributed on an
14+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
KIND, either express or implied. See the License for the
16+
specific language governing permissions and limitations
17+
under the License.
18+
-->
19+
20+
# AI Policy
21+
22+
DataFusion has the following policy for AI-assisted PRs:
23+
24+
- We welcome AI-assisted PRs from anyone. We do not welcome unreviewed "AI
25+
dumps" (defined below).
26+
- The PR author should have personally read the entire PR they submit and
27+
**understand the core ideas end-to-end**. Authors should be ready to justify
28+
and help reviewers understand the design and code during review.
29+
- **Call out unknowns and assumptions**. It's okay to not fully understand
30+
some bits of AI-generated code. Please point these cases out so we can work
31+
together to clear up any concerns.
32+
- **Follow [ASF Generative Tooling Guidance]**. Among other things,
33+
the author must ensure the PR does not include undisclosed third-party
34+
code or copyrighted material.
35+
36+
[asf generative tooling guidance]: https://www.apache.org/legal/generative-tooling.html
37+
38+
While "understand the core ideas" is partly subjective, it means more than being
39+
able to follow the diff textually. We expect the PR author to take an active
40+
role in responding to feedback and crafting the PR to make sure it fits well
41+
into the project as a whole. The worst situation is one where the reviewer ends
42+
up simply driving the contributor's LLM, for the reasons described in the next
43+
section.
44+
45+
## What is an "AI dump" and why it is not helpful
46+
47+
An "AI dump" is a PR, or a series of PRs, consisting largely of AI-generated
48+
code and descriptions that the author has not reviewed and does not understand.
49+
The code may even be correct. The problem is that all the work of understanding
50+
falls on the reviewer.
51+
52+
Code review serves two purposes:
53+
54+
1. Finishing the intended task.
55+
2. Sharing knowledge between authors and reviewers, as a long-term investment in
56+
the project. For this reason, even if someone familiar with the codebase
57+
could finish a task more quickly by themselves, we are still happy to help
58+
a new contributor work on it.
59+
60+
An AI dump meets neither purpose. Maintainers could finish the task faster by
61+
running the AI tool themselves, and an author who acts only as a pass-through
62+
proxy for the tool learns little from the review.
63+
64+
Reviewing capacity for the project is **very limited**, so PRs that appear to be
65+
AI dumps may not get reviewed, and may eventually be closed.
66+
67+
Multiple PRs created in a short amount of time, especially by a first-time
68+
contributor, that in our judgment show a lack of understanding or author
69+
engagement may be treated as spam and closed. One high-quality PR that you work
70+
with maintainers to merge is far more valuable to you and the project than ten
71+
PRs you have your agent generate and submit for you.
72+
73+
## Responding to review comments
74+
75+
The same policy applies to review discussion as to the code itself: reviewers
76+
want to talk to **you**, not to your AI tool. Please do not respond to reviewer
77+
comments with unreviewed, fully AI-generated responses, as this again puts all
78+
the burden on the reviewer and you learn nothing. We expect your responses to be
79+
in your own words, though it is fine to use AI to help prepare your response
80+
(for example to understand the comment, or help translate your response to
81+
English). Examples of unreviewed AI-generated replies:
82+
83+
- Summarizes the diff rather than answering the question that was asked.
84+
- Lists the commands run locally (e.g. `cargo fmt`, `cargo test`)
85+
and whether they passed. This is not useful to reviewers because CI already runs
86+
these checks.
87+
- Contains statements that don't make sense in context, such as claiming
88+
tests could not be run because `cargo` is not installed.
89+
90+
For example, see [this review thread in arrow-rs][arrow-rs-review-example] where
91+
the reviewer asked a design question, and received several replies that
92+
described what had changed and which commands had been run, rather than an
93+
answer to the question.
94+
95+
Again, the point of code review is to help the project **and** to help you grow
96+
as an engineer, so please read each comment, make sure you understand it, and
97+
reply in your own words.
98+
99+
[arrow-rs-review-example]: https://github.com/apache/arrow-rs/pull/11209#discussion_r4145512124
100+
101+
## AI-assisted reviews
102+
103+
The same standard applies to AI-generated reviews as to AI-generated code: you
104+
should have read and understood anything you post. Raw AI review output is often
105+
verbose with unnecessary details, and it can take substantial effort to figure
106+
out what is actually being asked.
107+
108+
If you review PRs with the help of an AI tool, read all comments first, remove
109+
detail that is unnecessary or you don't understand, and explain the rest in your
110+
own words so that each comment makes a clear, specific request.
111+
112+
## Better ways to contribute than an "AI dump"
113+
114+
It's recommended to write a high-quality issue with a clear problem statement
115+
and a minimal, reproducible example. The reproducer should focus on the
116+
end-user-visible behavior rather than explaining the details of some code
117+
defect.
118+
119+
This will make it easier for others to contribute and for reviewers to
120+
understand the problem being addressed.

‎docs/source/contributor-guide/index.md‎

Lines changed: 26 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -54,24 +54,29 @@ Contributors drive the project forward based on their own priorities and
5454
interests and thus you are free to work on any issue that interests you.
5555

5656
If someone is already working on an issue that you want or need but hasn't
57-
been able to finish it yet, you should feel free to work on it as well. In
58-
general it is both polite and will help avoid unnecessary duplication of work if
59-
you leave a note on an issue when you start working on it.
57+
been able to finish it yet, feel free to help them out.
6058

61-
If you want to work on an issue which is not already assigned to someone else
62-
and there are no comment indicating that someone is already working on that
63-
issue then you can assign the issue to yourself by submitting a single word
64-
comment `take`. This will assign the issue to yourself. However, if you are
65-
unable to make progress you should unassign the issue by commenting a single
66-
word `untake`.
59+
If there is an existing open PR for an issue you plan to work on, please review that
60+
PR before opening a new one. Duplicate, unacknowledged PRs consume valuable
61+
reviewer time and we may close them. If there is an existing PR, please identify
62+
it in the PR description and explain why you are opening a new one and not
63+
helping with the previous one. In general it is both polite and will help avoid
64+
unnecessary duplication of work if you also leave a note on an issue when you
65+
start working on it.
66+
67+
If you want to work on an issue which is not already assigned to someone and has
68+
no comment indicating someone is already working on it, you can assign the issue
69+
to yourself by submitting a single word comment `take`. However, if you are unable
70+
to make progress please unassign the issue by commenting a single word `untake`.
6771

6872
# Developer's guide
6973

7074
## Pull Request Overview
7175

7276
We welcome pull requests (PRs) from anyone in the community.
7377

74-
DataFusion is a rapidly evolving project and we try to review and merge PRs quickly.
78+
DataFusion is a rapidly evolving project and we try to review and merge PRs
79+
quickly.
7580

7681
Review bandwidth is currently our most limited resource, and we highly encourage reviews by the broader community. If you are waiting for your PR to be reviewed, consider helping review other PRs that are waiting. Such review both helps the reviewer to learn the codebase and become more expert, as well as helps identify issues in the PR (such as lack of test coverage), that can be addressed and make future reviews faster and more efficient.
7782

@@ -100,7 +105,7 @@ When possible, we recommend splitting your contributions into multiple smaller f
100105

101106
1. The PR is more likely to be reviewed quickly -- our reviewers struggle to find the contiguous time needed to review large PRs.
102107
2. The PR discussions tend to be more focused and less likely to get lost among several different threads.
103-
3. It is often easier to accept and act on feedback when it comes early on in a small change, before a particular approach has been polished too much.
108+
3. It is often easier to accept and act on feedback when it comes early in a small change, before a particular approach has been polished too much.
104109

105110
If you are concerned that a larger design will be lost in a string of small PRs, creating a large draft PR that shows how they all work together can help.
106111

@@ -134,6 +139,12 @@ Please ensure your PR follows the [testing guide](testing.md). In particular:
134139
[Choosing What Kind of Test to Write](testing.md#choosing-what-kind-of-test-to-write).
135140
- Run any relevant commands from the [testing quick start](testing.md#testing-quick-start).
136141

142+
## AI-Assisted Contributions
143+
144+
We welcome AI-assisted PRs, but not unreviewed "AI dumps". See the
145+
[AI Policy](ai-policy.md) page for what we expect from authors and reviewers
146+
who use AI tools.
147+
137148
## Conventional Commits & Labeling PRs
138149

139150
We generate change logs for each release using an automated process that will categorize PRs based on the title
@@ -197,33 +208,9 @@ The good thing about open code and open development is that any issues in one ch
197208
Pull requests will be marked with a `stale` label after 60 days of inactivity and then closed 7 days after that.
198209
Commenting on the PR will remove the `stale` label.
199210

200-
## AI-Assisted contributions
201-
202-
DataFusion has the following policy for AI-assisted PRs:
203-
204-
- The PR author should **understand the core ideas** behind the implementation **end-to-end**, and be able to justify the design and code during review.
205-
- **Calls out unknowns and assumptions**. It's okay to not fully understand some bits of AI generated code. You should comment on these cases and point them out to reviewers so that they can use their knowledge of the codebase to clear up any concerns. For example, you might comment "calling this function here seems to work but I'm not familiar with how it works internally, I wonder if there's a race condition if it is called concurrently".
206-
207-
### Why fully AI-generated PRs without understanding are not helpful
208-
209-
Today, AI tools cannot reliably make complex changes to DataFusion on their own, which is why we rely on pull requests and code review.
210-
211-
The purposes of code review are:
212-
213-
1. Finish the intended task.
214-
2. Share knowledge between authors and reviewers, as a long-term investment in the project. For this reason, even if someone familiar with the codebase can finish a task quickly, we're still happy to help a new contributor work on it even if it takes longer.
215-
216-
An AI dump for an issue doesn’t meet these purposes. Maintainers could finish the task faster by using AI directly, and the submitters gain little knowledge if they act only as a pass through AI proxy without understanding.
217-
218-
Please understand the reviewing capacity is **very limited** for the project, so large PRs which appear to not have the requisite understanding might not get reviewed, and eventually closed or redirected.
219-
220-
### Better ways to contribute than an “AI dump”
221-
222-
It's recommended to write a high-quality issue with a clear problem statement and a minimal, reproducible example. This can make it easier for others to contribute.
223-
224-
### CI Runners
211+
## CI Runners
225212

226-
#### Runs-On
213+
### Runs-On
227214

228215
We use [Runs-On](https://runs-on.com/) for some actions in the main repository, which run in the ASF AWS account to speed up CI. In forks, these actions run on the default GitHub runners since forks do not have access to ASF infrastructure.
229216

@@ -239,7 +226,7 @@ For those actions we also use the [Runs-On action](https://runs-on.com/caching/m
239226

240227
For the standard GitHub runners, this action will do nothing.
241228

242-
##### Spot Instances
229+
#### Spot Instances
243230

244231
By default, Runs-On actions run as [spot instances](https://runs-on.com/configuration/spot-instances/), which means they might occasionally be interrupted. In the CI you would see:
245232

@@ -249,6 +236,6 @@ Error: The operation was canceled.
249236

250237
According to Runs-On, spot instance termination is extremely rare for instances running for less than 1h. Those actions will be restarted automatically.
251238

252-
#### GitHub Runners
239+
### GitHub Runners
253240

254241
We also use standard GitHub runners for some actions in the main repository; these are also runnable in forks.

‎docs/source/contributor-guide/pr_review.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -234,6 +234,12 @@ Rather than a bare "LGTM", say what you actually checked ("traced the state
234234
transitions by hand", "confirmed the hasher change cannot affect ordering")
235235
so it is clear what was verified and what was not.
236236

237+
### Use AI Review Tools Thoughtfully
238+
239+
AI tools can help you review, but please read and edit their output before
240+
posting it so each comment is a clear, specific request in your own words. See
241+
[AI-assisted reviews](ai-policy.md#ai-assisted-reviews) in the contributor guide.
242+
237243
### Invite Additional Committers on Core Changes
238244

239245
For changes to core, widely shared code, leave the PR open for other

‎docs/source/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -159,6 +159,7 @@ To get started, see
159159
:caption: Contributor Guide
160160

161161
contributor-guide/index
162+
contributor-guide/ai-policy
162163
contributor-guide/pr_review
163164
contributor-guide/communication
164165
contributor-guide/development_environment

0 commit comments

Comments
 (0)