Repository navigation
Require disclosure of genAI usage in ai policy - #538
Conversation
pllim
left a comment
pllim
left a comment
There was a problem hiding this comment.
I think this is reasonable request, but whether people/bot would comply or not is a different story.
@astropy/coordinators , could you please add this to your agenda?
Thanks, all!
stargaser
left a comment
stargaser
left a comment
There was a problem hiding this comment.
I think this is a good addition to the policy.
I don't think we should care about comply at this point, this would serve as a clear reason to close PRs. |
|
LGTM |
|
Was going to post more or less the exact same thing as #538 (comment). Strong +1 there. |
Co-authored-by: Leo Singer <[email protected]>
| 3. Consistency with Astropy conventions and existing style. In the context of Generative AI, this particularly means conveying information (comments, documentation, etc) in a style the Astropy community uses, and that is appropriate and to the point. | ||
| 3. **Licensing:** Contributors are responsible for ensuring all submitted code complies with Astropy's licensing, regardless of whether generative AI tools were used. | ||
| 2. **Disclosure:** Contributors must disclose whether they leveraged generative AI tools when making changes (including autonomous agents). This also helps downstream citations comply with academic conference policies that may require disclosure of genAI usage. | ||
| 3. **Authentic engagement:** The pull request process is collaborative and iterative. Contributors are expected to actively engage with reviewer feedback themselves. Copying and pasting replies to / from a generative AI tool does not count as engaging with the reviewer; contributors are expected to understand the changes they are proposing, not act as message courier for a third party. |
There was a problem hiding this comment.
I think this needs some polishing :). "not act as message courier for a third party." 😆
There was a problem hiding this comment.
I think one of those LinkedIn profiles call themselves "agent sitters".
There was a problem hiding this comment.
"contributors are expected to understand the changes they are proposing"
is repetitive with "Contributors must be able to explain all changes during review."
| 3. **Authentic engagement:** The pull request process is collaborative and iterative. Contributors are expected to actively engage with reviewer feedback themselves. Copying and pasting replies to / from a generative AI tool does not count as engaging with the reviewer; contributors are expected to understand the changes they are proposing, not act as message courier for a third party. | |
| 3. **Authentic engagement:** The pull request process is collaborative and iterative. Contributors are expected to actively engage with reviewer feedback themselves. Copying and pasting replies to / from a generative AI tool does not count as engaging with the reviewer. |
The rest is already covered in text
There was a problem hiding this comment.
It looks like we already had this in text, so it's fine to keep. But also, maybe good to polish.
There was a problem hiding this comment.
Do keep this in, especially about the third party part.
There was a problem hiding this comment.
I think based on the comment I'm about tot leave below re: CoCo discussions it's probably better to not make that change yet, but maybe in the follow-on PR (i.e., this is more about "philosophy" than policy, which we want to split up)
|
"Leveraging generative AI tools when making changes" is vague and subject to widely different interpretations. Do these all require clicking the box?
I don't care about clicking the box for every PR I put in, but I'm not sure what the project or reviewers are learning. Is the intent here really a more specific question like "Were most of the PR code changes directly created by a GenAI tool?" |
that was my understanding at least, with the original AI policy PR. Maybe something like "did a genAI tool create a significant portion of these changes?" or "were these changes generated via prompt?" |
I still am having trouble seeing the value in a Yes/No checkbox response to a question that is inherently subjective, in particular what does "significant portion" mean? If I got AI suggestions on 2 lines of a 10 line PR and then it wrote half the tests and the changelog (but then I edited it a bunch), does that count? And again, how is that Yes/No actionable? What about instead requiring contributors to answer the expository question "Describe your use of genAI in this PR" (with some additional guidance). |
| 2. Authentic engagement. The pull request process is collaborative and iterative. Contributors are expected to actively engage with reviewer feedback themselves. Copying and pasting replies to / from a generative AI tool does not count as engaging with the reviewer; contributors are expected to understand the changes they are proposing, not act as message courier for a third party. | ||
| 3. Consistency with Astropy conventions and existing style. In the context of Generative AI, this particularly means conveying information (comments, documentation, etc) in a style the Astropy community uses, and that is appropriate and to the point. | ||
| 3. **Licensing:** Contributors are responsible for ensuring all submitted code complies with Astropy's licensing, regardless of whether generative AI tools were used. | ||
| 2. **Disclosure:** Contributors must disclose whether they leveraged generative AI tools when making changes (including autonomous agents). This also helps downstream citations comply with academic conference policies that may require disclosure of genAI usage. |
There was a problem hiding this comment.
This also helps downstream citations comply with academic conference policies that may require disclosure of genAI usage.
I don't have a clue what that means.
There was a problem hiding this comment.
if academic publishing requires disclosure of AI usage, I would think that also means people using astropy should be aware if a module or new version they're using has AI-generated code right? But maybe this isn't really the place to state that, I can remove it if desired:
| 2. **Disclosure:** Contributors must disclose whether they leveraged generative AI tools when making changes (including autonomous agents). This also helps downstream citations comply with academic conference policies that may require disclosure of genAI usage. | |
| 2. **Disclosure:** Contributors must disclose whether they leveraged generative AI tools when making changes (including autonomous agents). |
There was a problem hiding this comment.
if academic publishing requires disclosure of AI usage, I would think that also means people using astropy should be aware if a module or new version they're using has AI-generated code right?
It is important to remember that AI-generated code is not a bright shiny line in the sand. Many developers use this to varying degrees, and there is already no way to say which subpackages in astropy have genAI code. I believe that in a matter of years, most astropy subpackages that are being actively maintained will have such code. So to my mind, we can already just make a blanket statement that astropy contains AI-generated code and folks can use that information as needed.
There was a problem hiding this comment.
I believe that in a matter of years, most astropy subpackages that are being actively maintained will have such code.
I'd rather that be a deliberate choice than a vague sentiment. Of course I'd personally prefer we take the opposite choice.
| 3. Contributers are responsible for ensuring all submitted code complies with Astropy's licensing, regardless of whether generative AI tools were used. | ||
| 2. Authentic engagement. The pull request process is collaborative and iterative. Contributors are expected to actively engage with reviewer feedback themselves. Copying and pasting replies to / from a generative AI tool does not count as engaging with the reviewer; contributors are expected to understand the changes they are proposing, not act as message courier for a third party. | ||
| 3. Consistency with Astropy conventions and existing style. In the context of Generative AI, this particularly means conveying information (comments, documentation, etc) in a style the Astropy community uses, and that is appropriate and to the point. | ||
| 3. **Licensing:** Contributors are responsible for ensuring all submitted code complies with Astropy's licensing, regardless of whether generative AI tools were used. |
There was a problem hiding this comment.
This isn't actually a change in this PR, but this line seems unenforceable to me. If I submit a PR with a new function and tests that were AI generated, how exactly do I "ensure that all the code complies with Astropy's licensing"?
There was a problem hiding this comment.
I'm tempted to conclude that use of genAI is intrinsically incompatible with the ethics of FOSS, and probably copyright law too 🤔
There was a problem hiding this comment.
There was a problem hiding this comment.
Linus isn't the Pope of FOSS, as far a I'm aware. And he seems widely disconnected with the actual arguments against genAI, so I'm not feeling to bad that I disagree with him.
There was a problem hiding this comment.
I'm not sure there's any Pope of FOSS, but continuing on this metaphor, there are definitely bishops, and archbishops given major FOSS projects.
I don't think we should be disallowing the next set of demonstrably useful tools.
We should make sure the work for us, and not us work for them.
There was a problem hiding this comment.
What do you mean by "social warrior project" here? I'm also unclear what Linus meant by it. As far as I understand, Astropy, and open source in general, is based on ideals of community, collaboration, and the idea that software should be accessible and transparent, and that sounds a lot like a social stance to me. Personal convictions contribute to community sentiment, because persons make up the community; I don't think it's productive to dismiss that. In the end this software doesn't exist in a vacuum, it's made by and for people.
Putting aside the ethical and environmental issues however and focusing more on this PR, I still don't personally think entirely-LLM-generated pull requests are a net positive for any software product. LLM coding assistants, by their very nature, are not able to build abstract architectures without reflecting or mashing already-existing architecture from different projects with different goals. They can't plan ahead because they can't plan, or think in the first place. Without a human helming the change and PR, most of what I've seen from fully-autonomous PRs is band-aid solutions that don't account for future changes, and tailor-made modules that aren't maintainable, and all that turns into tech debt that a human has to deal with later. Not to mention, as I brought up in another comment, I've seen LLMs state with utter confidence that they've taken care of some problem you bring up for them to fix, and then you check the actual code and they did nothing of the sort, or they added a test that passes green without actually covering the edge case it says it does, or the conditionals are so specific to fit your request that it fails to be useful for general cases.
All of this checking and validation and not trusting the output is an additional mental load on reviewers, and I personally am of the conviction that purely-AI-generated pull requests are not worth it, which is why I added this note: #538 (review)
There was a problem hiding this comment.
but again, I myself am not a regular reviewer of astropy, mostly just a downstream user. I don't mean to speak over anyone else's views.
There was a problem hiding this comment.
I 100% adhere to @zacharyburnett's point. I'll just add that collaborative work is a social and political activity, especially in FOSS, and that politics cannot be waved away from it.
There was a problem hiding this comment.
Howdy, I'm a maintainer over in JuliaAstro that happened to see this thread. I can only speak for myself, but as a person whose life is political by virtue of being a minority here, I just wanted to say that I deeply appreciate the thoughtful discourse being had by the astropy team
There was a problem hiding this comment.
FWIW, I tend to agree with @zacharyburnett's perspective that everything we do has a social and community impact. Astropy is a community project, and it is right and proper for us to consider the large-scale impacts of the things we do.
That said, to me this is an area where we are very far from community consensus. @neutrinoceros has a very strongly-held view, that he has every right to have, and to advocate for that view in project discussions like this. But I think this thread shows that plenty of other members of the community do not share that stance. I for one, am one of them: I am very suspicious of both the questionable promises AI companies make, and have serious concerns of some of the knock-on effects (not the same as @neutrinoceros, FWIW, but different concerns to be discussed elsewhere)... but I also see very real benefit for some AI applications that seem like a net good. Others in this and other discussion threads lie in other parts of that spectrum.
So how I am going to choose to interpret @taldcroft's comment (but can be corrected if that's mis-understanding!) is that it is not fair for one member (or even a handful) to have their views reflected in this policy if we have not reached community consensus.
On this specific point, though, I tend to agree with @taldcroft that this is not enforceable, but I also think it should still be in here for now... but see my comment at the end about the pathway the CoCo wants to propose to address that.
taldcroft
left a comment
taldcroft
left a comment
There was a problem hiding this comment.
Within the narrow scope the original PR objective of adding a policy requiring disclosure of AI usage, this looks good to me.
Co-authored-by: Nathaniel Starkman <[email protected]>
There was a problem hiding this comment.
Thanks for bringing this up, @zacharyburnett! No surprises that this policy needs ongoing refinement but I for one am really happy to see constructive suggestions in that direction.
That said, since this topic is clearly struggling to come to consensus on some areas (and we were prompted by @pllim), the CoCo discussed this today, and here is how we propose to proceed:
- We leave this PR open for another week, over which time @zacharyburnett can make decisions on wordsmithing, etc based on some of the discussion threads here, but not some of the more major changes proposed by some in the comments.
- After that, a CoCo member will merge this - the committee agreed this is a net improvement even if some of the discussion reveals different perspectives on some of the details (and indeed the big picture of the policy as a whole).
- Once this is merged, @kbwestfall will start another PR with a goal of refining this policy into a two-part document. The first part will by as clear as we can make of a concrete policy. The second part will be something like "why/philosophy behind this policy". That will give a space for us to clearly state some of the parts we all agree on: that Astropy depends on real human engagement and is for and by the community, not for or by robots. But we can at the same time emphasize that the community is not necessarily in consensus on the bigger-picture concerns about AI and its use for FOSS, and that the policy is meant to make sure there is space for people in various camps to engage even if they don't necessarily like all of it. That will be a place to make sure everyone feels their perspective is represented even if the community as a whole doesn't feel the policy should go that far/less far/etc.
| 3. Contributers are responsible for ensuring all submitted code complies with Astropy's licensing, regardless of whether generative AI tools were used. | ||
| 2. Authentic engagement. The pull request process is collaborative and iterative. Contributors are expected to actively engage with reviewer feedback themselves. Copying and pasting replies to / from a generative AI tool does not count as engaging with the reviewer; contributors are expected to understand the changes they are proposing, not act as message courier for a third party. | ||
| 3. Consistency with Astropy conventions and existing style. In the context of Generative AI, this particularly means conveying information (comments, documentation, etc) in a style the Astropy community uses, and that is appropriate and to the point. | ||
| 3. **Licensing:** Contributors are responsible for ensuring all submitted code complies with Astropy's licensing, regardless of whether generative AI tools were used. |
There was a problem hiding this comment.
FWIW, I tend to agree with @zacharyburnett's perspective that everything we do has a social and community impact. Astropy is a community project, and it is right and proper for us to consider the large-scale impacts of the things we do.
That said, to me this is an area where we are very far from community consensus. @neutrinoceros has a very strongly-held view, that he has every right to have, and to advocate for that view in project discussions like this. But I think this thread shows that plenty of other members of the community do not share that stance. I for one, am one of them: I am very suspicious of both the questionable promises AI companies make, and have serious concerns of some of the knock-on effects (not the same as @neutrinoceros, FWIW, but different concerns to be discussed elsewhere)... but I also see very real benefit for some AI applications that seem like a net good. Others in this and other discussion threads lie in other parts of that spectrum.
So how I am going to choose to interpret @taldcroft's comment (but can be corrected if that's mis-understanding!) is that it is not fair for one member (or even a handful) to have their views reflected in this policy if we have not reached community consensus.
On this specific point, though, I tend to agree with @taldcroft that this is not enforceable, but I also think it should still be in here for now... but see my comment at the end about the pathway the CoCo wants to propose to address that.
| 3. Consistency with Astropy conventions and existing style. In the context of Generative AI, this particularly means conveying information (comments, documentation, etc) in a style the Astropy community uses, and that is appropriate and to the point. | ||
| 3. **Licensing:** Contributors are responsible for ensuring all submitted code complies with Astropy's licensing, regardless of whether generative AI tools were used. | ||
| 2. **Disclosure:** Contributors must disclose whether they leveraged generative AI tools when making changes (including autonomous agents). This also helps downstream citations comply with academic conference policies that may require disclosure of genAI usage. | ||
| 3. **Authentic engagement:** The pull request process is collaborative and iterative. Contributors are expected to actively engage with reviewer feedback themselves. Copying and pasting replies to / from a generative AI tool does not count as engaging with the reviewer; contributors are expected to understand the changes they are proposing, not act as message courier for a third party. |
There was a problem hiding this comment.
I think based on the comment I'm about tot leave below re: CoCo discussions it's probably better to not make that change yet, but maybe in the follow-on PR (i.e., this is more about "philosophy" than policy, which we want to split up)
|
Ack some of the above came up a bit unintentionally intermingled - my inline comments were meant to come as me just Erik, while the review comment is the summary of the Coordination Committee discussion/decision. That said, one additional thing I wanted to mention mainly to @taldcroft @zacharyburnett @neutrinoceros (as people in the now-collapsed/outdated thread about academic policies about disclosure): I am on a committee that oversees the AAS Journals policy on AI use, and at least in that case the policy about disclosure is intended to be about disclosure of AI directly used by a researcher. That is, in our discussion on that topic, no one expected paper authors to go look up the AI use status of every single piece of software they use and have that in their disclosure statement. To be clear, I myself am actually quite comfortable with an expectation of AI disclosure (although it falls in the category of "impossible to enforce", I think that doesn't mean it's not worth stating it's an expectation). But I don't think it really needs to be intertwined with the publication/presentation policies on this topic. |
|
Thanks @eteq and the CoCo for putting forward a path to getting this PR merged. It may well be that AI is a topic that exposes irreconcilable differences in our community. This is a good thing to recognize and I look forward to the "philosophy" document that outlines a way that we can continue to effectively work together even with those differences. |
|
With the thought in mind of an astropy-specific addendum to the policy document, I do wonder if there would be any support for adopting the NumPy AI-policy verbatim. It says essentially the same thing as our policy, but with a format and wording that I find to be far more direct and understandable. I also think there is value in consolidating on a common policy. |
|
I've been following this discussion silently for the last month. I'm chiming in now because Larry's case is parallel to this one (astropy/astroquery#3641 - same bot), which triggered me because another bot came in with the same PR a few days later (astropy/astroquery#3642). These are minor typos that a human is not likely to see even if they look ( |
|
In my case, I was suspicious because the docstring fix was in a private function, which of course appears nowhere in the rendered docs. |
|
Just for another perspective. In my other packages, the way I've been engaging with AI content is in how much work it puts on me. If it's going to take a lot of time fixing / asking the (AI?) contributor to fix things then I want a human on the other end. Particularly if it's a big change that I want to understand. |
|
That docstring bot was more than annoying, it also showed up on pyvo. And overall we interacted with it way more than we should have and it's draining. From the first look at the first PR it was absolutely clear it's being a bot rather than a human who has even the slightest interest of using these libraries. (I mean running numpydoc in CI is on the wishlist for a while, but it's low priority (as well as in many cases for astroquery/pyvo I don't want enforcing that extreme level of nitpicking). |
That is true, but again, is this really a high enough priority and should we really set our priorities as dictated by AI bots? (in this case, fixing up typos in private function docstrings, really) |
|
@bsipocz - the astroquery PR(s) fixed the docstrings for a number of public functions, e.g. see the typo here: https://astroquery.readthedocs.io/en/latest/api/astroquery.ipac.irsa.IrsaClass.html#astroquery.ipac.irsa.IrsaClass.query_ssa I'm with @nstarman on looking at effort. If a PR comes in that is quick to look at and does something limited in scope like fixing obvious docstring errors, I personally would just merge it rather than spend time fussing about the provenance. But to be clear, I recognize we all can have different metrics for progress and different tolerance for how AI gets used. I'm personally quite excited about the opportunity to just fix the documentation in my packages with very little effort. See astropy/astropy#20217 that fixes real mistakes in the public API docs and many other problems in semi-public/private (but still important to devs) functions/methods. Just out of curiosity, would there be pushback if I used some of my free Claude Max fixing documentation errors in the rest of astropy (on a per-subpackage basis)? |
|
.. And we really should just get this PR merged so we have a published policy about up front AI disclosure! I think we are all in strong agreement on that point. |
|
@taldcroft no, if you use claude, declare it, and take ownership of the PRs, that's great. Autobots going around and looking for trivial problems is not great. The documentation mistakes in astroquery are real but harmless, and reviewer attention is finite. Verifying the correctness of even a simple PR is expensive. (also, hey! i didn't get my free claudes! dangit!) |
I can review for cosmo and units. But to avoid stepping on another docs PR maybe hold off on cosmo for a few days. |
|
See the entirely-human-generated PR to update the PR template: astropy/astropy#20220. This is meant to provide a concrete idea of what implementing this astropy project PR #538 might look like. |
|
What Adam said above. Yes, there are harmless typos, but I have zero interest of looking into the diff and merging bot PRs for fixing up those. Getting a numpydoc job and fixing up everything in one go is a different beast, but again, it's a very low priority thing and I am not interested in reviewing any PRs for that that is not coming from maintainers -- we have enough actually improving PRs in the queue and should focus on helping those along, yet getting side tracked with these uninvited ones. At this point, I would even go as far as closing down the repo for unvetted new contributors. Anyone new from the archives could ping us to get added to the team first, and should cut off the bots and non-user, zero stake in the game outsiders who only soak up human resources. |
|
I have been thinking something similar about not actually accepting random contributors anymore but having a contributor sign up process to at least have some kind of barrier. I've come to realise what I dislike is having random people who don't actually care about the project contributing one off fixes. They are just trying to get something merged but as they are not actually users of astropy they don't have a vested interest to make sure it's good or useful. |
|
I found this interesting - https://www.softwareseni.com/how-ai-generated-contributions-are-reshaping-open-source-supply-chain-risk/ I actually worry about review capacity even for PRs from within the team. From my experience so far with Claude Opus, I can generate PRs that I'm happy with very quickly. For time and table there is basically just @mhvk... |
|
Yes I mean we were reviewer limited before LLMs came along 😅 |
|
Writing/editing code was never the bottleneck. If anything, LLMs are amplifying an existing asymmetry. And before anyone suggests it: I personally have no interest in having my work reviewed by LLMs (and I don't want to review LLM generated code) |
|
The conversation is very constructive and I don't want to close it down... but we also don't want this to get in the way of positive changes to the policy that this PR represents. @zacharyburnett , can you clarify if you had any more wordsmithing-style changes you wanted to make in the essential elements already in this PR? If not, I'll merge it now, and we can continue this discussion in the follow-on issue about larger changes to the policy. |
While I have lots of thoughts about this idea (both for and against...), I think it's a separate topic worthy of its own discussion distinct from AI policy. So to help organize the discussion, I'll try to make a distinct issue for that idea once we've gotten this PR merged. It probably also needs discussion at coordination meeting and/or APE, but this would at least be the first step. |
|
Sorry to keep abusing this PR for generic AI-in-astropy discussion, but this happens to be where the discussion is happening now. With that, welcome to one vision of the future: astropy/astropy#20211. This is a substantive, API-changing PR in coordinates that is almost entirely AI-generated. I spent a lot of time on it, trying to take care that Claude had the necessary skills to make non-trivial changes within a complex code base. That said, I only mostly understand all the changes. My hope is that the extensive documentation (far far beyond any human generated PR would ever include) makes it straightforward for the coordinates maintainers to review. But honestly this is an experiment and I welcome feedback. I would never have even considered doing this change myself. So basically I used AI to fix some real and long-standing issues in coordinates. It won't be trivial to review. What will happen?? 😄 |
|
My prediction is that given the amount of text and code that has to be read by a human, it will likely languish for quite a while! On a serious note though, I think this is a good example where the changes should definitely be split into smaller PRs, and also trying to get Claude to be more concise. Or we will accentuate this writing vs reviewing asymmetry further. Claude and other LLMs can be told to be more concise, separate changes into several logical branches, and use short commit messages, all to match how we usually do things. |
We can certainly go back to the good old days where I dumped a big PR that fundamentally changed coordinates, and provided exactly 4 sentences to describe what was going on. I can easily generate a concise management summary (basically the What's New), but IMO having a detailed description of the changes that (being honest) takes just a few minutes to read is a big win, not a burden. I think it is easier to evaluate the API changes introduced by the PR when they are very clearly delineated with examples. I specifically told Claude to do so, so that was a human decision not AI. 😄 [EDIT] - the management summary is there now and I put all the details into click-to-expand dropdowns. Like I said, this is a bit of a test case and as a project we need to understand how to make AI work effectively for us (for those who choose to use it). |
Nope! I think this PR should be fine to merge in. Thanks for the discussion! |
|
So the discussion here makes me confused. I see that astropy/astroquery#3642 was closed for violating policy, but this same author then makes astropy/astropy#20212 that does declare AI usage and also provides human-ish response (to my naive self anyway), so should I let astropy/astropy#20212 proceed or close it? |
|
This PR has spawned important discussion that is also out of scope of this particular PR. We will be discussing Astropy's stance on AI at the upcoming coordination meeting, so I have opened this issue to collect discussion in advance of that: #540 |
|
Thanks @zacharyburnett ! |
changes:
Important
Autonomous workflows ("agents") are not human contributors. Astropy reviewers expect to interact with the human controlling the account, or we will close the PR.