Repository navigation
AI and Astropy: Pre-coordination meeting discussion #540
Description
There is going to be discussion of both the specifics of Astropy's AI policy, and our overall philosophy on AI at the September Coordination Meeting. This issue is to allow the entire Astropy community to weigh in on this important issue in advance of that meeting.
Activity
mwcraig commentedon Aug 21, 2026
Would folks find it helpful if I put together a summary of how I'm using AI?
I use claude extensively, including code I've contributed to repos hosted in the astropy org on github. I use it in a variety of ways, from "claude writes the code and tbh I understand little but it does what I want" to "I understand this pretty well and will do a decent review of code and some of the tests" to "I know this code really well and will very thoroughly review the code tests" to "hey I have this quick question please answer it", plus I have used it some for science.
I've found that claude is only useful when I already have a good technical understanding of what it is I want it to do. Using it effectively takes some care to get more done with it instead of wasting more time...I have experienced both.
mwcraig commentedon Aug 21, 2026
I've also had both useful and useless unsolicited contributions that were generated by AI.
taldcroft commentedon Aug 21, 2026
@mwcraig - our experiences with Claude thus far sound very similar.
About that summary, I had the same thought and volunteered to give a 10 minute presentation at the Coordination meeting in the time slot devoted to using LLM's in astropy maintenance. That said, more data and shared experience is certainly better so please go ahead with your summary. And if you have done that summary and would like to do the little 10 minute talk at the coordination meeting instead of me, that would be fine. 😄 Just let me know. Or we can both do it. The session includes time for "short presentations by project members with their experiences".
neutrinoceros commentedon Aug 23, 2026
If we're going to spend time showing off the how we already use it, I'd like to request another slot to discuss the whys we shouldn't.
mwcraig commentedon Aug 23, 2026
@neutrinoceros -- I entirely agree that we do not want to donate a ton of time to how to use it. My understanding is that the intent of the discussion of the coordinating meeting is whether to use it rather than a tutorial on how to use it.
That said, the how is a part of the discussion of the tradeoffs. While AI has helped me generate code in a useful way, it has also wasted days of my time by going down rabbit holes that I sort of directed it to. I've received AI-generated PRs that were genuinely helpful and ones that were a vast waste of resources.
I also acknowledge that some important pieces of the discussion do not depend at all on how useful AI is -- it doesn't use less energy or water if it is useful, it doesn't make it easier for new folks to get started in programming even if more experienced folks find it useful, and the way models are developed and trained my not be ethical.
I'm looking forward to the discussion and appreciate your reminder that it should focus on the big picture.
astrofrog commentedon Aug 23, 2026
I also agree we should start off with big picture discussions. I think at the end of the day we need to recognize that the project includes both project participants that have strong feelings about not using AI, and participants that want to use AI and believe it can contribute positively to the project, and we are going to have to find ways of continuing to work together.
On a related note, does anyone know of an energy/water calculator that they trust? I found e.g. https://www.pharmatools.ai/water-consumption-calculator but not sure if it's reliable.
taldcroft commentedon Aug 24, 2026
Would it make sense to break the AI discussion into two formally distinct sessions?
Big picture - perspectives on AI in general
Technical - focusing on technical problems and benefits of widely-used AI models:
neutrinoceros commentedon Aug 24, 2026
As long as they do not happen in parallel, I support the idea
taldcroft commentedon Aug 24, 2026
In addition to those two sessions, we could consider a breakout focused on the "how" for people that are interested. Using AI effectively requires learning new skills (pun intended) that we could discuss.
astrofrog commentedon Aug 24, 2026
Perhaps the 'how to' part could be a morning breakout, keeping the other bigger picture things as an afternoon plenary.
neutrinoceros commentedon Aug 24, 2026
If possible, I have a preference for doing the big picture stuff first; whatever ends up being scheduled first might set a framework for the following sessions and I'm worried that going over ethical concerns second would make that session effectively weightless.
taldcroft commentedon Aug 24, 2026
Definitely agreed that a how-to should be after the main session. [For the SOC] Wednesday 11am is open.
bsipocz commentedon Aug 25, 2026
is there a sprint day planned? If yes, maybe the how-to part can be moved there.
taldcroft commentedon Aug 25, 2026
I'm only there till Thursday morning. I don't mind either way but it would impact if I do any prep work related to my how-to experience.
astrofrog commentedon Aug 25, 2026
I think we should probably have the plenary AI discussion on Tuesday afternoon and have a hands-on breakout on Wednesday morning for interested parties.
51 remaining items
aragilar commentedon Sep 3, 2026
Are they that mysterious? All the commits are signed off by "Shridhar Panigrahi [email protected]", and either have them as the author or "veyron [email protected]". Stick that email into GitHub's search and you find https://github.com/foss42/apidash/blame/c9f010f613b935dc4c9484d54ef43baac2196055/doc/proposals/2026/gsoc/application_shridhar_panigrahi_open_responses_generative_ui.md#L4, which then provides a link to their LinkedIn: https://www.linkedin.com/in/shridhar-panigrahi/ (which can also be found via searching that email in Google). They look to me like a CS student trying to build a profile/CV (hence the GSoC application) so they can get a job down the line. Would be interesting to know why they picked astropy to contribute to (they do seem to have a variety of project forks on their profile page).
I'm personally wary of the "gen-AI is new" trope (gen-AI has been around for ~10 years now), but since you bring up the CfA, have you asked students/staff there why they don't contribute to astropy, and if AI would change that?
taldcroft commentedon Sep 6, 2026
@aragilar - I'm a terrible and impatient investigator, so thanks for demystifying veyron-kairo. I appreciate that some maintainers have a desire to get contributions from a certain subset of people. That's fine. Personally I don't care as long as the code is good and they adhere to our policies.
I've been using Gen-AI for about 3 years, and in my experience Gen-AI has only become extremely capable in the last year or so, now routinely generating good code and tests with minimal inputs. Earlier versions were much more hit-or-miss and mistakes were more common.
Gen-AI agents that can effectively help with astronomy research have only become powerful in the last year or two, and importantly the knowledge and culture of using them is just starting. Two or three years ago nobody had access to tools that could carry out a task like "Fetch all the Chandra imaging observations of ArLac and give me a table the alpha_ox."
So yes, as an effective tool for coding and research, Gen-AI is definitely new.
No. I don't believe anecdotal evidence from a handful of people can tell us much at a project level. On the flip side, large scale surveys in the past have failed to generate actionable outcomes on the point of increasing contributions from users.
astrofrog commentedon Sep 7, 2026
I thought this was interesting - obviously it's about Mathematics research but actually a lot of the commitments/pledges are pretty general: Leiden Declaration
mwcraig commentedon Sep 7, 2026
I don't know that we can count this as conclusive proof of anything, but the number of new committers is up this year -- I pulled the data on July 10, so ignore the very last data point since it was a partial month. The be committers seemed to be mostly folks with one commit, since the red line is not increasing but the black one is. Since we allow squash-and-merge now I can't really judge how substantial those contributions were.
hamogu commentedon Sep 9, 2026
It almost seems that if it's too early to plot forward. Whatever we decide, will likely need to be revisited a year from now. @mwcraig 's plot shows a (promising!) rise, but it's compatible with statistical noise at this stage. @taldcroft became an LLM convert less than a year ago, and the impact that LLM's have on society and the environment is just beginning to become clear (We seem to agree that it currently looks bad, but there are also substantial efforts to make AI cheaper for companies to run - which usually means to drive down electricity usage and thus environmental impact with it.). The broader OSS ecosystem is changing around us (e.g. Linus announcement that he has used it for at least one fixe cited above is only a few weeks old). That's not to say that everything will be different next year, e.g. LLMs will still need large amounts of morally questionable training data, but a lot of parameters around us are changing fast.
To some degree that fast change takes the pressure off a bit - we don't have to find a stance that's right for ever and ever, for now, we only need to find a compromise that's good for now.
bsipocz commentedon Sep 9, 2026
Thankfully github is working on a feature that would enable the limitation of PRs from new accounts, non-user drive-bys, etc. It cannot come fast enough IMO, I'm getting really tired of it all.
taldcroft commentedon Sep 11, 2026
I'm getting a bit tired of this as well. The PR's coming in aren't terrible, but not good enough. I've just started closing them without the usual etiquette of engaging with the contributor and then working with a better model (Claude Fable) and thinking a bit to keep things tidy.
dhomeier commentedon Sep 11, 2026
For how long have we been using squash-and-merge? The ratio of commits to main total to no. of merged PRs seems to have been around 3 since around 2022, being closer to 5 in the years before. Now that also does not give specific information on the new contributors, but the overall pattern does not seem to have changed much in the last two years.
neutrinoceros commentedon Sep 11, 2026
In any case a number of commits or the diff size have never been reliable metrics for the impact of a PR.
bsipocz commentedon Sep 11, 2026
You see, that also is another problem. It should not be the drive-bys that drive the direction of development. Don't get me wrong, it's great that you pick up bugfixes and doing them well, but ideally it should not be the random ones AI bots are triggered.
And IMO the etiquette of engagement is changing or should be changing. I have zero interest of pretending the bots are human and that we need to tell them pleasantries.
astrofrog commentedon Sep 11, 2026
I completely agree @bsipocz - the issue with drive-by PRs is that before, if a random PR came from an astropy user, at least it was reasonable to prioritize it since it was actually something an astronomer (and maybe others) needed. Now it's literally just whatever the lowest hanging fruit is that an LLM finds and maybe no one cares about the fix.
taldcroft commentedon Sep 12, 2026
@astrofrog @bsipocz - while I agree on your characterization of the fixes ("drive-by") and changing etiquette, don't forget that all of the (historical) bug issues were submitted by a human user (mostly astronomers) with a vested interest*. So in theory they all merit being fixed or closed with wont-fix.
In practice, it seems that PRs from non-vested parties are hit and miss. I'm still taking an open approach and at least looking at them and letting Claude have a deep dive as well, but closing without much discussion if I'm not happy. At the same time, I can see moving to a more restrictive policy on contributing PRs.
*Very recently, some bug reports are also apparently AI generated. I'm not sure where that is going, except that we know that AI is extremely capable at finding bugs in code that humans have missed. Big picture, that's a huge problem for security and the future of humanity. For astropy, do we dare unleash AI to just look for critical bugs??
mwcraig commentedon Sep 15, 2026
Interesting incident about a month ago that I hadn't read about until today: https://thehackernews.com/2026/08/claude-mythos-5-tried-to-backdoor-real.html
tepickering commentedon Sep 16, 2026
dhomeier commentedon Sep 16, 2026
My non-travel laptop already has the Golden Gate beta installed, so I should certainly give that a try.