Repository navigation
Python-graphblas: high-performance sparse linear algebra for scalable graph analytics #81
Description
Submitting Author: Erik Welch (@eriknw)
All current maintainers: (@eriknw, @jim22k, @SultanOrazbayev)
Package Name: Python-graphblas
One-Line Description of Package: Python library for GraphBLAS: high-performance sparse linear algebra for scalable graph analytics
Repository Link: https://github.com/python-graphblas/python-graphblas
Version submitted: 2023.1.0
Editor: @tomalrussell
Reviewer 1: @sneakers-the-rat
Reviewer 2: @szhorvat
Archive:
JOSS DOI: N/A
Version accepted: 2023.7.0
Date accepted (month/day/year): 07/14/2023
Description
Python-graphblas is like a faster, more capable scipy.sparse that can implement NetworkX. It is a Python library for GraphBLAS: high-performance sparse linear algebra for scalable graph analytics. Python-graphblas mimics the math notation, making it the most natural way to learn, use, and think about GraphBLAS. In contrast to other high level GraphBLAS bindings, Python-graphblas can fully and cleanly support any implementation of the GraphBLAS C API specification thereby allowing us to be vendor-agnostic.
Scope
- Please indicate which category or categories this package falls under:
- Data retrieval
- Data extraction
- Data munging
- Data deposition
- Reproducibility
- Geospatial
- Education
- Data visualization*
- Scientific software wrappers (added from here)
Please fill out a pre-submission inquiry before submitting a data visualization package. For more info, see notes on categories of our guidebook.
-
For all submissions, explain how the and why the package falls under the categories you indicated above. In your explanation, please address the following points (briefly, 1-2 sentences for each):
- Who is the target audience and what are scientific applications of this package?
Audience: anybody who works with sparse data or graphs. We are also implementing a backend to NetworkX (which supports dispatching in version 3.0) written in Python-graphblas called graphblas-algorithms, so we are quite literally targeting NetworkX users!
Python-graphblas provides a faster, easier, more flexible, and more scalable way to operate on sparse data, including for graph algorithms. There are too many scientific applications to list ranging from neuroscience, genomics, biology, etc. It may be useful wherever scipy.sparse or NetworkX are used. Although GraphBLAS was designed to build graph algorithms, it is flexible enough to be used in other applications. Anecdotally, most of our current users that I know about are from research groups in universities and laboratories.
We are also targeting applications that need very large distributed graphs. We have experimented with Dask-ifying python-graphblas here, and we get regular interest from people who want e.g. distributed PageRank or connected components.
- Are there other Python packages that accomplish the same thing? If so, how does yours differ?
pygraphblas, which hasn't been updated in more than 16 months. There are many differences in syntax, functionality, philosophy, architecture, and (I would argue) robustness and maturity. python-graphblas syntax targets the math syntax, whereas pygraphblas is much closer to C. python-graphblas handles dtypes much more robustly, has efficient conversions to/from numpy and other formats, is architected to handle additional GraphBLAS implementations (more are on the way!), has exceptional error messages, has many more tests and functionality, supports Windows, and much, much more. We have also been growing our team, because sustainability is very important to us.
Although we have/had irreconcilable differences (which is why we decided to create python-graphblas), the authors have always been cordial. We all believe strongly in the ethos of open source, and I would describe our relationship as having "radical generosity". For example, we have an outstanding agreement that each library is welcome to "borrow" from the other (with credit). We may "borrow" some of their documentation :)
We also worked together to create and maintain the C binding to SuiteSparse:GraphBLAS:
https://github.com/GraphBLAS/python-suitesparse-graphblas/
We could use help automatically generating wheels for this library on major platforms via cibuildwheel.
- If you made a pre-submission enquiry, please paste the link to the corresponding issue, forum post, or other discussion, or
@tagthe editor you contacted:
Limited prior discussion in this issue: pyOpenSci/python-package-guide#21 (comment)
Technical checks
For details about the pyOpenSci packaging requirements, see our packaging guide. Confirm each of the following by checking the box. This package:
- does not violate the Terms of Service of any service it interacts with.
- has an OSI approved license.
- contains a README with instructions for installing the development version.
- includes documentation with examples for all functions.
- We're working on this! The C API and SuiteSparse:GraphBLAS C library are both well documented. We have a very large API surface area to cover, so "documentation with examples for all functions" is a really, really high bar, but one I hope we achieve someday :)
- contains a vignette with examples of its essential functions and uses.
- has a test suite.
- has continuous integration, such as Travis CI, AppVeyor, CircleCI, and/or others.
Publication options
- Do you wish to automatically submit to the Journal of Open Source Software? If so:
- Undecided. We don't have a paper, but in principle I would like for us to submit a paper to JOSS someday.
JOSS Checks
- The package has an obvious research application according to JOSS's definition in their submission requirements. Be aware that completing the pyOpenSci review process does not guarantee acceptance to JOSS. Be sure to read their submission requirements (linked above) if you are interested in submitting to JOSS.
- The package is not a "minor utility" as defined by JOSS's submission requirements: "Minor ‘utility’ packages, including ‘thin’ API clients, are not acceptable." pyOpenSci welcomes these packages under "Data Retrieval", but JOSS has slightly different criteria.
- The package contains a
paper.mdmatching JOSS's requirements with a high-level description in the package root or ininst/. - The package is deposited in a long-term repository with the DOI:
Note: Do not submit your package separately to JOSS
Are you OK with Reviewers Submitting Issues and/or pull requests to your Repo Directly?
This option will allow reviewers to open smaller issues that can then be linked to PR's rather than submitting a more dense text based review. It will also allow you to demonstrate addressing the issue via PR links.
- Yes I am OK with reviewers submitting requested changes as issues to my repo. Reviewers will then link to the issues in their submitted review.
Code of conduct
- I agree to abide by pyOpenSci's Code of Conduct during the review process and in maintaining my package should it be accepted.
Other comments (manually added)
Given a product mindset, we believe that Python-graphblas is a great product, but I think our go-to-market strategy has been lacking. We have been very engineering-heavy, and even our goal of targeting NetworkX users is engineering-heavy via creating graphblas-algorithms. I hope this peer-review process can help us prioritize our efforts (such as a plan to improve documentation) as well as a place to write a blog post or two.
Please fill out our survey
- Last but not least please fill out our pre-review survey. This helps us track
submission and improve our peer review process. We will also ask our reviewers
and editors to fill this out.
P.S. *Have feedback/comments about our review process? Leave a comment here
Editor and Review Templates
Metadata
Metadata
Assignees
Type
Projects
- Status
Activity
NickleDave commentedon Feb 7, 2023
Hi @eriknw!
We're very glad to see you all have gone ahead with a full submission, after discussion in pyOpenSci/python-package-guide#21 (comment) as you linked above.
I just want to welcome you and let you know we are working on this.
I will finish the initial editor checks by the end of this week. @lwasser is traveling and we will need to co-ordinate about editors, but I expect to get back to you about that by the middle of next week at the latest.
Thank you for providing all the detail in the submission. The context you've provided will be helpful for the review (and will definitely help me with editor checks!). It sounds like you've anticipated some points @lwasser brought up when we discussed in Slack. Looking forward to helping you improve the docs and giving you some blog post material! 😁
NickleDave commentedon Feb 9, 2023
Editor in Chief checks
These are the basic checks that the package needs to pass to begin review.
Please check our Python packaging guide for more information on the elements below.
import package-name.pip install python-graphbalsin a python 3.10 venv on LinuxREADME.mdfile with clear explanation of what the package does, instructions on how to install it, and a link to development instructions.CONTRIBUTING.mdfile that details how to install and contribute to the package.Code of Conductfile.NOTE: We prefer that you have development instructions in your documentation too.
YAMLheader of the issue (located at the top of the issue template).We appreciate each maintainer of the package filling out this survey individually. 🙌
Thank you authors in advance for setting aside five to ten minutes to do this. It truly helps our organization. 🙌
Editor comments
This package passes all checks; we can begin review.
It's obvious the developers and maintainers have done a ton of work.
Our goal here should be to help make sure everyone can appreciate how much work they have done, and how much functionality is packed into python-graphblas.
Along those lines, some notes for the review (none of these need to be addressed before we start):
NickleDave commentedon Feb 9, 2023
@eriknw @jim22k, @SultanOrazbayev the tl;dr is that python-graphblas passed editor checks 🙂 🎉
Like I said above, I'll need to co-ordinate with @lwasser who is traveling about an editor for this review, but would expect us to reply back here by middle of next week at the latest
eriknw commentedon Feb 9, 2023
Hooray! 🎉
Thanks @NickleDave. We appreciate the attention y'all are giving us, and thanks for telling us what (and when) to expect next. We're in no particular rush--it's more important to give the right people the right amount of time to do things right :)
NickleDave commentedon Feb 9, 2023
🙌
finding the right people now! 🙂
NickleDave commentedon Feb 14, 2023
Hi again @eriknw, @jim22k, @SultanOrazbayev -- just letting you know that I did have a chance to talk with @lwasser now that she has returned, and we are in the process of finding an editor
When you have a chance could you please (all) fill out the pre-review survey?
It's here: https://forms.gle/F9mou7S3jhe8DMJ16
(I know it's easy to miss in the template)
NickleDave commentedon Feb 22, 2023
Hi @eriknw, @jim22k, @SultanOrazbayev, brief update:
very happy to inform you that @tomalrussell will be guest editor for this review! 🎉 NetworkX contributor, developer of spatial & network tools like snkit.
I will let @tomalrussell take it from here!
tomalrussell commentedon Feb 24, 2023
Hi @eriknw, @jim22k, @SultanOrazbayev, and thanks to @NickleDave for the introduction.
I've reached out to potential reviewers, and incidentally look forward to taking a closer look at
python-graphblasmyself. I'll update here as soon as, definitely within a week.tomalrussell commentedon Mar 7, 2023
👋 Hi @sneakers-the-rat and @szhorvat! Thank you for volunteering to review
for pyOpenSci!
The following resources will help you complete your review:
here as a comment, once your review is complete.
Please get in touch with any questions or concerns! Your review is due in three weeks: 29th March 2023
47 remaining items
szhorvat commentedon Jul 13, 2023
Yes. The suggestions I made are mostly for the long term.
sneakers-the-rat commentedon Jul 14, 2023
Same, full approval from me :)
tomalrussell commentedon Jul 14, 2023
Thanks all, time and effort very much appreciated. The review process is done!
All that's left is to wrap up, publish the version of record and acknowledge all your contributions.
🎉
python-graphblashas been approved by pyOpenSci! Thank you @eriknw for submittingpython-graphblasand many thanks to @szhorvat and @sneakers-the-rat for reviewing this package! 😸Author and Reviewer Wrap-Up Tasks
There are just a few things left to do to wrap up this submission, @eriknw, @jim22k, @SultanOrazbayev:
python-graphblas. The badge should be[](https://github.com/pyOpenSci/software-review/issues/81).python-graphblasto the pyOpenSci website. Please open a PR to update this file: to add your package and name to the list of contributors.Both reviewers and maintainers (@sneakers-the-rat, @szhorvat too):
Editor Final Checks
Please complete the final steps to wrap up this review. @tomalrussell, please do the following:
6/pyOS-approved6 🚀🚀🚀.If you have any feedback for us about the review process please feel free to share it here. We are always looking to improve our process and documentation in the peer-review-guide.
tomalrussell commentedon Jul 14, 2023
@eriknw and team, I'd also like to invite you to write a blog post on your package for us to promote your work. If you are interested - here are a few examples of other blog posts:
2019-11-18-pandera-dataframe-validation.mdis a markdown example that you could use as a guide when creating your post.This can be a really high-level motivation for the package, for a slightly-scientific-Python-user-audience, or could draw on your introductory tutorial material to get straight to what the package does..
This is totally optional and not a requirement, but if you have time, we'd love to spread the word about☺️
python-graphblasto pyOpenSci blog readerssneakers-the-rat commentedon Jul 17, 2023
did the post-review survey and submitted contributors.yml patch :)
lwasser commentedon Jul 27, 2023
Friends - i Believe this issue can be closed!! if it should be opened please just reopen or let me know! congratulations on a successful review and thank you everyone for participating in our pyOpenSci review process! I am so appreciative of you all!! ✨
eriknw commentedon Jul 27, 2023
w00t! Also, we'd be happy to write a short blog post :)
Update packages.yml