Skip to content

Support for the Research Organization Registry (ROR)#4483

Merged
mauromsl merged 57 commits into
masterfrom
3168-ror
Mar 19, 2025
Merged

Support for the Research Organization Registry (ROR)#4483
mauromsl merged 57 commits into
masterfrom
3168-ror

Conversation

@joemull

@joemull joemull commented Nov 8, 2024

Copy link
Copy Markdown
Member

Closes #3168.

Overview

This piece of work adds a ROR data model, including affiliations, organizations, organization names, and locations. It includes an import routine to fetch and process ROR's full database. I've included as much backwards compatibility as possible for importers that do not know about ROR. It also adds a user interface for editing one's own affiliations, and an admin interface for all of the relevant models.

Command line interface

❯ python src/manage.py import_ror_data
DEBUG 2024-12-09 18:41:33,666 connectionpool P:131157 T:135176669522816 Starting new HTTPS connection (1): zenodo.org:443
DEBUG 2024-12-09 18:41:34,697 connectionpool P:131157 T:135176669522816 https://zenodo.org:443 "GET /api/communities/ror-data/records?sort=newest HTTP/1.1" 200 None
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<00:00, 101.06it/s]
INFO 2024-12-09 18:41:38,658 import_ror_data P:131157 T:135176669522816 successful

User interface

image
image
image
image
image
image
image

@joemull joemull linked an issue Nov 15, 2024 that may be closed by this pull request
1 task
@joemull joemull force-pushed the 3168-ror branch 2 times, most recently from d314d51 to cc46446 Compare December 9, 2024 18:20
@joemull joemull requested a review from ajrbyers December 9, 2024 18:43
@joemull joemull requested a review from MartinPaulEve December 9, 2024 18:44
@joemull joemull marked this pull request as ready for review December 9, 2024 18:44
@MartinPaulEve

Copy link
Copy Markdown
Contributor

If you enter an organization name that isn't found (e.g., in test import, "Birkbeck") and then click on one of the pagination items at the bottom, you are redirected to an erroring page eg:

http://127.0.0.1:8000/JRNL/profile/organization/search/?q=birkbeck&page=3&paginate_by=25

This errors because the item isn't found, but the search query is inserted into the pagination link

@MartinPaulEve MartinPaulEve left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This mostly looks really good. Just a few small things.

  • Pagination in ROR search currently redirects to page not found error
  • List of affiliations should show which is primary?
  • RORImportStatus should not be duplicated?

Comment thread src/core/logic.py Outdated
Comment thread src/submission/tests.py
Comment thread src/utils/migrations/0035_rorimport_rorimporterror.py Outdated
@MartinPaulEve MartinPaulEve removed their assignment Dec 13, 2024
@joemull

joemull commented Dec 13, 2024

Copy link
Copy Markdown
Member Author
  • Pagination in ROR search currently redirects to page not found error

This turned out to be a Django bug I believe. I think it's obscure because most people don't encounter it, since they submit their search query to change the list of items before selecting a page.

But I've gone ahead and fixed it for this view.

  • List of affiliations should show which is primary?

Yes, good call. I added a label.

I also updated the save method to make the affiliation primary if it's the first one.

image

  • RORImportStatus should not be duplicated?

That's standard, I'm pretty sure. See comment inline.

@joemull joemull assigned MartinPaulEve and unassigned ajrbyers Dec 13, 2024
@joemull

joemull commented Jan 6, 2025

Copy link
Copy Markdown
Member Author

Martin approved this off GitHub on 13 Dec. Moving it along to @ajrbyers.

@joemull joemull requested review from mauromsl and removed request for MartinPaulEve January 6, 2025 10:53
@joemull joemull assigned ajrbyers and unassigned MartinPaulEve Jan 6, 2025
@joemull joemull removed the request for review from mauromsl January 6, 2025 10:53

@ajrbyers ajrbyers left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is generally very good. The CBVs are growing on me... slowly.

There are some comments inline that should be addressed. Here are some general comments in addition:

  • Institution and Affiliation fields are removed without preservation.
  • Mauro made a suggestion to handle setting inst/dept etc using the Manager, this should be added.
  • When my affiliation was snapshotted it did not include the start/end date or the title.
  • There is a conflict currently listed
Screenshot 2025-01-15 at 10 52 31

Comment thread src/utils/management/commands/import_ror_data.py Outdated
Comment thread src/utils/management/commands/import_ror_data.py
Comment thread src/core/models.py Outdated
Comment thread src/core/models.py Outdated
Comment thread src/core/models.py Outdated
Comment thread src/core/models.py Outdated
Comment thread src/templates/admin/elements/layout/key_value_above.html
@joemull joemull assigned mauromsl and unassigned ajrbyers Jan 15, 2025
@joemull joemull requested a review from mauromsl January 15, 2025 11:18

@mauromsl mauromsl left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mega job here @joemull

I've added a few comments regarding backwards compatibility and database integrity but very happy overall.

I'm testing out the ROR data import procedure locally to try and anticipate any possible hiccups in production but I don't want to hold you up further, good work!

Comment thread src/core/models.py Outdated
Comment thread src/core/models.py Outdated
Comment on lines +239 to +242
def get_or_create(self, defaults=None, **kwargs):
"""
Backwards-compatible override for affiliation-related kwargs
"""

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should instead do this at the Queryset level, since django will delegate this call to AccountQuerySet and the queryset also exposes these ORM operations.

Also, don't forget about get, create, filter and update_or_create

The bulk operations can be ignored IMO

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, I've given this a shot with AffiliationCompatibleQueryset. Let me know what you think.

Comment thread src/core/models.py
Comment on lines +2020 to +2024
language = models.CharField(
max_length=10,
blank=True,
choices=submission_models.LANGUAGE_CHOICES,
)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a rationale behind avoiding the use of django-modeltranslations here? Does it have to do with the need for a distinction between translations and ror_display_for?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for raising this. Yes, ROR data sometimes has multiple names per language. I've written it up here: #4657

Coincidentally, the importer was bugged because ROR gives 2-character language codes but our LANGUAGE_CHOICES uses 3-character codes. So I've fixed that by using a small ISO-639 utility. I've also documented which version of ISO-639 we hardcode, for future reference.

Comment thread src/core/models.py Outdated
Comment thread src/core/models.py Outdated
Comment thread src/core/models.py Outdated
Comment thread src/core/models.py Outdated
Comment thread src/cron/management/commands/send_publication_notifications.py Outdated
Comment thread src/submission/models.py Outdated
Comment thread src/submission/tests.py Outdated
@mauromsl mauromsl assigned joemull and unassigned mauromsl Mar 11, 2025

@mauromsl mauromsl left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome! a couple minor nitpicks and we are good to go. This is turning into a very nice piece of kit :)

Comment thread src/core/model_utils.py Outdated
Comment on lines +847 to +868
backwards_keys = (
(
# Account and FrozenAuthor had 'institution'
re.compile(r'^institution'),
'controlledaffiliation__organization__labels__value',
),
(
# PreprintAuthor had 'affiliation'
re.compile(r'^affiliation'),
'controlledaffiliation__organization__labels__value',
),
(
# Account and FrozenAuthor had 'department'
re.compile(r'^department'),
'controlledaffiliation__department',
),
(
# Account and FrozenAuthor had 'country'
re.compile(r'^country'),
'controlledaffiliation__organization__locations__country',
)
)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can these be turned into a constant / module level variable so that they don't get redefined on every call to this method

Comment thread src/core/models.py
Comment on lines +188 to +196
affil_kwargs = self._pop_old_affiliation_lookups(kwargs)

obj = self.model(**kwargs)
obj.clean()
self._for_write = True
obj.save(force_insert=True, using=self.db)

if affil_kwargs:
self._create_affiliation(affil_kwargs, obj)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a bit of an anti-pattern, we need to document why we are overriding the parent behaviour without calling super, either as a comment or in the docstring

@mauromsl mauromsl assigned joemull and unassigned mauromsl Mar 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add support for Research Organization Registry (ROR)

4 participants