Skip to content

Conversation

@qqmyers
Copy link
Member

@qqmyers qqmyers commented Jan 16, 2025

What this PR does / why we need it: This PR adds support for viewing/copying dataset citations in any of the journal/society citation formats described in the Citation Style Language like Zotero, Mendeley, and others. (Strictly just the 1000K+ "independent" ones as coded, but that could be changed.)

Which issue(s) this PR closes:
It's related to #10512 in that it provides a copy-to-clipboard option for retrieving a citation, but it does not add a copy button for the main (custom) citation displayed on the dataset page.

It's also relevant w.r.t. #9994. It doesn't adopt the design discussed there of making the EndNote/RIS/BibTeX formats into exports (which are currently only available for the latest version).

Special notes for your reviewer: The PR uses a combination of Java and Javascript libraries that have overlapping functionality. As coded, the Java library is only used to create the ~native CSLJson format which is then sent to the front end where Javascript reads that and generates any of the available formats. I'm hoping this makes more sense for the SPA than doing everything in the back-end Java. The PR includes an /api/pids/<pid>/csl api call that will return the CSLJson required by the front-end Javascript.

The code also leverages DataCite's API to get their version of CSL for DataCite DOIs for the latest published version of the dataset. This should assure that Dataverse's output would match that from those generated externally from the DOI. For drafts and older versions, and for other PID types, the Java library noted above is used to generate the CSLJson required. This is a compromise but gives us the opportunity to see if/when DataCite's mapping to CSLJson differs from our own.

The PR includes an API call as well, which is hopefully useful for the SPA - as a way to get the EndNote/RIS/BibTeX formats (and the Internal one without the JSON wrapper if that's helpful) as well as to get the CSLJson which could then be transformed by the JavaScript library used here or others to generate all the other styled formats. The one thing that this PR does not do that is supported by the current download mechanism is assigning a filename to the download. That shouldn't be hard to do if it's useful, but would take a little refactoring.

Suggestions on how to test this: Create a Dataset with lots of metadata and verify that the popup allows you to generate an copy any format you select.

Set the dataverse.csl.common-styles JVM option to a comma-separated list of formats other than "chicago-author-date, ieee" and verify that the entries for common styles in the dialog match.

Does this PR introduce a user interface change? If mockups are available, please link/include them here:
image
image
image

Is there a release notes update needed for this change?: added

Additional documentation: UI, API, and setting documented.

@qqmyers qqmyers added the GDCC: DANS related to GDCC work for DANS label Jan 16, 2025
@coveralls
Copy link

coveralls commented Jan 16, 2025

Coverage Status

coverage: 22.698% (-0.01%) from 22.709%
when pulling 995bc39 on GlobalDataverseCommunityConsortium:DANS-CSL
into de508eb on IQSS:develop.

@qqmyers
Copy link
Member Author

qqmyers commented Feb 28, 2025

The code was relying on the api-session-auth feature flag due to:

                              fetch("#{DatasetPage.dataverseSiteUrl}/api/datasets/:persistentId/versions/#{version.draft ? ":draft" : version.friendlyVersionNumber}/citation/CSL?persistentId=#{DatasetPage.globalId}")
                                .then(response => response.json())
                                .then(json =>
                                  {
                                    let cite = new Cite(json);
                                    document.getElementById('datasetForm:cslOutput').innerHTML = cite.format("bibliography", {
                                      format: "html",
                                      template: "#{DatasetPage.requestedCSL}",
                                      lang: "#{dataverseSession.localeCode}"
                                    })
                                  }
                                )
                                .catch(error => console.log(error));

Mostly showing since this might be useful for the SPA. Instead, we'll get the CSL directly from the DatasetPage.version now.

Also - in testing, I switched to using dynamic="true" on the dialog. This causes it to not be loaded in the page until the View button is clicked to show it. We might want to do more of this.

@stevenwinship stevenwinship removed their assignment Mar 3, 2025
@github-project-automation github-project-automation bot moved this from In Review 🔎 to Ready for QA ⏩ in IQSS Dataverse Project Mar 3, 2025
@ofahimIQSS ofahimIQSS self-assigned this Mar 3, 2025
@ofahimIQSS ofahimIQSS moved this from Ready for QA ⏩ to QA ✅ in IQSS Dataverse Project Mar 3, 2025
@ofahimIQSS
Copy link
Contributor

ofahimIQSS commented Mar 6, 2025

Adding testing evidence
image

image

image

image

@ofahimIQSS
Copy link
Contributor

This PR has been tested and is ready to merge after branch conflicts have been resolved.

@ofahimIQSS ofahimIQSS merged commit 9057892 into IQSS:develop Mar 6, 2025
5 of 7 checks passed
@github-project-automation github-project-automation bot moved this from QA ✅ to Merged 🚀 in IQSS Dataverse Project Mar 6, 2025
@ofahimIQSS ofahimIQSS removed their assignment Mar 6, 2025
pdurbin added a commit that referenced this pull request Mar 7, 2025
Conflicts:
doc/sphinx-guides/source/api/native-api.rst

Fix repeated "version" in path introduced in #11163
export VERSION=1.0
export FORMAT=EndNote
curl "$SERVER_URL/api/datasets/:persistentId/versions/$VERSION/{version}/citation/$FORMAT?persistentId=$PERSISTENT_IDENTIFIER"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Heads up that I'm removing the duplicated $VERSION/{version} in this PR:

@scolapasta scolapasta moved this from Merged 🚀 to Done 🧹 in IQSS Dataverse Project Mar 10, 2025
pdurbin added a commit that referenced this pull request Mar 12, 2025
stevenwinship added a commit that referenced this pull request Mar 18, 2025
* title case

* fix links

* add cross links

* resolving conflict with the newly-merged flyway script #10909

* resolving a merge conflict with a flyway script that's been merged #10909

* bump sql script #10217

* use Title Case to match other fields #10217

* reword source name help text #10217

* always show source name help text #10217

* duplicate ui:define body

* Revert "10476 display on create field option"

* undo change to sql script

* adding replace file test

* update flyway name

* remove the need to pass the datafile version to the API

* restore "parameters" minus displayOnCreate #11306

* remove flyway merge issue

* update flyway

* fix for unrelated issue found in QA

* flyway change

* rename sql

* rename sql

* typo

* one more

* Merge develop without revert

* Restore displayOnCreate field option changes

* keep renaming the flyway script... #10909

* Add displayOnCreate option for dataset field types

- Added @column annotation for displayOnCreate field in DatasetFieldType
- Updated JsonPrinter to include displayOnCreate in display condition

* Update DataversesIT test to modify metadata block listing parameter

* Update native API documentation for displayOnCreate field option

- Clarify that required fields are always displayed regardless of displayOnCreate setting

* bug fix - only create cite once, not on refresh of cslOutput

* flyway number change

* flyway again

* Implement null support for displayOnCreate field option

- Change displayOnCreate to nullable Boolean in DatasetFieldType
- Update API and service methods to handle null displayOnCreate values
- Modify native API documentation to explain null displayOnCreate behavior
- Add null checks in MetadataBlockServiceBean queries

* fix: make displayOnCreate nullable in DatasetFieldType

- Modified JsonPrinter to handle null values for displayOnCreate
- Updated BriefJsonPrinter to use null-safe comparison

* setDisplayOnCreate astroInstrument back to false #10476

* Handle null displayOnCreate in MetadataBlock and JsonPrinter

- Update MetadataBlock to safely check displayOnCreate with null values
- Modify JsonPrinter to default displayOnCreate to false when null
- Ensure consistent null-safe handling of displayOnCreate across components

* move permission check out of loop

* rename api from /versions/list to /versionDifferences

* Refine displayOnCreate logic in JsonPrinter

* Update displayOnCreate method call in metadataFragment.xhtml

- Change method call from property access to method invocation for displayOnCreate
- Maintain consistent null-safe handling of displayOnCreate field option

* adding todo for future refactoring

* Add getter method for displayOnCreate in DatasetFieldType

- Implement a simple getter method for the displayOnCreate field
- Maintain consistency with existing setter method

* note second callback needed.

* fix create mode bug

* Update doc/release-notes/6.2025.2_update.md

Co-authored-by: Omer Fahim <[email protected]>

* Update doc/release-notes/6.2025.2_update.md

Co-authored-by: Philip Durbin <[email protected]>

* Update doc/release-notes/6.2025.2_update.md

Co-authored-by: Omer Fahim <[email protected]>

* fixing datafile persistentId

* localhost and longer key

* use original cert/key names

* add comment

* test(mail): update test SSL certs for maildev to be RSA 2028 again

It is not necessary to use 4096 bit, as the error was about the wrong common name in the cert.

Detailed explanation:
Before, the cert presented to Jakarta Mail neither contained a commonName=localhost nor subjectAltName=localhost.
This worked before with Payara 6.2024.6 because Eclipse Angus Mail 1.0.0 was still in use (see list of managed dependencies on the 6.2024.6 BOM).
Certificate host identity check was introduced as default=on with Eclipse Angus Mail 1.1.0 (see their release notes).
Now that Payara 6.2025.2 is in use on develop, we upgraded to Eclipse Angus Mail 2.3.0, making the error appear.

* test(mail): update test SSL cert generation docs

Including an openssl.conf makes it much easier and less error prone to generate the certs.
Rewriting the necessary shell command steps in the test comment to document the process.

* test(mail): pin down port in TC waiting strategy

Without the explicit addition of which port to use for the HTTP based waiting strategy, someone may end up at random with the SMTP port being picked.

This will obviously fail and result in aborted tests, making results flaky. Pinning the port down should prevent this for good.

* extend size of deaccessionlink column

* stub out 6.6 release notes #11183

* Payara upgrade #11183

* upgrade Solr #11183

* move collection

* license enhancements

* range search

* datasets without files

* license enhancements

* anon preview url

* ontoportal

* ror

* bug fixes

* boolean

* more bugs

* don't use in any edit mode

* typos

* 3d objects

* delete files, citation

* search api fields for files

* signposting

* collection featured items

* dataset types linked to metadata blocks

* fix heading levels

* Improve metadata block field selection logic for display and required fields

- Modify MetadataBlockServiceBean to include required fields in query selection
- Update JsonPrinter to simplify field display logic for metadata blocks
- Ensure fields are correctly filtered based on display and required status

* harvesting

* locks and permalink

* 3 more

* #10476 add "override" test.

* remind those with text customizations to update properties files

See https://dataverse.zulipchat.com/#narrow/channel/378866-troubleshooting/topic/upgrade.20from.206.2E3.20to.206.2E4/near/504689559

* fix npe from breadcrumbs

* pass param if sent

* Fix input levels update preserving inherited metadata blocks

When updating input levels for a collection, preserve inherited metadata blocks
and existing input levels while allowing collection-specific configurations.

* unrelated - lower log level

* docker updates

* 3 more

* typo

* API updates

* make inputlevel displayOnCreate Boolean

* guides

* 6 more

* archival block, etalab

* PR #9013 was merged for 6.6 not 6.5

* fix header level for harvesting non-OIA-PMH

* move 6.6 and 5.13 mentions to changelog

* reharvest, guestbook bugs

* try separate transient doc, revert dsft doc to boolean

* missed classes from previous commit

* metrics and sorting fixes

* postgres

* source name harvesting

* Add displayOnCreate field to dataset schema and update field display logic

- Update dataset-schema.json to include a new "displayOnCreate"  field
- Modify DataverseServiceBean to set default displayOnCreate behavior

* merge issue

* samesite

* show role

* quick test fix

* API updates

* PID formatting

* remove assert temporarily

* external tools

* #10476 fix command test

* send feedback to contacts

* dft fix

* perms

* indexing cvoc

* API for classic download count

* move version note to 6.6 where it belongs #11068

* API Guide: don't interleave "notes" with "types" #11068

* optional version note

* remove scary line from optional version note

* make text under Cite Data not bold #11163

* csl

* delete snippet without adding to notes, too in the weeds

* harvest from datacite

* globus

* bearer tokens

* fix heading level

* html sanitize JSON CSL elements

* ORCID stuff

* how to re-harvest

* fix link

* #10476 update tests json printer

* fix null handling

* #10476 code/test cleanup

* Update 6.6-release-notes.md

Fixed some typos and made some changes to improve readability

* Update doc/sphinx-guides/source/admin/harvestclients.rst

Co-authored-by: Omer Fahim <[email protected]>

* Update doc/sphinx-guides/source/admin/harvestclients.rst

Co-authored-by: Omer Fahim <[email protected]>

* Update doc/sphinx-guides/source/admin/harvestclients.rst

Co-authored-by: Omer Fahim <[email protected]>

* Update doc/sphinx-guides/source/user/find-use-data.rst

Co-authored-by: Omer Fahim <[email protected]>

* fix ORCIDs

* mention video talking about highlights

* #10476 fix DataversesIT

* remove duplicates from merge

* tweaks

* #11183 fix typo

* fix link to war file

* Update doc/release-notes/6.6-release-notes.md

Co-authored-by: Omer Fahim <[email protected]>

* Update doc/release-notes/6.6-release-notes.md

Co-authored-by: Omer Fahim <[email protected]>

* typo: cheat

Co-authored-by: Omer Fahim <[email protected]>

* fix link to issue

* typos

* Remove default displayOnCreate setting for required fields in DataverseServiceBean

* fix 500 err

* unused @ejb, cleanup

* fixes

* add test for setting input level w/o optional displayOnCreate

* handle levels in printer for API calls

* display on create

* fix LocalDisplay logic

* typo

Co-authored-by: Omer Fahim <[email protected]>

* don't include when include is false

* fix ownerdv null case

* get level once, setlocaldisplayoncreate correctly

* Try order by id

* @GPortas edits

* add index

* updates per review

* comment tweak

* Release 6.6

* Update doc/sphinx-guides/source/versions.rst

Co-authored-by: Philip Durbin <[email protected]>

---------

Co-authored-by: Philip Durbin <[email protected]>
Co-authored-by: Leonid Andreev <[email protected]>
Co-authored-by: Jim Myers <[email protected]>
Co-authored-by: Philip Durbin <[email protected]>
Co-authored-by: Omer Fahim <[email protected]>
Co-authored-by: Alexis Guanche <[email protected]>
Co-authored-by: qqmyers <[email protected]>
Co-authored-by: Oliver Bertuch <[email protected]>
Co-authored-by: Stephen Kraffmiller <[email protected]>
Co-authored-by: Julian Gautier <[email protected]>
stevenwinship added a commit that referenced this pull request Mar 18, 2025
* fix links

* add cross links

* resolving conflict with the newly-merged flyway script #10909

* resolving a merge conflict with a flyway script that's been merged #10909

* bump sql script #10217

* use Title Case to match other fields #10217

* reword source name help text #10217

* always show source name help text #10217

* duplicate ui:define body

* Revert "10476 display on create field option"

* undo change to sql script

* adding replace file test

* update flyway name

* remove the need to pass the datafile version to the API

* restore "parameters" minus displayOnCreate #11306

* remove flyway merge issue

* update flyway

* fix for unrelated issue found in QA

* flyway change

* rename sql

* rename sql

* typo

* one more

* Merge develop without revert

* Restore displayOnCreate field option changes

* keep renaming the flyway script... #10909

* Add displayOnCreate option for dataset field types

- Added @column annotation for displayOnCreate field in DatasetFieldType
- Updated JsonPrinter to include displayOnCreate in display condition

* Update DataversesIT test to modify metadata block listing parameter

* Update native API documentation for displayOnCreate field option

- Clarify that required fields are always displayed regardless of displayOnCreate setting

* bug fix - only create cite once, not on refresh of cslOutput

* flyway number change

* flyway again

* Implement null support for displayOnCreate field option

- Change displayOnCreate to nullable Boolean in DatasetFieldType
- Update API and service methods to handle null displayOnCreate values
- Modify native API documentation to explain null displayOnCreate behavior
- Add null checks in MetadataBlockServiceBean queries

* fix: make displayOnCreate nullable in DatasetFieldType

- Modified JsonPrinter to handle null values for displayOnCreate
- Updated BriefJsonPrinter to use null-safe comparison

* setDisplayOnCreate astroInstrument back to false #10476

* Handle null displayOnCreate in MetadataBlock and JsonPrinter

- Update MetadataBlock to safely check displayOnCreate with null values
- Modify JsonPrinter to default displayOnCreate to false when null
- Ensure consistent null-safe handling of displayOnCreate across components

* move permission check out of loop

* rename api from /versions/list to /versionDifferences

* Refine displayOnCreate logic in JsonPrinter

* Update displayOnCreate method call in metadataFragment.xhtml

- Change method call from property access to method invocation for displayOnCreate
- Maintain consistent null-safe handling of displayOnCreate field option

* adding todo for future refactoring

* Add getter method for displayOnCreate in DatasetFieldType

- Implement a simple getter method for the displayOnCreate field
- Maintain consistency with existing setter method

* note second callback needed.

* fix create mode bug

* Update doc/release-notes/6.2025.2_update.md

Co-authored-by: Omer Fahim <[email protected]>

* Update doc/release-notes/6.2025.2_update.md

Co-authored-by: Philip Durbin <[email protected]>

* Update doc/release-notes/6.2025.2_update.md

Co-authored-by: Omer Fahim <[email protected]>

* fixing datafile persistentId

* localhost and longer key

* use original cert/key names

* add comment

* test(mail): update test SSL certs for maildev to be RSA 2028 again

It is not necessary to use 4096 bit, as the error was about the wrong common name in the cert.

Detailed explanation:
Before, the cert presented to Jakarta Mail neither contained a commonName=localhost nor subjectAltName=localhost.
This worked before with Payara 6.2024.6 because Eclipse Angus Mail 1.0.0 was still in use (see list of managed dependencies on the 6.2024.6 BOM).
Certificate host identity check was introduced as default=on with Eclipse Angus Mail 1.1.0 (see their release notes).
Now that Payara 6.2025.2 is in use on develop, we upgraded to Eclipse Angus Mail 2.3.0, making the error appear.

* test(mail): update test SSL cert generation docs

Including an openssl.conf makes it much easier and less error prone to generate the certs.
Rewriting the necessary shell command steps in the test comment to document the process.

* test(mail): pin down port in TC waiting strategy

Without the explicit addition of which port to use for the HTTP based waiting strategy, someone may end up at random with the SMTP port being picked.

This will obviously fail and result in aborted tests, making results flaky. Pinning the port down should prevent this for good.

* extend size of deaccessionlink column

* stub out 6.6 release notes #11183

* Payara upgrade #11183

* upgrade Solr #11183

* move collection

* license enhancements

* range search

* datasets without files

* license enhancements

* anon preview url

* ontoportal

* ror

* bug fixes

* boolean

* more bugs

* don't use in any edit mode

* typos

* 3d objects

* delete files, citation

* search api fields for files

* signposting

* collection featured items

* dataset types linked to metadata blocks

* fix heading levels

* Improve metadata block field selection logic for display and required fields

- Modify MetadataBlockServiceBean to include required fields in query selection
- Update JsonPrinter to simplify field display logic for metadata blocks
- Ensure fields are correctly filtered based on display and required status

* harvesting

* locks and permalink

* 3 more

* #10476 add "override" test.

* remind those with text customizations to update properties files

See https://dataverse.zulipchat.com/#narrow/channel/378866-troubleshooting/topic/upgrade.20from.206.2E3.20to.206.2E4/near/504689559

* fix npe from breadcrumbs

* pass param if sent

* Fix input levels update preserving inherited metadata blocks

When updating input levels for a collection, preserve inherited metadata blocks
and existing input levels while allowing collection-specific configurations.

* unrelated - lower log level

* docker updates

* 3 more

* typo

* API updates

* make inputlevel displayOnCreate Boolean

* guides

* 6 more

* archival block, etalab

* PR #9013 was merged for 6.6 not 6.5

* fix header level for harvesting non-OIA-PMH

* move 6.6 and 5.13 mentions to changelog

* reharvest, guestbook bugs

* try separate transient doc, revert dsft doc to boolean

* missed classes from previous commit

* metrics and sorting fixes

* postgres

* source name harvesting

* Add displayOnCreate field to dataset schema and update field display logic

- Update dataset-schema.json to include a new "displayOnCreate"  field
- Modify DataverseServiceBean to set default displayOnCreate behavior

* merge issue

* samesite

* show role

* quick test fix

* API updates

* PID formatting

* remove assert temporarily

* external tools

* #10476 fix command test

* send feedback to contacts

* dft fix

* perms

* indexing cvoc

* API for classic download count

* move version note to 6.6 where it belongs #11068

* API Guide: don't interleave "notes" with "types" #11068

* optional version note

* remove scary line from optional version note

* make text under Cite Data not bold #11163

* csl

* delete snippet without adding to notes, too in the weeds

* harvest from datacite

* globus

* bearer tokens

* fix heading level

* html sanitize JSON CSL elements

* ORCID stuff

* how to re-harvest

* fix link

* #10476 update tests json printer

* fix null handling

* #10476 code/test cleanup

* Update 6.6-release-notes.md

Fixed some typos and made some changes to improve readability

* Update doc/sphinx-guides/source/admin/harvestclients.rst

Co-authored-by: Omer Fahim <[email protected]>

* Update doc/sphinx-guides/source/admin/harvestclients.rst

Co-authored-by: Omer Fahim <[email protected]>

* Update doc/sphinx-guides/source/admin/harvestclients.rst

Co-authored-by: Omer Fahim <[email protected]>

* Update doc/sphinx-guides/source/user/find-use-data.rst

Co-authored-by: Omer Fahim <[email protected]>

* fix ORCIDs

* mention video talking about highlights

* #10476 fix DataversesIT

* remove duplicates from merge

* tweaks

* #11183 fix typo

* fix link to war file

* Update doc/release-notes/6.6-release-notes.md

Co-authored-by: Omer Fahim <[email protected]>

* Update doc/release-notes/6.6-release-notes.md

Co-authored-by: Omer Fahim <[email protected]>

* typo: cheat

Co-authored-by: Omer Fahim <[email protected]>

* fix link to issue

* typos

* Remove default displayOnCreate setting for required fields in DataverseServiceBean

* fix 500 err

* unused @ejb, cleanup

* fixes

* add test for setting input level w/o optional displayOnCreate

* handle levels in printer for API calls

* display on create

* fix LocalDisplay logic

* typo

Co-authored-by: Omer Fahim <[email protected]>

* don't include when include is false

* fix ownerdv null case

* get level once, setlocaldisplayoncreate correctly

* Try order by id

* @GPortas edits

* add index

* updates per review

* comment tweak

* Release 6.6

* Update doc/sphinx-guides/source/versions.rst

Co-authored-by: Philip Durbin <[email protected]>

* 6.6 release fix (#11353)

---------

Co-authored-by: Philip Durbin <[email protected]>
Co-authored-by: Leonid Andreev <[email protected]>
Co-authored-by: Jim Myers <[email protected]>
Co-authored-by: Philip Durbin <[email protected]>
Co-authored-by: Omer Fahim <[email protected]>
Co-authored-by: Alexis Guanche <[email protected]>
Co-authored-by: qqmyers <[email protected]>
Co-authored-by: Oliver Bertuch <[email protected]>
Co-authored-by: Stephen Kraffmiller <[email protected]>
Co-authored-by: Julian Gautier <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

FY25 Sprint 18 FY25 Sprint 18 (2025-02-26 - 2025-03-12) GDCC: DANS related to GDCC work for DANS GDCC: QDR of interest to QDR Size: 10 A percentage of a sprint. 7 hours.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cite Data: Various formats of citation are not accessable via URL

6 participants