Metadata fields can be "display on create" per collection #11312

Saixel · 2025-03-06T13:34:19Z

This PR restores the previous changes for metadata block handling in #11224 that were reverted, and addresses the test failures that led to the revert. The changes include:

Restore of the original metadata block changes
Investigation and fix of the test failures
Analysis of the displayOnCreate field behavior in metadata blocks

See #11224 for screenshots and more details about the feature.

Current status: Investigating the root cause of the test failures to ensure proper functionality of the restored changes.

…reate-field-option-new

qqmyers · 2025-03-06T14:47:55Z

src/main/java/edu/harvard/iq/dataverse/MetadataBlockServiceBean.java

+                .where(
+                    criteriaBuilder.equal(subqueryRoot.get("dataverse"), dataverseRoot),
+                    criteriaBuilder.equal(subqueryRoot.get("datasetFieldType"), datasetFieldTypeJoin)
+                );


I suggested changing this in the earlier PR since, at that point, displayOnCreate could be false incorrectly due to the default of setting it false in the code changes above. Is this still an issue - i.e. the default should be null and this subquery should only return 1 if displayOnCreate is not null (so that line 84 below lets you get the default value)?

@qqmyers You raise a good point. Currently, when no input level is configured, displayOnCreate defaults to false, which might hide fields that should be shown by default. We should consider making displayOnCreate nullable in DatasetFieldType and only setting it to false when explicitly configured in the input level. This would allow the default displayOnCreate value from the field type to be used when no input level is specified. Do you agree? Or am I missing something?

Yes - that's what I was suggesting - I think you then have to adjust this subquery to add a check for displayOnCreate not null so that you get the default when it is null.

@qqmyers Changes added in the commit 15fc52c

Looks good!

Sorry - Or maybe not - I think the nullable Boolean is needed on the DataverseFieldTypeInputLevel and not on the DatasetFieldType - the field type will always be true or false, but the input level might not be set it at all.

- Added @column annotation for displayOnCreate field in DatasetFieldType - Updated JsonPrinter to include displayOnCreate in display condition

Saixel · 2025-03-06T15:00:58Z

First fix: Resolved an issue where the displayOnCreate field was not being properly handled in the API response. The problem was in the JsonPrinter logic, which wasn't correctly processing the displayOnCreate property when generating the JSON response.

Specific Changes:

Modified the JsonPrinter logic to properly handle the displayOnCreate property in the JSON output
Verified that the field is correctly included in the API response
Confirmed that the testLinkInstrumentToAstro test passes successfully

Verification:

The testLinkInstrumentToAstro test passes successfully
Verified that the astroInstrument field's displayOnCreate property is correctly included in the API response

Saixel · 2025-03-06T15:09:17Z

The test testListMetadataBlocks in DataversesIT shows different results depending on the test execution order:

Behavior:

When run in isolation (with a clean application state), the test passes successfully
When run after testLinkInstrumentToAstro, the test fails because it receives 3 metadata blocks instead of the expected 2

Root Cause:
The issue occurs because testLinkInstrumentToAstro modifies the displayOnCreate property of the astrophysics metadata block to true. This state change persists in the application and affects subsequent tests. When testListMetadataBlocks is executed, it receives the modified state where astrophysics has displayOnCreate=true, causing the test to fail.

Impact:
This reveals a test dependency issue where tests are not properly isolated from each other. While the functionality works correctly in a clean state, the test suite's reliability is compromised by these hidden dependencies. Also, I have checked and this also happens in develop without the new feature changes.

src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java

doc/sphinx-guides/source/api/native-api.rst

src/test/java/edu/harvard/iq/dataverse/api/DataversesIT.java

coveralls · 2025-03-06T16:15:10Z

coverage: 22.729% (-0.01%) from 22.741%
when pulling 404eb0b on 10476-display-on-create-field-option-new
into 7a29522 on develop.

coveralls · 2025-03-06T16:15:26Z

coverage: 22.595% (-0.02%) from 22.614%
when pulling 56cfa92 on 10476-display-on-create-field-option-new
into 8553d77 on develop.

- Clarify that required fields are always displayed regardless of displayOnCreate setting

Saixel · 2025-03-06T18:53:03Z

It seems that the build is still failing. I've run the 4 failing tests locally and they all pass successfully:

testLinkInstrumentToAstro (DatasetTypesIT)
testUpdateDatasetTypeLinksWithMetadataBlocks (DatasetTypesIT)
testSummaryDatasetVersionsDifferencesAPI (DatasetsIT)
testListMetadataBlocks (MetadataBlocksIT)

This suggests there might be an environment difference in Jenkins causing these failures. Could you please check the Jenkins logs for any specific details about why these tests are failing? Any hints or tips on how I should proceed would be very helpful.

I have run the tests one by one by restarting the application. I did the tests in the commit 3c4e5cc.

- Change displayOnCreate to nullable Boolean in DatasetFieldType - Update API and service methods to handle null displayOnCreate values - Modify native API documentation to explain null displayOnCreate behavior - Add null checks in MetadataBlockServiceBean queries

- Modified JsonPrinter to handle null values for displayOnCreate - Updated BriefJsonPrinter to use null-safe comparison

- Update MetadataBlock to safely check displayOnCreate with null values - Modify JsonPrinter to default displayOnCreate to false when null - Ensure consistent null-safe handling of displayOnCreate across components

…seServiceBean

github-actions · 2025-03-13T15:23:45Z

📦 Pushed preview images as

ghcr.io/gdcc/dataverse:10476-display-on-create-field-option-new

ghcr.io/gdcc/configbaker:10476-display-on-create-field-option-new

🚢 See on GHCR. Use by referencing with full name as printed above, mind the registry name.

sekmiller · 2025-03-13T19:03:18Z

doc/sphinx-guides/source/api/native-api.rst

 - ``include``: Whether the field is included (boolean)
+- ``displayOnCreate`` (optional): Whether the field is displayed during dataset creation, even when not required (boolean)

 .. code-block:: bash


by the way - the "fully expanded" example down below here has bad data - "required": true, "included": false

qqmyers · 2025-03-13T19:18:12Z

doc/sphinx-guides/source/api/native-api.rst

 - ``datasetFieldTypeName``: Name of the metadata field
 - ``required``: Whether the field is required (boolean)
 - ``include``: Whether the field is included (boolean)
+- ``displayOnCreate`` (optional): Whether the field is displayed during dataset creation, even when not required (boolean)


I don't think the code currently allows this to be optional - line 806 in Dataverses sets it to false by default. I've tried to handle this in #11330, which should be ~ up to date with this PR, but still checking to see if I've broken tests. (I also added a test updating an inputLevel with no displayOnCreate value in that PR).

for the doc can we just say that it default to false if omitted?

I'm the dummy who wrote the migration script and didn't default the column to false. Should I add another update? Are we ready for more flyway migration errors?

This would be a backward incompatibility - if you don't include displayOnCreate you will override any true on the datasetfieldtype itself. If that's OK, then yes a doc change could address it.

You're right. didn't think of that.
\

Would it be better if it's omitted that the value would come from the table for that datasetFieldType?

sekmiller

Saw a minor doc issue - noted below, but it existed prior to this PR.

Thanks for all of your hard work on this, Alexis

ofahimIQSS · 2025-03-13T20:53:22Z

adding testing evidence:

tests are passing:

ofahimIQSS · 2025-03-13T21:21:55Z

Merging PR - no issues found.

Saixel added 3 commits March 6, 2025 12:09

Merge develop without revert

5b37919

Merge remote-tracking branch 'origin/develop' into 10476-display-on-c…

9518acd

…reate-field-option-new

Restore displayOnCreate field option changes

404eb0b

Saixel added NIH CAFE Issues related to and/or funded by the NIH CAFE project FY25 Sprint 18 FY25 Sprint 18 (2025-02-26 - 2025-03-12) labels Mar 6, 2025

Saixel added this to the 6.6 milestone Mar 6, 2025

Saixel requested a review from pdurbin March 6, 2025 13:34

Saixel self-assigned this Mar 6, 2025

Saixel added this to IQSS Dataverse Project Mar 6, 2025

Saixel requested a review from qqmyers March 6, 2025 13:34

Saixel moved this to In Progress 💻 in IQSS Dataverse Project Mar 6, 2025

This comment has been minimized.

Sign in to view

qqmyers reviewed Mar 6, 2025

View reviewed changes

Add displayOnCreate option for dataset field types

97b454d

- Added @column annotation for displayOnCreate field in DatasetFieldType - Updated JsonPrinter to include displayOnCreate in display condition

This comment has been minimized.

Sign in to view

sekmiller reviewed Mar 6, 2025

View reviewed changes

src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java Outdated Show resolved Hide resolved

sekmiller reviewed Mar 6, 2025

View reviewed changes

doc/sphinx-guides/source/api/native-api.rst Outdated Show resolved Hide resolved

pdurbin reviewed Mar 6, 2025

View reviewed changes

src/test/java/edu/harvard/iq/dataverse/api/DataversesIT.java Show resolved Hide resolved

Update DataversesIT test to modify metadata block listing parameter

3c4e5cc

This comment has been minimized.

Sign in to view

Update native API documentation for displayOnCreate field option

190f604

- Clarify that required fields are always displayed regardless of displayOnCreate setting

Saixel and others added 4 commits March 6, 2025 19:23

fix: make displayOnCreate nullable in DatasetFieldType

df39fb4

- Modified JsonPrinter to handle null values for displayOnCreate - Updated BriefJsonPrinter to use null-safe comparison

setDisplayOnCreate astroInstrument back to false #10476

944911b

Handle null displayOnCreate in MetadataBlock and JsonPrinter

1db344b

- Update MetadataBlock to safely check displayOnCreate with null values - Modify JsonPrinter to default displayOnCreate to false when null - Ensure consistent null-safe handling of displayOnCreate across components

This comment has been minimized.

Sign in to view

Saixel added 2 commits March 13, 2025 15:12

Remove default displayOnCreate setting for required fields in Dataver…

e31db3d

…seServiceBean

Merge branch 'develop' into 10476-display-on-create-field-option-new

4085c14

This comment has been minimized.

Sign in to view

Saixel requested review from pdurbin and sekmiller March 13, 2025 16:54

Saixel moved this from In Progress 💻 to Ready for Review ⏩ in IQSS Dataverse Project Mar 13, 2025

pdurbin marked this pull request as ready for review March 13, 2025 18:18

pdurbin changed the title ~~fix: Restore metadata block changes and fix test issues~~ Metadata fields can be set to "display on create" per collection Mar 13, 2025

pdurbin changed the title ~~Metadata fields can be set to "display on create" per collection~~ Metadata fields can be "display on create" per collection Mar 13, 2025

pdurbin unassigned Saixel Mar 13, 2025

sekmiller reviewed Mar 13, 2025

View reviewed changes

qqmyers reviewed Mar 13, 2025

View reviewed changes

sekmiller approved these changes Mar 13, 2025

View reviewed changes

github-project-automation bot moved this from Ready for Review ⏩ to Ready for QA ⏩ in IQSS Dataverse Project Mar 13, 2025

ofahimIQSS self-assigned this Mar 13, 2025

ofahimIQSS moved this from Ready for QA ⏩ to QA ✅ in IQSS Dataverse Project Mar 13, 2025

ofahimIQSS merged commit 1480dbc into develop Mar 13, 2025
12 checks passed

github-project-automation bot moved this from QA ✅ to Merged 🚀 in IQSS Dataverse Project Mar 13, 2025

ofahimIQSS deleted the 10476-display-on-create-field-option-new branch March 13, 2025 21:22

ofahimIQSS removed their assignment Mar 13, 2025

ofahimIQSS mentioned this pull request Mar 14, 2025

Revert "Metadata fields can be "display on create" per collection" #11346

Closed

qqmyers mentioned this pull request Mar 17, 2025

DisplayOnCreate Fixes #11343

Merged

scolapasta moved this from Merged 🚀 to Done 🧹 in IQSS Dataverse Project Mar 18, 2025

This was referenced Apr 1, 2025

Project: NIH CAFE IQSS/dataverse-pm#161

Open

Input Level API overwrites existing displayOnCreate settings #11387

Closed

Fields with displayOnCreate setting not visible if collection has a default template #11391

Closed

Metadata fields can be "display on create" per collection #11312

Metadata fields can be "display on create" per collection #11312

Uh oh!

Conversation

Saixel commented Mar 6, 2025 • edited by pdurbin Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

qqmyers Mar 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qqmyers Mar 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qqmyers Mar 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Saixel commented Mar 6, 2025

Uh oh!

This comment has been minimized.

Saixel commented Mar 6, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

coveralls commented Mar 6, 2025

Uh oh!

coveralls commented Mar 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

Saixel commented Mar 6, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Mar 13, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sekmiller left a comment

Choose a reason for hiding this comment

Uh oh!

ofahimIQSS commented Mar 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ofahimIQSS commented Mar 13, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Saixel commented Mar 6, 2025 •

edited by pdurbin

Loading

qqmyers Mar 6, 2025 •

edited

Loading

qqmyers Mar 6, 2025 •

edited

Loading

qqmyers Mar 7, 2025 •

edited

Loading

coveralls commented Mar 6, 2025 •

edited

Loading

ofahimIQSS commented Mar 13, 2025 •

edited

Loading