TDL: Provide guidance for site admins w.r.t. big data #11850
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
scolapasta
moved this from Ready for Triage
to Ready for Review ⏩
in IQSS Dataverse Project
pdurbin
reviewed
Nov 13, 2025
Member
pdurbin
left a comment
pdurbin
left a comment
There was a problem hiding this comment.
Wow, this guide is incredible. 🎉
I know I'm leaving many, many annoying nitpicky comment but I hope they are outweighed by some useful ones! 71 comments total! Sorry! 😅
Docs like this set Dataverse apart from other platforms. How are people supposed to use you software if you don't tell them how??! Great job!
| - DatasetChecksumValidationSizeLimit - by default, Dataverse checks fixity (assuring the file contents match the recorded checksum) as part of publication. This setting specifies a maximum aggregate dataset size, above which validation will not be done. | ||
| - DataFileChecksumValidationSizeLimit - by default, Dataverse checks fixity (assuring the file contents match the recorded checksum) as part of publication. This setting specifies a maximum file size, above which validation will not be done. | ||
| - FilePIDsEnabled - false is recommended when datasets have many files. Related settings allow file PIDS to be enabled/disabled per collection and per file | ||
| - CustomZipDownloadServiceUrl - allows use of a separate process/machine to handle zipping up multi-file downloads. Requires installation of the separate Zip Download app. |
Member
There was a problem hiding this comment.
Suggested change
| - CustomZipDownloadServiceUrl - allows use of a separate process/machine to handle zipping up multi-file downloads. Requires installation of the separate Zip Download app. | |
| - CustomZipDownloadServiceUrl - allows use of a separate process/machine to handle zipping up multi-file downloads. Requires installation of the separate Zip Download app |
for consistency
| - DataFileChecksumValidationSizeLimit - by default, Dataverse checks fixity (assuring the file contents match the recorded checksum) as part of publication. This setting specifies a maximum file size, above which validation will not be done. | ||
| - FilePIDsEnabled - false is recommended when datasets have many files. Related settings allow file PIDS to be enabled/disabled per collection and per file | ||
| - CustomZipDownloadServiceUrl - allows use of a separate process/machine to handle zipping up multi-file downloads. Requires installation of the separate Zip Download app. | ||
| - WebloaderUrl - enables use of an installed DVWebloader (by specifying it's web location) which is more efficient for uploading many files |
Member
There was a problem hiding this comment.
Suggested change
| - WebloaderUrl - enables use of an installed DVWebloader (by specifying it's web location) which is more efficient for uploading many files | |
| - WebloaderUrl - enables use of an installed DVWebloader (by specifying its web location) which is more efficient for uploading many files |
| - DisableSolrFacets - disables facets, which are costly to generate, in search results (including the main collection page) | ||
| - DisableSolrFacetsForGuestUsers - only disable facets for guests | ||
| - DisableSolrFacetsWithoutJsession - disables facets for users who have disabled cookies (e.g. for bots) | ||
| - DisableUncheckedTypesFacet -only disables the facet showing the number of collections, datasets, files matching the query (this facet is potentially less useful than others) |
Member
There was a problem hiding this comment.
Suggested change
| - DisableUncheckedTypesFacet -only disables the facet showing the number of collections, datasets, files matching the query (this facet is potentially less useful than others) | |
| - DisableUncheckedTypesFacet - only disables the facet showing the number of collections, datasets, files matching the query (this facet is potentially less useful than others) |
| - Investigate performance tuning options for Payara, Solr, and Postgres | ||
| - Coordinate with others in the community - there is a lot of aggregate knowledge | ||
| - Consider contributing to software design changes - Dataverse scaling has improved dramatically over the past several years, but more can be done | ||
| - Watch for the new single page application (SPA) front-end for Dataverse. It includes features such as infinite scrolling through files with much faster initial page load times |
Member
There was a problem hiding this comment.
Just a thought, we could have a section at the end called "Resources" or "Getting Involved" that links to https://www.gdcc.io/working-groups/large-data-support.html and #large-data. It could also invite people to contribute to this guide.
Member
Author
There was a problem hiding this comment.
Seems like something to keep centralized?
Co-authored-by: Philip Durbin <[email protected]>
Member
Author
|
Thanks for the detailed read. Hopefully I addressed everything in some way or other. |
github-project-automation
bot
moved this from In Review 🔎
to Ready for QA ⏩
in IQSS Dataverse Project
github-project-automation
bot
moved this from Ready for QA ⏩
to Merged 🚀
in IQSS Dataverse Project
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
This PR adds a Big Data Admin guide that tries to gather information from other parts of the guides into a more coherent guide for managing a Dataverse instance being used for larger data files, more files per dataset, and/or more datasets.
A work in progress, but hopefully useful.
Preview at https://dataverse-guide--11850.org.readthedocs.build/en/11850/admin/big-data-administration.html
Which issue(s) this PR closes:
Special notes for your reviewer:
Suggestions on how to test this:
Does this PR introduce a user interface change? If mockups are available, please link/include them here:
Is there a release notes update needed for this change?:
Additional documentation: