Database-independent json export and import#885
Merged
lonvia merged 18 commits intokomoot:masterfrom May 8, 2025
Merged
Conversation
Linked places should never be imported.
lonvia
force-pushed
the
json-dump-exchange-format
branch
from
ab4f843 to
21ac674
Compare
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR replaces the existing backend-dependent json dump code with a generic json dumper that writes out a generalised dump format directly from the PhotonDoc. It then adds an import function to read from these files again. Both, export and import take into account the settings for
-languages,-extra-tags,-countriesand-import-geometry-column. That means that you can on one-side produce dump of different degrees of detail and on the other side, have simple filtering of a large dump in place.The dump file remains a line-delimited json format. Photon will ensure to dump one document per line, so that it is easy to use tools like grep to pre-filter the data when saving or importing the file. On importing, however, it can handle any concatenated json.
There is now also support for a special CountryInfo document which can contain country-level information like country names. When available, Photon can use the information to set the country names in a place document. That saves a lot of duplication in the dump file. Having the country names on each place document separately is supported as well but country info takes precedence.
Dumping the full planet takes about 3.5 hours for the standard configuration and produces a 20GB bzipped-file. A full dump with all features takes 9.5h and has about 33GB. Reading the full dump takes about 11h. There is clearly room for improvement here. To some extend bzipping might have been the limiting factor.
Supersedes #438.
Closes #868.
Closes #291.
Fixes #411.