Repository navigation
[QMS-1281] Support Mapsforge POI files of version 3 and 4 - #1283
Merged
Merged
Conversation
Mapsforge POI version 3 replaced the R-tree poi_index (minLat, maxLat, minLon, maxLon) with a plain table (lat, lon) and one index per column. Version 4 brought the R-tree back and added a full-text index and an index on poi_category_map.category. The other tables QMapShack reads are the same in all three versions. OpenAndroMaps ships version 3 by default today, version 2 only in its "mapsforge_old" folder. The queries move to CPoiFilePOIQuery and are picked by the version in the metadata table. Versions 2 and 4 keep the existing R-tree query per category and 0.1 degree square. Without an R-tree every query walks all POIs of the full 0.1 degree latitude row and looks up their categories, whatever longitude range and how many categories are asked for. Version 2's way of loading, one query per category and square, is therefore 10 to 40 times slower on a version 3 file. For version 3 a single query now loads the row segment in view for all checked categories that still miss it. On OpenAndroMaps' Belgium file and a view around Durbuy with the eleven "Transport Car" categories checked, loading takes 0.4 s with version 3 against 0.2 to 0.3 s with version 2 and 4. The writer adds a normalized_name tag (lower case, no accents) for searching. It is no longer shown as an alternative name or as a tag.
Collaborator
|
Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is the linked issue for this pull request:
QMS-#1281
What you have done:
QMapShack reads Mapsforge POI files of version 2, 3 and 4 (OpenAndroMaps ships version 3 by default now, version 4 in
mapsforge_V4/).poi/CPoiFilePOIQuery, which picks the query bymetadata.version.poi_indexis(id, lat, lon)with one index per column). Loading it the v2 way (one query per category and square) is 10–40× slower, because every query walks the whole 0.1° latitude row. Instead, one query loads the visible part of a row for all checked categories that still miss it. The join order is fixed withCROSS JOINand a unary+keeps SQLite off the lon/category indexes (otherwise a single query takes 7–15 s).normalized_nametag the v3/v4 writer adds (lower case, no accents) is no longer shown as an alternative name or tag in the tooltip.Timing with OpenAndroMaps' Belgium file, view around Durbuy, the eleven "Transport Car" categories checked: v2 0.18–0.30 s, v4 ~0.2 s, v3 ~0.4 s (v3 loaded per square: ~5.5 s). I know performance was the reason an earlier v3 attempt was not taken; if 0.4 s is still too slow, a fallback would be a one-time R-tree file next to the POI file (Belgium: 7.8 s, 41 MB) — happy to do that instead.
Checks: for 12 categories over latitude rows 50.0–51.0° N every square returns the same POIs from v2 and v3 (2 extra in v3, exactly on a square edge, which the v2 R-tree misses because it stores 32-bit floats). v2, v3 and v4 show the same POIs on the map; v2 and v4 pixel-identical. A QtTest with tiny v2/v3/v4 files exists locally but is not part of this PR, as QMapShack has no test setup (see also QMS-1280 / #1282, which proposes one).
Built on dev (Qt 6.11.2, GDAL 3.13.2, PROJ 9.4.1), no new dependencies; clang-format applied (also fixes the old tabs in
CPoiFilePOI.cpp), clang-tidy and clazy clean on the changed lines apart fromqstring-allocations(README_CODING asks not to use QStringLiteral by habit). The wiki pages DocGisItemsPOI ("only version 2") and the FAQ's v3→v2 script will need an update after a merge.Steps to perform a simple smoke test:
normalized_name..../pois/mapsforge_V4/europe/Belgium.Poi.zip(version 4) and.../mapsforge_old/europe/Belgium.PoiOld.zip(version 2): same POIs.Does the code comply to the coding rules and naming conventions Coding Guideline:
Is every user facing string in a tr() macro?
Did you add the ticket number and title into the changelog? Keep the numeric order in each release block.