Add support for disabling feature datasets#6666
Merged
Conversation
include/storage/storage_config.hpp
Outdated
| #include "storage/io_config.hpp" | ||
| #include "osrm/datasets.hpp" | ||
|
|
||
| #include "set" |
Member
There was a problem hiding this comment.
Nit.
Suggested change
| #include "set" | |
| #include <set> |
include/util/exception.hpp
Outdated
| { | ||
| } | ||
|
|
||
| std::string Dataset() const { return dataset; } |
Member
There was a problem hiding this comment.
Nit.
Suggested change
| std::string Dataset() const { return dataset; } | |
| const std::string& Dataset() const { return dataset; } |
SiarheiFedartsou
approved these changes
Jul 31, 2023
mjjbell
force-pushed
the
mbell/disable_feature_dataset
branch
from
3b90f44 to
d34f259
Compare
This change adds support for disabling datasets, such that specific files are not loaded into memory when running OSRM. This enables users to not pay the memory cost for features they do not intend to use. Initially, there are two options: - ROUTE_GEOMETRY, for disabling overview, steps, annotations and waypoints. - ROUTE_STEPS, for disabling steps only. Attempts to query features for which the datasets are disabled will lead to a DisabledDatasetException being returned.
mjjbell
force-pushed
the
mbell/disable_feature_dataset
branch
from
d34f259 to
189f6eb
Compare
Member
Author
|
For completeness, I ran performance tests using the same setup as here with the first three coordinate sets. All requests had the parameters The results show no change in performance with this PR. Tests with request parameters that disabled geometries and steps had the same outcome.
|
eliseier
pushed a commit
to wanderlog/osrm-backend
that referenced
this pull request
Mar 25, 2025
This change adds support for disabling datasets, such that specific files are not loaded into memory when running OSRM. This enables users to not pay the memory cost for features they do not intend to use. Initially, there are two options: - ROUTE_GEOMETRY, for disabling overview, steps, annotations and waypoints. - ROUTE_STEPS, for disabling steps only. Attempts to query features for which the datasets are disabled will lead to a DisabledDatasetException being returned.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue
This change adds support for disabling datasets, such that specific files are not loaded into memory when running OSRM. This enables users to not pay the memory cost for features they do not intend to use.
Initially, there are two options:
ROUTE_GEOMETRY, for disabling overview, steps, annotations and waypoints.ROUTE_STEPS, for disabling steps only.Attempts to query a feature for which the dataset is disabled will lead to a
DisabledDatasetExceptionbeing returned.Full details are on a new wiki page: https://github.com/Project-OSRM/osrm-backend/wiki/Disabled-Datasets
Design Choices
The implementation separates the "feature dataset" concept from the actual files loaded into memory. This gives some flexibility in how the data is stored, allowing for changes in the file structure without breaking the user facing API.
The feature datasets to disable are passed as an array argument to
osrm-routed,osrm-datastoreand the NodeJS engine configuration. Currently, you would only select at most one option (ROUTE_GEOMETRYis a superset ofROUTE_STEPS), but the array interface will allow us to add more options in the future whilst maintaining backwards compatibility.On every data facade call to optional datasets, we check a boolean to see if its loaded. I attempted other solutions that don't perform a check on every call, such as virtual functions and lambdas. These didn't improve performance and were significantly more complicated, so I stuck with the simplest implementation.
Performance
For a contraction hierarchy dataset,
--disable-feature-dataset ROUTE_GEOMETRYreduces RAM usage by ~15%.There appears to be no impact on query performance. This is likely due to the number of data facade calls affected being very small. Will add full query performance results in a subsequent post.
Future Direction
Some optional datasets are stored in files with required datasets. One such example is OSM node IDs being stored with coordinates. We could reduce memory usage further when disabling
ROUTE_GEOMETRYby splitting such files, or refactoring how files are loaded into memory.Tasklist
Requirements / Relations
#5218
#5838
#6045