Do not generate intermediate .osrm file in osrm-extract.#6354
Do not generate intermediate .osrm file in osrm-extract.#6354SiarheiFedartsou merged 17 commits intomasterfrom
Conversation
|
Some things to discuss:
@mjjbell WDYT? |
.github/workflows/osrm-backend.yml
Outdated
| docker run $MEMORY_ARGS -t -v "${PWD}:/data" "${TAG}" osrm-customize /data/berlin-latest.osrm | ||
| docker run $MEMORY_ARGS --name=osrm-container -t -p 5000:5000 -v "${PWD}:/data" "${TAG}" osrm-routed --algorithm mld /data/berlin-latest.osrm & | ||
| docker run $MEMORY_ARGS -t -v "${PWD}:/data" "${TAG}" osrm-extract --dump-nbg-graph -p /opt/car.lua /data/berlin-latest.osm.pbf | ||
| docker run $MEMORY_ARGS -t -v "${PWD}:/data" "${TAG}" osrm-components /data/berlin-latest.osrm /data/berlin-latest.geojson |
There was a problem hiding this comment.
Added kind of smoke test for osrm-components.
| std::vector<InputTrafficSignal> external_traffic_signals; | ||
| TrafficSignals internal_traffic_signals; | ||
|
|
||
| std::vector<NodeBasedEdge> normal_edges; |
There was a problem hiding this comment.
Not sure about naming here, tbh haven't tried to dive into the logic there, just made it available from outside.
There was a problem hiding this comment.
I would go with used_edges, given that's the log message at the point of generation
https://github.com/Project-OSRM/osrm-backend/pull/6354/files#diff-e52614280a87ada8afc739a67e55e7d2cc111d11a9f23fc429b7c2d20dd1d8d7R861
|
|
||
| std::vector<InputManeuverOverride> external_maneuver_overrides_list; | ||
| std::vector<UnresolvedManeuverOverride> internal_maneuver_overrides; | ||
| std::unordered_set<NodeID> internal_barrier_nodes; |
There was a problem hiding this comment.
The same about naming.
There was a problem hiding this comment.
barrier_nodes should be fine. It's just a set of standard node IDs, we haven't converted from a different representation.
There was a problem hiding this comment.
Renamed to used_barrier_nodes as we already use barrier_nodes for barrier's OSM ids.
| std::vector<UnresolvedManeuverOverride>, | ||
| TrafficSignals> | ||
| ParseOSMData(ScriptingEnvironment &scripting_environment, const unsigned number_of_threads); | ||
| ParsedOSMData ParseOSMData(ScriptingEnvironment &scripting_environment, |
There was a problem hiding this comment.
IMO this ParsedOSMData is a bit more readable than tuple we used before.
| auto node_id_iterator = used_node_id_list.begin(); | ||
| const auto all_nodes_list_end = all_nodes_list.end(); | ||
|
|
||
| for (const auto index : util::irange<NodeID>(0, used_node_id_list.size())) |
There was a problem hiding this comment.
We were running it while writing nodes on disk here in WriteNodes method:
Probably can be refactored to be more readable(less iterator juggling etc), but IMO it is safer to keep it with as less changes as possible for now.
| osm_node_ids.push_back(current_node.node_id); | ||
| } | ||
|
|
||
| if (config.dump_nbg_graph) |
There was a problem hiding this comment.
Now we are dumping .osrm file only if it was explicitly requested.
|
|
||
| extractor::files::readRawNBGraph( | ||
| path, nop, coordinate_list, osm_node_ids, edge_list, annotation_data); | ||
| extractor::files::readRawNBGraph(path, coordinate_list, osm_node_ids, edge_list); |
.github/workflows/osrm-backend.yml
Outdated
| exit 1 | ||
| fi | ||
|
|
||
| # here we check that `osrm-partition` accepts base file path with `.osrm` extension even if `.osrm` file doesn't exist |
There was a problem hiding this comment.
I noticed the command args refer to the .osrm file. We probably want to change the wording there too.
osrm-backend/src/tools/customize.cpp
Lines 78 to 80 in 41dda32
| std::vector<InputTrafficSignal> external_traffic_signals; | ||
| TrafficSignals internal_traffic_signals; | ||
|
|
||
| std::vector<NodeBasedEdge> normal_edges; |
There was a problem hiding this comment.
I would go with used_edges, given that's the log message at the point of generation
https://github.com/Project-OSRM/osrm-backend/pull/6354/files#diff-e52614280a87ada8afc739a67e55e7d2cc111d11a9f23fc429b7c2d20dd1d8d7R861
|
|
||
| std::vector<InputManeuverOverride> external_maneuver_overrides_list; | ||
| std::vector<UnresolvedManeuverOverride> internal_maneuver_overrides; | ||
| std::unordered_set<NodeID> internal_barrier_nodes; |
There was a problem hiding this comment.
barrier_nodes should be fine. It's just a set of standard node IDs, we haven't converted from a different representation.
| std::vector<InputManeuverOverride> external_maneuver_overrides_list; | ||
| std::vector<UnresolvedManeuverOverride> internal_maneuver_overrides; | ||
| std::unordered_set<NodeID> internal_barrier_nodes; | ||
| NodeVector internal_nodes; |
There was a problem hiding this comment.
Similarly, I'd go with used_nodes
| // turn the graph into the routing graph to be used with the navigation algorithms. | ||
| NodeBasedGraphFactory(const boost::filesystem::path &input_file, | ||
| ScriptingEnvironment &scripting_environment, | ||
| NodeBasedGraphFactory(ScriptingEnvironment &scripting_environment, |
There was a problem hiding this comment.
Above comment will need updating.
README.md
Outdated
| docker run -t -v "${PWD}:/data" osrm/osrm-backend osrm-customize /data/berlin-latest | ||
|
|
||
| Note that `berlin-latest.osrm` has a different file extension. | ||
| Note that `berlin-latest` has no file extension. |
There was a problem hiding this comment.
Referring just to the base path feels a bit odd, but I don't really have a better suggestion. Maybe always use the OSM file as the argument?
There was a problem hiding this comment.
| docker run -t -v "${PWD}:/data" osrm/osrm-backend osrm-customize /data/berlin-latest.osrm | ||
|
|
||
| Note that `berlin-latest.osrm` has a different file extension. | ||
| Note there is no `berlin-latest.osrm` file, but multiple `berlin-latest.osrm.*` files, i.e. `berlin-latest.osrm` is not file path, but "base" path referring to set of files and there is an option to omit this `.osrm` suffix completely(e.g. `osrm-partition /data/berlin-latest`). |
There was a problem hiding this comment.
@mjjbell I noticed that all our files match .osrm.* pattern, so IMO it feels as a good idea to still use .osrm extension everywhere, but consider it as "base" path, but not separate file. WDYT?
|
@mjjbell can you take a look again? And also let's discuss these questions:
For 1 I doubt someone really relies on it and it is super straightforward to just pass this additional flag if it is really needed. For 3 I now more inclined to leave this |
It's the internal node-based-graph representation prior to compression, creation of dummy edges and other stuff that doesn't exist in the OSM data. So I would say it's the correct arg name, but I agree that it's not obvious that this should generate an Renaming it to
Thinking longer-term, I can see an approach where we pass around a metadata/config file, that includes information about the settings used during processing, and detailing which datasets have been generated (e.g. if we wanted to support optional loading), which maybe could be a good use for an |
Sorry I'm reading everything in the wrong order. If that's the case, then I agree, and it also aligns with the potential of a config/metadata file in the future. |
| docker run -t -v "${PWD}:/data" osrm/osrm-backend osrm-customize /data/berlin-latest.osrm | ||
|
|
||
| Note that `berlin-latest.osrm` has a different file extension. | ||
| Note there is no `berlin-latest.osrm` file, but multiple `berlin-latest.osrm.*` files, i.e. `berlin-latest.osrm` is not file path, but "base" path referring to set of files and there is an option to omit this `.osrm` suffix completely(e.g. `osrm-partition /data/berlin-latest`). |
src/extractor/extractor.cpp
Outdated
|
|
||
| if (config.dump_nbg_graph) | ||
| { | ||
| storage::tar::FileWriter writer(config.GetPath(".osrm").string(), |
There was a problem hiding this comment.
Only suggested change would be to make this .nbg
Agreed. Also, given the |
v5.27.0
- Changes from 5.26.0
- API:
- ADDED: Add Flatbuffers support to NodeJS bindings. [Project-OSRM#6338](Project-OSRM#6338)
- CHANGED: Add `data_version` field to responses of all services. [Project-OSRM#5387](Project-OSRM#5387)
- FIXED: Use Boost.Beast to parse HTTP request. [Project-OSRM#6294](Project-OSRM#6294)
- FIXED: Fix inefficient osrm-routed connection handling [Project-OSRM#6113](https://gihub.com/Project-OSRM/osrm-backend/pull/6113)
- FIXED: Fix HTTP compression precedence [Project-OSRM#6113](Project-OSRM#6113)
- NodeJS:
- FIXED: Support `skip_waypoints` in Node bindings [Project-OSRM#6060](Project-OSRM#6060)
- Misc:
- ADDED: conanbuildinfo.json for easy reading of dependencies [Project-OSRM#6388](Project-OSRM#6388)
- CHANGED: Improve performance of JSON rendering. Fix undefined behaviour in JSON numbers formatting. [Project-OSRM#6380](Project-OSRM#6380)
- ADDED: Add timestamps for logs. [Project-OSRM#6375](Project-OSRM#6375)
- CHANGED: Improve performance of map matching via getPathDistance optimization. [Project-OSRM#6378](Project-OSRM#6378)
- CHANGED: Optimize RestrictionParser performance. [Project-OSRM#6344](Project-OSRM#6344)
- ADDED: Support floats for speed value in traffic updates CSV. [Project-OSRM#6327](Project-OSRM#6327)
- CHANGED: Use Lua 5.4 in Docker image. [Project-OSRM#6346](Project-OSRM#6346)
- CHANGED: Remove redundant nullptr check. [Project-OSRM#6326](Project-OSRM#6326)
- CHANGED: missing files list is included in exception message. [Project-OSRM#5360](Project-OSRM#5360)
- CHANGED: Do not use deprecated Callback::Call overload in Node bindings. [Project-OSRM#6318](Project-OSRM#6318)
- FIXED: Fix distance calculation consistency. [Project-OSRM#6315](Project-OSRM#6315)
- FIXED: Fix performance issue after migration to sol2 3.3.0. [Project-OSRM#6304](Project-OSRM#6304)
- CHANGED: Pass osm_node_ids by reference in osrm::updater::Updater class. [Project-OSRM#6298](Project-OSRM#6298)
- FIXED: Fix bug with reading Set values from Lua scripts. [Project-OSRM#6285](Project-OSRM#6285)
- FIXED: Bug in bicycle profile that caused exceptions if there is a highway=bicycle in the data. [Project-OSRM#6296](Project-OSRM#6296)
- FIXED: Internal refactoring of identifier types used in data facade [Project-OSRM#6044](Project-OSRM#6044)
- CHANGED: Update docs to reflect recent build and dependency changes [Project-OSRM#6383](Project-OSRM#6383)
- Build:
- REMOVED: Get rid of Mason. [Project-OSRM#6387](Project-OSRM#6387)
- CHANGED: Use clang-format from CI base image. [Project-OSRM#6391](Project-OSRM#6391)
- ADDED: Build Node bindings on Windows. [Project-OSRM#6334](Project-OSRM#6334)
- ADDED: Configure cross-compilation for Apple Silicon. [Project-OSRM#6360](Project-OSRM#6360)
- CHANGED: Use apt-get to install Clang on CI. [Project-OSRM#6345](Project-OSRM#6345)
- CHANGED: Fix TBB in case of Conan + NodeJS build. [Project-OSRM#6333](Project-OSRM#6333)
- CHANGED: Migrate to modern TBB version. [Project-OSRM#6300](Project-OSRM#6300)
- CHANGED: Enable performance-move-const-arg clang-tidy check. [Project-OSRM#6319](Project-OSRM#6319)
- CHANGED: Use the latest node on CI. [Project-OSRM#6317](Project-OSRM#6317)
- CHANGED: Migrate Windows CI to GitHub Actions. [Project-OSRM#6312](Project-OSRM#6312)
- ADDED: Add smoke test for Docker image. [Project-OSRM#6313](Project-OSRM#6313)
- CHANGED: Update libosmium to version 2.18.0. [Project-OSRM#6303](Project-OSRM#6303)
- CHANGED: Remove EXACT from find_package if using Conan. [Project-OSRM#6299](Project-OSRM#6299)
- CHANGED: Configure Undefined Behaviour Sanitizer. [Project-OSRM#6290](Project-OSRM#6290)
- CHANGED: Use Conan instead of Mason to install code dependencies. [Project-OSRM#6284](Project-OSRM#6284)
- CHANGED: Migrate to C++17. Update sol2 to 3.3.0. [Project-OSRM#6279](Project-OSRM#6279)
- CHANGED: Update macOS CI image to macos-11. [Project-OSRM#6286](Project-OSRM#6286)
- CHANGED: Enable even more clang-tidy checks. [Project-OSRM#6273](Project-OSRM#6273)
- CHANGED: Configure CMake to not build flatbuffers tests and samples. [Project-OSRM#6274](Project-OSRM#6274)
- CHANGED: Enable more clang-tidy checks. [Project-OSRM#6270](Project-OSRM#6270)
- CHANGED: Configure clang-tidy job on CI. [Project-OSRM#6261](Project-OSRM#6261)
- CHANGED: Use Github Actions for building container images [Project-OSRM#6138](Project-OSRM#6138)
- CHANGED: Upgrade Boost dependency to 1.70 [Project-OSRM#6113](Project-OSRM#6113)
- CHANGED: Upgrade Ubuntu CI builds to 20.04 [Project-OSRM#6119](Project-OSRM#6119)
- CHANGED: Make building osrm-routed optional [Project-OSRM#6144](Project-OSRM#6144)
- FIXED: Run all unit tests in CI [Project-OSRM#5248](Project-OSRM#5248)
- FIXED: Fix installation of Mason CMake and 32 bit CI build [Project-OSRM#6170](Project-OSRM#6170)
- FIXED: Fixed Node docs generation check in CI. [Project-OSRM#6058](Project-OSRM#6058)
- CHANGED: Docker build, enabled arm64 build layer [Project-OSRM#6172](Project-OSRM#6172)
- CHANGED: Docker build, enabled apt-get update/install caching in separate layer for build phase [Project-OSRM#6175](Project-OSRM#6175)
- FIXED: Bump CI complete meta job to ubuntu-20.04 [Project-OSRM#6323](Project-OSRM#6323)
- CHANGED: Node packages are now scoped by @Project-OSRM [Project-OSRM#6386](Project-OSRM#6386)
- Routing:
- CHANGED: Lazily generate optional route path data [Project-OSRM#6045](Project-OSRM#6045)
- FIXED: Completed support for no_entry and no_exit turn restrictions. [Project-OSRM#5988](Project-OSRM#5988)
- ADDED: Add support for non-round-trips with a single fixed endpoint. [Project-OSRM#6050](Project-OSRM#6050)
- FIXED: Improvements to maneuver override processing [Project-OSRM#6125](Project-OSRM#6125)
- ADDED: Support snapping to multiple ways at an input location. [Project-OSRM#5953](Project-OSRM#5953)
- FIXED: Fix snapping target locations to ways used in turn restrictions. [Project-OSRM#6339](Project-OSRM#6339)
- ADDED: Support OSM traffic signal directions. [Project-OSRM#6153](Project-OSRM#6153)
- FIXED: Ensure u-turn exists in intersection view. [Project-OSRM#6376](Project-OSRM#6376)
- FIXED: Gracefully handle no-turn intersections in guidance processing. [Project-OSRM#6382](Project-OSRM#6382)
- Profile:
- CHANGED: Bicycle surface speeds [Project-OSRM#6212](Project-OSRM#6212)
- Tools:
- CHANGED: Do not generate intermediate .osrm file in osrm-extract. [Project-OSRM#6354](Project-OSRM#6354)
v5.27.0
- Changes from 5.26.0
- API:
- ADDED: Add Flatbuffers support to NodeJS bindings. [Project-OSRM#6338](Project-OSRM#6338)
- CHANGED: Add `data_version` field to responses of all services. [Project-OSRM#5387](Project-OSRM#5387)
- FIXED: Use Boost.Beast to parse HTTP request. [Project-OSRM#6294](Project-OSRM#6294)
- FIXED: Fix inefficient osrm-routed connection handling [Project-OSRM#6113](https://gihub.com/Project-OSRM/osrm-backend/pull/6113)
- FIXED: Fix HTTP compression precedence [Project-OSRM#6113](Project-OSRM#6113)
- NodeJS:
- FIXED: Support `skip_waypoints` in Node bindings [Project-OSRM#6060](Project-OSRM#6060)
- Misc:
- ADDED: conanbuildinfo.json for easy reading of dependencies [Project-OSRM#6388](Project-OSRM#6388)
- CHANGED: Improve performance of JSON rendering. Fix undefined behaviour in JSON numbers formatting. [Project-OSRM#6380](Project-OSRM#6380)
- ADDED: Add timestamps for logs. [Project-OSRM#6375](Project-OSRM#6375)
- CHANGED: Improve performance of map matching via getPathDistance optimization. [Project-OSRM#6378](Project-OSRM#6378)
- CHANGED: Optimize RestrictionParser performance. [Project-OSRM#6344](Project-OSRM#6344)
- ADDED: Support floats for speed value in traffic updates CSV. [Project-OSRM#6327](Project-OSRM#6327)
- CHANGED: Use Lua 5.4 in Docker image. [Project-OSRM#6346](Project-OSRM#6346)
- CHANGED: Remove redundant nullptr check. [Project-OSRM#6326](Project-OSRM#6326)
- CHANGED: missing files list is included in exception message. [Project-OSRM#5360](Project-OSRM#5360)
- CHANGED: Do not use deprecated Callback::Call overload in Node bindings. [Project-OSRM#6318](Project-OSRM#6318)
- FIXED: Fix distance calculation consistency. [Project-OSRM#6315](Project-OSRM#6315)
- FIXED: Fix performance issue after migration to sol2 3.3.0. [Project-OSRM#6304](Project-OSRM#6304)
- CHANGED: Pass osm_node_ids by reference in osrm::updater::Updater class. [Project-OSRM#6298](Project-OSRM#6298)
- FIXED: Fix bug with reading Set values from Lua scripts. [Project-OSRM#6285](Project-OSRM#6285)
- FIXED: Bug in bicycle profile that caused exceptions if there is a highway=bicycle in the data. [Project-OSRM#6296](Project-OSRM#6296)
- FIXED: Internal refactoring of identifier types used in data facade [Project-OSRM#6044](Project-OSRM#6044)
- CHANGED: Update docs to reflect recent build and dependency changes [Project-OSRM#6383](Project-OSRM#6383)
- Build:
- REMOVED: Get rid of Mason. [Project-OSRM#6387](Project-OSRM#6387)
- CHANGED: Use clang-format from CI base image. [Project-OSRM#6391](Project-OSRM#6391)
- ADDED: Build Node bindings on Windows. [Project-OSRM#6334](Project-OSRM#6334)
- ADDED: Configure cross-compilation for Apple Silicon. [Project-OSRM#6360](Project-OSRM#6360)
- CHANGED: Use apt-get to install Clang on CI. [Project-OSRM#6345](Project-OSRM#6345)
- CHANGED: Fix TBB in case of Conan + NodeJS build. [Project-OSRM#6333](Project-OSRM#6333)
- CHANGED: Migrate to modern TBB version. [Project-OSRM#6300](Project-OSRM#6300)
- CHANGED: Enable performance-move-const-arg clang-tidy check. [Project-OSRM#6319](Project-OSRM#6319)
- CHANGED: Use the latest node on CI. [Project-OSRM#6317](Project-OSRM#6317)
- CHANGED: Migrate Windows CI to GitHub Actions. [Project-OSRM#6312](Project-OSRM#6312)
- ADDED: Add smoke test for Docker image. [Project-OSRM#6313](Project-OSRM#6313)
- CHANGED: Update libosmium to version 2.18.0. [Project-OSRM#6303](Project-OSRM#6303)
- CHANGED: Remove EXACT from find_package if using Conan. [Project-OSRM#6299](Project-OSRM#6299)
- CHANGED: Configure Undefined Behaviour Sanitizer. [Project-OSRM#6290](Project-OSRM#6290)
- CHANGED: Use Conan instead of Mason to install code dependencies. [Project-OSRM#6284](Project-OSRM#6284)
- CHANGED: Migrate to C++17. Update sol2 to 3.3.0. [Project-OSRM#6279](Project-OSRM#6279)
- CHANGED: Update macOS CI image to macos-11. [Project-OSRM#6286](Project-OSRM#6286)
- CHANGED: Enable even more clang-tidy checks. [Project-OSRM#6273](Project-OSRM#6273)
- CHANGED: Configure CMake to not build flatbuffers tests and samples. [Project-OSRM#6274](Project-OSRM#6274)
- CHANGED: Enable more clang-tidy checks. [Project-OSRM#6270](Project-OSRM#6270)
- CHANGED: Configure clang-tidy job on CI. [Project-OSRM#6261](Project-OSRM#6261)
- CHANGED: Use Github Actions for building container images [Project-OSRM#6138](Project-OSRM#6138)
- CHANGED: Upgrade Boost dependency to 1.70 [Project-OSRM#6113](Project-OSRM#6113)
- CHANGED: Upgrade Ubuntu CI builds to 20.04 [Project-OSRM#6119](Project-OSRM#6119)
- CHANGED: Make building osrm-routed optional [Project-OSRM#6144](Project-OSRM#6144)
- FIXED: Run all unit tests in CI [Project-OSRM#5248](Project-OSRM#5248)
- FIXED: Fix installation of Mason CMake and 32 bit CI build [Project-OSRM#6170](Project-OSRM#6170)
- FIXED: Fixed Node docs generation check in CI. [Project-OSRM#6058](Project-OSRM#6058)
- CHANGED: Docker build, enabled arm64 build layer [Project-OSRM#6172](Project-OSRM#6172)
- CHANGED: Docker build, enabled apt-get update/install caching in separate layer for build phase [Project-OSRM#6175](Project-OSRM#6175)
- FIXED: Bump CI complete meta job to ubuntu-20.04 [Project-OSRM#6323](Project-OSRM#6323)
- CHANGED: Node packages are now scoped by @Project-OSRM [Project-OSRM#6386](Project-OSRM#6386)
- Routing:
- CHANGED: Lazily generate optional route path data [Project-OSRM#6045](Project-OSRM#6045)
- FIXED: Completed support for no_entry and no_exit turn restrictions. [Project-OSRM#5988](Project-OSRM#5988)
- ADDED: Add support for non-round-trips with a single fixed endpoint. [Project-OSRM#6050](Project-OSRM#6050)
- FIXED: Improvements to maneuver override processing [Project-OSRM#6125](Project-OSRM#6125)
- ADDED: Support snapping to multiple ways at an input location. [Project-OSRM#5953](Project-OSRM#5953)
- FIXED: Fix snapping target locations to ways used in turn restrictions. [Project-OSRM#6339](Project-OSRM#6339)
- ADDED: Support OSM traffic signal directions. [Project-OSRM#6153](Project-OSRM#6153)
- FIXED: Ensure u-turn exists in intersection view. [Project-OSRM#6376](Project-OSRM#6376)
- FIXED: Gracefully handle no-turn intersections in guidance processing. [Project-OSRM#6382](Project-OSRM#6382)
- Profile:
- CHANGED: Bicycle surface speeds [Project-OSRM#6212](Project-OSRM#6212)
- Tools:
- CHANGED: Do not generate intermediate .osrm file in osrm-extract. [Project-OSRM#6354](Project-OSRM#6354)
Issue
closes #6329
Tasklist
Requirements / Relations
Link any requirements here. Other pull requests this PR is based on?