Use tar-format to encapsulate data#4960

Merged

TheMarex merged 53 commits intomasterfrom

refactor/tar_files

Mar 27, 2018

Member

TheMarex commented Mar 15, 2018 •

edited

Loading

Issue

This PR is another step towards implementing #4952. The goal is to make any .osrm.* file a tar file internally and address data by (file)name. As example the .mldgr file now looks like this:

/mld/multilevelgraph/node_array.meta
/mld/multilevelgraph/node_array
/mld/multilevelgraph/edge_array.meta
/mld/multilevelgraph/edge_array
/mld/multilevelgraph/node_to_edge_offset.meta
/mld/multilevelgraph/node_to_edge_offset
/mld/multilevelgraph/connectivity_checksum.meta
/mld/multilevelgraph/connectivity_checksum

Where every file contains raw binary data for data structures in OSRM that we can deserialize or read directly to memory. The .meta files contain additional information for setting up the data layout, e.g. the number of values for serialized std::vector, or the fingerprint.

Tasklist

add tests (see testing documentation
review
adjust for comments

TheMarex added the Work In Progress label

TheMarex self-assigned this

TheMarex force-pushed the refactor/tar_files branch 3 times, most recently from c72d85a to 24dc369 Compare

March 22, 2018 16:43

TheMarex requested a review from oxidase

March 22, 2018 18:19

TheMarex added Review and removed Work In Progress labels

TheMarex requested a review from danpat

March 22, 2018 18:27

oxidase approved these changes

View reviewed changes

Contributor

oxidase left a comment

Looks good! I have some comments to the last commit

include/storage/serialization.hpp Outdated

+              namespace detail
+              {
+              template <typename T, typename BlockT = unsigned char>
               inline unsigned char packBits(const T &data, std::size_t index, std::size_t count)

Contributor

oxidase Mar 23, 2018

return value must be BlockT

include/storage/serialization.hpp Outdated

               {
                   static_assert(std::is_same<typename T::value_type, bool>::value, "value_type is not bool");
-                  const unsigned char mask = 1 << (count - 1);
+                  const BlockT mask = 1 << (count - 1);

Contributor

oxidase Mar 23, 2018

here must be BlockT{1} << (count - 1)

include/storage/serialization.hpp Outdated

+                  const auto decode = [&](const std::uint64_t block) {
+                      auto read_size = std::min<std::size_t>(count - index, WORD_BITS);
+                      unpackBits(data, index, read_size, block);

Contributor

oxidase Mar 23, 2018

and here unpackBits<VectorT, std::uint64_t>

include/storage/serialization.hpp Outdated

+                  // FIXME on old boost version the function_input_iterator does not work with lambdas
+                  // so we need to wrap it in a function here.
+                  const std::function<std::uint64_t()> encode_function = [&]() -> char {

Contributor

oxidase Mar 23, 2018

s/-> char/-> std::uint64_t/

include/storage/serialization.hpp Outdated

+                      return packed;
+                  };
+                  std::uint64_t number_of_blocks = std::ceil((double)count / WORD_BITS);

Contributor

oxidase Mar 23, 2018

(count + WORD_BITS - 1) / WORD_BITS would be better to avoid floating-point arithmetic

Member Author

TheMarex Mar 26, 2018

Ah nice trick. I know a few places were this would come in useful.

danpat added this to the 5.17.0 milestone

TheMarex added Review - In feedback and removed Review labels

TheMarex force-pushed the refactor/tar_files branch from 3aa22d7 to 2707a2b Compare

March 26, 2018 10:59

TheMarex added 16 commits

March 26, 2018 10:59


          Add microtar to dependencies

385bba4


          Squashed 'third_party/microtar/' content from commit 956791770

098d77c

git-subtree-dir: third_party/microtar
git-subtree-split: 956791770defa4d06696b30db276e88a09ad3538


          Merge commit '098d77c658fa846aff55a35c0ad42955e9f736ee' as 'third_par…

c04e1b2

…ty/microtar'


          First step towards reading/writing tar

86bfe1e


          Add unit tests for tar reading

2eb633b


          Add vector serialization

495131e


          Write/read tar for mldgr

f7b7335


          Use static alignment for DataLayout

653f647


          Serialize mldgr using the new tar writer

6d96a9a


          Switch over partition/ files to tar format

15b53de


          Remove file size checks from compability logic

6a09d2a


          Port cell metrics to tar files

fed77c4


          Port hsgr file to tar

cb31f9e


          Fix vector<bool> serialization for tar files and add unit tests

aaf3916


          Add test cases for int and unsigned vector for tar

c664d03


          Port .osrm.icd file over to tar

5d1b4ce

TheMarex added 9 commits

March 26, 2018 11:02


          Add function to stop and continue writing to a tar file

8e800c4


          Get offset of data inside tar file

a52213c


          Add tar file mmaping

bde51a9


          Use pair instead of tuple for implicit construction

921235b


          Wrap lambda in std::function because of boost version issues

f2bace6


          Fix test include path

f1a392c


          Remove old io::FileWrite serialization code

a542da3


          Use OSRM runtime error for tar reader as well

993f5ba


          Fix formating

c0dd5d7

TheMarex force-pushed the refactor/tar_files branch from 2707a2b to 8f75d0e Compare

March 26, 2018 11:02

TheMarex mentioned this pull request

Refactor shared memory data layout to use named identifiers #4975

Merged

4 tasks

TheMarex added 2 commits

March 26, 2018 12:12


          Make the block size of vector<bool> consistent

c322d93


          Fix formating again

3ee8a96

TheMarex force-pushed the refactor/tar_files branch from 8f75d0e to 3ee8a96 Compare

March 26, 2018 12:13

TheMarex and others added 2 commits

March 26, 2018 13:13


          Make sure to return const char* paths for mtar

0c48f5f


          msvc fixes

e572d6c

Contributor

oxidase commented Mar 27, 2018

@TheMarex i pushed fixes for msvc to trigger CI builds ("fixes" are reducing c++ syntactic sugar for msvc). Feel free to kick-out or squash the last commit.

Member Author

TheMarex commented Mar 27, 2018

unknown location(0): fatal error: in "route/test_manual_setting_of_annotations_property": class osrm::util::exception: File "C:/projects/osrm/test/data/ch/monaco.osrm.fileIndex" mapping failed: failed opening file: The process cannot access the file because it is being used by another process.

Some more problems with windows. I suspect this is because I had to change the r-tree to allow writing to an mmap files. Maybe I can work around this.


          Use read-only mmap as member in StaticRTree

4ef331d

Member Author

TheMarex commented Mar 27, 2018 •

edited

Loading

Wuhu finally! It compiles. 🚀 I need to merge this because we add the new dependency tree which doesn't work with rebase + merge.

TheMarex merged commit 2690dd0 into master

TheMarex deleted the refactor/tar_files branch

March 27, 2018 12:34

TheMarex added Ready To Merge and removed Review - In feedback labels

karenzshea mentioned this pull request

Add microtar to executables in unit tests CMakeLists #4977

Merged

6 tasks

TheMarex mentioned this pull request

New OSRM data format #2242

Closed

Member

mloskot commented Mar 28, 2018

Follow-up to @TheMarex -s #2242 (comment) here

One of #2242 requirements was "Only one final file". Does this implement it too. It's unclear form the description. I've failed to fish out from the commits if it is about tar-ifying individual files only or also about packaging into single master file.

Will the docs or wiki:Processing Flow wiki updated on how to use the new format?

Member Author

TheMarex commented Mar 28, 2018

@mloskot you are right, this doesn't meet the single-file requirement yet.

However, since we now use named identifiers for data, osrm-datastore does not really need to care anymore in which files the data is stored, we could figure that out automatically. That would unlock just passing a single file to osrm-datastore that is just a bunch of merged tar files. I captured this in #4978 for further discussion.

Member

mloskot commented Mar 28, 2018

@TheMarex It makes sense, thanks for the clarification.

danpat mentioned this pull request

Optionally cache CH/MLD edge distances #5197

Closed

6 tasks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels