Add an AbsoluteIndexOffsets option for readonly blockstore#490
Add an AbsoluteIndexOffsets option for readonly blockstore#490
AbsoluteIndexOffsets option for readonly blockstore#490Conversation
…ginning of the car file, rather than from the beginning of the carv1 contained file.
| // WithAbsoluteIndexOffsets is a read option which interprets index offsets as | ||
| // absolute offsets from the beginning of the CARv2 file, instead of relative | ||
| // offsets from the beginning of the Carv1 header. | ||
| func WithAbsoluteIndexOffsets(absolute bool) Option { |
There was a problem hiding this comment.
This would violate the specification of CARv2. Is there any chance we can capture this complexity inside boost? I am sympathetic that this might be hard to do in the existing Boost implementation, but am worried that this would open a can of worms for all sorts of CARv2 parsing errors since the specification dictates offset in index format is always from the beginning of inner CARv1.
If not, would it make sense to add this as a characteristic in CARv2 header and spec it out that way?
|
@willscott did you close because you found a way around this or do we need to explore this further? there's probably a few options here, but if we can get by with a quick parse of the start of the file then that would be the easiest. |
|
because it isn't worth blocking on a discussion at the car semantics level. Here's my ugly work-around at the boost level: |
3 participants
Allow opening a blockstore with an index that has offsets from the beginning of the car file, rather than from the beginning of the carv1 contained file.
Absolute indexes of this type are generated by boost. Once the car is stored to a remote location separate from the index, it's unclear if it was a v1 or v2 car. Using an absolute index means individual CIDs can be efficiently fetched without re-parses of the header.
The carv2 blockstore is used over this situation by boost for cases where a blockstore interface is needed. in this case, we would like to be able to support a mode where the reader offset for index reads doesn't offset based on the indicated start of the data section in the carv2 header.