Skip to content

repoindex improvements#6689

Closed
ThomasWaldmann wants to merge 2 commits intoborgbackup:masterfrom
ThomasWaldmann:repoindex
Closed

repoindex improvements#6689
ThomasWaldmann wants to merge 2 commits intoborgbackup:masterfrom
ThomasWaldmann:repoindex

Conversation

@ThomasWaldmann
Copy link
Member

@ThomasWaldmann ThomasWaldmann commented May 12, 2022

improvements:

  • NSIndex: adds the payload size (aka csize) and extra to the repo index
  • NSIndex1: keeps the repo index code for old borg
  • open_index peeks into the hashindex header to find out what's old and what's new.

Comment on lines 1523 to +1536
if include_data:
yield tag, key, offset, data
else:
yield tag, key, offset, size
yield tag, key, offset, size - header_size(tag) # corresponds to len(data)
assert size >= 0
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i changed this so that it either returns the data (so caller could do len(data)) or it does not return the data, but precisely len(data) without any header overheads.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if callers now want the size with overhead, they add header_size(tag) again.

Comment on lines +1458 to +1467
if self._write_fd is not None:
# without this, we have test failure now
self._write_fd.sync()
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this was needed now to avoid a test failure.
otherwise it could not read the MAGIC back from a recently created segment file.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is done a bit more than necessary, sufficient would be:

self._write_fd.f.flush()

But this is not available yet via SyncFile api.

@codecov-commenter
Copy link

codecov-commenter commented May 12, 2022

Codecov Report

Merging #6689 (bf12d0d) into master (0e3ff0a) will increase coverage by 0.01%.
The diff coverage is 84.90%.

❗ Current head bf12d0d differs from pull request most recent head fa5ae14. Consider uploading reports for the commit fa5ae14 to get more accurate results

@@            Coverage Diff             @@
##           master    #6689      +/-   ##
==========================================
+ Coverage   82.59%   82.61%   +0.01%     
==========================================
  Files          39       39              
  Lines       10746    10762      +16     
  Branches     2115     2121       +6     
==========================================
+ Hits         8876     8891      +15     
+ Misses       1345     1343       -2     
- Partials      525      528       +3     
Impacted Files Coverage Δ
src/borg/repository.py 82.60% <84.90%> (+0.15%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0e3ff0a...fa5ae14. Read the comment docs.

…ries

This saves some segment file random IO that was previously necessary
just to determine the size of to be deleted data.

Keep old one as NSIndex1 for old borg compatibility.
Choose NSIndex or NSIndex1 based on repo index layout from HashHeader.

for an old repo index repo.get(key) returns segment, offset, None, None
this fixes a strange test failure that did not happen until now:
it could not read the MAGIC bytes from a (quite new) segment file,
it just returned the empty string.

maybe its appearance is related to the removed I/O calls.
@ThomasWaldmann
Copy link
Member Author

ThomasWaldmann commented May 18, 2022

will redo this against borg2 branch. #6705

@ThomasWaldmann ThomasWaldmann deleted the repoindex branch May 18, 2022 15:24
This was referenced Jun 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants