[breaking/format] Block level changes#880
Conversation
This commit adds the key-offset index used for searching the blocks in a table to the end the SST file. The length of the index is stored in the last 4 bytes of the file.
manishrjain
left a comment
There was a problem hiding this comment.
Good to go. I think the OpenTable can be pass opt instead of individual fields, but that can be done later. Just add a TODO.
Reviewed 1 of 9 files at r5, 9 of 12 files at r6, 4 of 4 files at r7.
Reviewable status: 13 of 14 files reviewed, 27 unresolved discussions (waiting on @ashish-goswami, @golangcibot, @jarifibrahim, and @manishrjain)
db.go, line 918 at r7 (raw file):
} tbl, err := table.OpenTable(fd, db.opt.TableLoadingMode, db.opt.ChecksumVerificationMode)
Add a todo to pass the opt object instead of specific fields.
table/table.go, line 293 at r7 (raw file):
} func (t *Table) block(idx int) (block, error) {
Wouldn't it make sense for this to return a pointer to block? That might be cheaper than getting all the block fields copied over.
jarifibrahim
left a comment
There was a problem hiding this comment.
Reviewable status: 13 of 14 files reviewed, 21 unresolved discussions (waiting on @ashish-goswami, @golangcibot, and @manishrjain)
table/table_test.go, line 657 at r8 (raw file):
n := int(5 * 1e6) tbl := getTableForBenchmarks(b, n) // defer tbl.DecrRef()
Uncomment please!
table/table_test.go, line 658 at r8 (raw file):
tbl := getTableForBenchmarks(b, n) // defer tbl.DecrRef() fmt.Println("********** ", tbl.Size())
Can we remove this?
ashish-goswami
left a comment
There was a problem hiding this comment.
Reviewable status: 12 of 14 files reviewed, 9 unresolved discussions (waiting on @golangcibot, @jarifibrahim, @manishrjain, and @martinmr)
db.go, line 918 at r7 (raw file):
Previously, manishrjain (Manish R Jain) wrote…
Add a todo to pass the opt object instead of specific fields.
Done.
options.go, line 155 at r5 (raw file):
Previously, martinmr (Martin Martinez Rivera) wrote…
Francesc is making a change to how options work (#874) make sure your change is compatible with the change (either you merge that PR into your branch or send another PR if yours is merged into master first).
Ok. I will take care of this.
table/builder.go, line 187 at r5 (raw file):
Previously, martinmr (Martin Martinez Rivera) wrote…
y.AssertTrue((len(b.entryOffsets)+1)*4+4+8+4 < math.MaxUint32) // check for below statements // we should include current entry also in size, thats why +1 to len(b.entryOffsets)minor: period at the end. Also start with uppercase.
Done.
table/table.go, line 310 at r6 (raw file):
Previously, golangcibot (Bot from GolangCI) wrote…
line is 101 characters (from
lll)
Done.
table/table.go, line 293 at r7 (raw file):
Previously, manishrjain (Manish R Jain) wrote…
Wouldn't it make sense for this to return a pointer to block? That might be cheaper than getting all the block fields copied over.
Done.
table/table.go, line 345 at r9 (raw file):
Previously, golangcibot (Bot from GolangCI) wrote…
line is 109 characters (from
lll)
Done.
table/table_test.go, line 657 at r8 (raw file):
Previously, jarifibrahim (Ibrahim Jarif) wrote…
Uncomment please!
Done.
table/table_test.go, line 658 at r8 (raw file):
Previously, jarifibrahim (Ibrahim Jarif) wrote…
Can we remove this?
Done.
martinmr
left a comment
There was a problem hiding this comment.
Reviewed 1 of 2 files at r9, 1 of 1 files at r10.
Reviewable status: all files reviewed, 7 unresolved discussions (waiting on @golangcibot, @jarifibrahim, and @manishrjain)
jarifibrahim
left a comment
There was a problem hiding this comment.
Reviewable status: all files reviewed, 10 unresolved discussions (waiting on @ashish-goswami, @golangcibot, and @manishrjain)
db.go, line 463 at r10 (raw file):
// VerifyChecksum verifies checksum for all tables on all levels. // This method can be used to verify checksum, if ChecksumVerificationMode is NoVerification.
This comment should be on the table.VerifyChecksum method.
levels.go, line 1008 at r10 (raw file):
for _, t := range tables { errChkVerify := t.VerifyChecksum() if err := t.DecrRef(); err != nil {
Shouldn't we check errChkVerify before this line? What if t.VerifyChecksum returns an error and t.DecrRef() also returns an error?
options/options.go, line 39 at r10 (raw file):
NoVerification ChecksumVerificationMode = iota // OnTableRead indicates checksum should be verified while opening SSTtable. OnTableRead
Why is this called OnTableRead and not OnTableOpen? The comment says checksum should be verified while opening SSTable
table/builder.go, line 188 at r10 (raw file):
y.AssertTrue((len(b.entryOffsets)+1)*4+4+8+4 < math.MaxUint32) // check for below statements // We should include current entry also in size, thats why +1 to len(b.entryOffsets).
nit: that's why
table/table_test.go, line 648 at r10 (raw file):
f.WriteAt(rb, rand.Int63n(fi.Size())) _, err = OpenTable(f, options.LoadToRAM, options.OnTableAndBlockRead)
The table should be closed.
ashish-goswami
left a comment
There was a problem hiding this comment.
Reviewable status: 12 of 14 files reviewed, 9 unresolved discussions (waiting on @golangcibot, @jarifibrahim, and @manishrjain)
db.go, line 463 at r10 (raw file):
Previously, jarifibrahim (Ibrahim Jarif) wrote…
This comment should be on the
table.VerifyChecksummethod.
I think it should be here. This tells even if opt.ChecksumVerificatinMode is NoVerification, then also this function can be called to verify checksum.
levels.go, line 1008 at r10 (raw file):
Previously, jarifibrahim (Ibrahim Jarif) wrote…
Shouldn't we check
errChkVerifybefore this line? What ift.VerifyChecksumreturns an error andt.DecrRef()also returns an error?
If both of them returns error, I am just logging error from t.DecrRef(). In any case, we should decrease table reference first.
options/options.go, line 39 at r10 (raw file):
Previously, jarifibrahim (Ibrahim Jarif) wrote…
Why is this called
OnTableReadand notOnTableOpen? The comment sayschecksum should be verified while opening SSTable
Table read here means, reading table from disk, which is table open.
table/builder.go, line 188 at r10 (raw file):
Previously, jarifibrahim (Ibrahim Jarif) wrote…
nit:
that's why
Done.
table/table_test.go, line 648 at r10 (raw file):
Previously, jarifibrahim (Ibrahim Jarif) wrote…
The table should be closed.
Here table creation should fail, hence I am not checking for error.
jarifibrahim
left a comment
There was a problem hiding this comment.
Reviewed 2 of 2 files at r11.
Reviewable status: all files reviewed, 7 unresolved discussions (waiting on @ashish-goswami, @golangcibot, and @manishrjain)
db_test.go, line 1831 at r11 (raw file):
func TestVerifyChecksum(t *testing.T) { // user stream write for writing.
user => use
levels.go, line 1009 at r11 (raw file):
errChkVerify := t.VerifyChecksum() if err := t.DecrRef(); err != nil { s.kv.opt.Errorf("unable to descrease reference of table: %s while "+
descrease => decrease.
ashish-goswami
left a comment
There was a problem hiding this comment.
Reviewable status: 12 of 14 files reviewed, 7 unresolved discussions (waiting on @golangcibot, @jarifibrahim, and @manishrjain)
db_test.go, line 1831 at r11 (raw file):
Previously, jarifibrahim (Ibrahim Jarif) wrote…
user=>use
Done.
levels.go, line 1009 at r11 (raw file):
Previously, jarifibrahim (Ibrahim Jarif) wrote…
descrease=>decrease.
Done.
* Change max value size to uint32 * Fix table/builder.go * Fix skl package * Fix existing tests * Add test
…hish/block-level-changes
This was missing when options.ChecksumVerificationMode was added in #880.
* Add WithChecksumVerificationMode option builder method. This was missing when options.ChecksumVerificationMode was added in #880.
**Description** Correct an incorrect comment on the value size in skl.node. Background: the PR #880 changed the value size from uint16 to uint32, but the did not update the comment. **Checklist** - [X] Code compiles correctly and linting passes locally - [X] Tests added for new functionality, or regression tests for bug fixes added as applicable Signed-off-by: Benjamin Wang <[email protected]>
This PR contains following changes:
Fixes #870
This change is