Skip to content

Comments

[breaking/format] Block level changes#880

Merged
ashish-goswami merged 91 commits intomasterfrom
ashish/block-level-changes
Jul 9, 2019
Merged

[breaking/format] Block level changes#880
ashish-goswami merged 91 commits intomasterfrom
ashish/block-level-changes

Conversation

@ashish-goswami
Copy link
Contributor

@ashish-goswami ashish-goswami commented Jun 20, 2019

This PR contains following changes:

  • Have block based on size.
  • Implementation of binary search inside block.
  • Exposing option in DB to verify checksum on start.

Fixes #870


This change is Reviewable

Copy link
Contributor

@manishrjain manishrjain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm: Good to go. I think the OpenTable can be pass opt instead of individual fields, but that can be done later. Just add a TODO.

Reviewed 1 of 9 files at r5, 9 of 12 files at r6, 4 of 4 files at r7.
Reviewable status: 13 of 14 files reviewed, 27 unresolved discussions (waiting on @ashish-goswami, @golangcibot, @jarifibrahim, and @manishrjain)


db.go, line 918 at r7 (raw file):

	}

	tbl, err := table.OpenTable(fd, db.opt.TableLoadingMode, db.opt.ChecksumVerificationMode)

Add a todo to pass the opt object instead of specific fields.


table/table.go, line 293 at r7 (raw file):

}

func (t *Table) block(idx int) (block, error) {

Wouldn't it make sense for this to return a pointer to block? That might be cheaper than getting all the block fields copied over.

Copy link
Contributor

@jarifibrahim jarifibrahim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 13 of 14 files reviewed, 21 unresolved discussions (waiting on @ashish-goswami, @golangcibot, and @manishrjain)


table/table_test.go, line 657 at r8 (raw file):

	n := int(5 * 1e6)
	tbl := getTableForBenchmarks(b, n)
	// defer tbl.DecrRef()

Uncomment please!


table/table_test.go, line 658 at r8 (raw file):

	tbl := getTableForBenchmarks(b, n)
	// defer tbl.DecrRef()
	fmt.Println("********** ", tbl.Size())

Can we remove this?

Copy link
Contributor Author

@ashish-goswami ashish-goswami left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 12 of 14 files reviewed, 9 unresolved discussions (waiting on @golangcibot, @jarifibrahim, @manishrjain, and @martinmr)


db.go, line 918 at r7 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

Add a todo to pass the opt object instead of specific fields.

Done.


options.go, line 155 at r5 (raw file):

Previously, martinmr (Martin Martinez Rivera) wrote…

Francesc is making a change to how options work (#874) make sure your change is compatible with the change (either you merge that PR into your branch or send another PR if yours is merged into master first).

Ok. I will take care of this.


table/builder.go, line 187 at r5 (raw file):

Previously, martinmr (Martin Martinez Rivera) wrote…
	y.AssertTrue((len(b.entryOffsets)+1)*4+4+8+4 < math.MaxUint32) // check for below statements
	// we should include current entry also in size, thats why +1 to len(b.entryOffsets)

minor: period at the end. Also start with uppercase.

Done.


table/table.go, line 310 at r6 (raw file):

Previously, golangcibot (Bot from GolangCI) wrote…

line is 101 characters (from lll)

Done.


table/table.go, line 293 at r7 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

Wouldn't it make sense for this to return a pointer to block? That might be cheaper than getting all the block fields copied over.

Done.


table/table.go, line 345 at r9 (raw file):

Previously, golangcibot (Bot from GolangCI) wrote…

line is 109 characters (from lll)

Done.


table/table_test.go, line 657 at r8 (raw file):

Previously, jarifibrahim (Ibrahim Jarif) wrote…

Uncomment please!

Done.


table/table_test.go, line 658 at r8 (raw file):

Previously, jarifibrahim (Ibrahim Jarif) wrote…

Can we remove this?

Done.

Copy link
Contributor

@martinmr martinmr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 1 of 2 files at r9, 1 of 1 files at r10.
Reviewable status: all files reviewed, 7 unresolved discussions (waiting on @golangcibot, @jarifibrahim, and @manishrjain)

Copy link
Contributor

@jarifibrahim jarifibrahim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: all files reviewed, 10 unresolved discussions (waiting on @ashish-goswami, @golangcibot, and @manishrjain)


db.go, line 463 at r10 (raw file):

// VerifyChecksum verifies checksum for all tables on all levels.
// This method can be used to verify checksum, if ChecksumVerificationMode is NoVerification.

This comment should be on the table.VerifyChecksum method.


levels.go, line 1008 at r10 (raw file):

		for _, t := range tables {
			errChkVerify := t.VerifyChecksum()
			if err := t.DecrRef(); err != nil {

Shouldn't we check errChkVerify before this line? What if t.VerifyChecksum returns an error and t.DecrRef() also returns an error?


options/options.go, line 39 at r10 (raw file):

	NoVerification ChecksumVerificationMode = iota
	// OnTableRead indicates checksum should be verified while opening SSTtable.
	OnTableRead

Why is this called OnTableRead and not OnTableOpen? The comment says checksum should be verified while opening SSTable


table/builder.go, line 188 at r10 (raw file):

	y.AssertTrue((len(b.entryOffsets)+1)*4+4+8+4 < math.MaxUint32) // check for below statements
	// We should include current entry also in size, thats why +1 to len(b.entryOffsets).

nit: that's why


table/table_test.go, line 648 at r10 (raw file):

	f.WriteAt(rb, rand.Int63n(fi.Size()))

	_, err = OpenTable(f, options.LoadToRAM, options.OnTableAndBlockRead)

The table should be closed.

Copy link
Contributor Author

@ashish-goswami ashish-goswami left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 12 of 14 files reviewed, 9 unresolved discussions (waiting on @golangcibot, @jarifibrahim, and @manishrjain)


db.go, line 463 at r10 (raw file):

Previously, jarifibrahim (Ibrahim Jarif) wrote…

This comment should be on the table.VerifyChecksum method.

I think it should be here. This tells even if opt.ChecksumVerificatinMode is NoVerification, then also this function can be called to verify checksum.


levels.go, line 1008 at r10 (raw file):

Previously, jarifibrahim (Ibrahim Jarif) wrote…

Shouldn't we check errChkVerify before this line? What if t.VerifyChecksum returns an error and t.DecrRef() also returns an error?

If both of them returns error, I am just logging error from t.DecrRef(). In any case, we should decrease table reference first.


options/options.go, line 39 at r10 (raw file):

Previously, jarifibrahim (Ibrahim Jarif) wrote…

Why is this called OnTableRead and not OnTableOpen? The comment says checksum should be verified while opening SSTable

Table read here means, reading table from disk, which is table open.


table/builder.go, line 188 at r10 (raw file):

Previously, jarifibrahim (Ibrahim Jarif) wrote…

nit: that's why

Done.


table/table_test.go, line 648 at r10 (raw file):

Previously, jarifibrahim (Ibrahim Jarif) wrote…

The table should be closed.

Here table creation should fail, hence I am not checking for error.

Copy link
Contributor

@jarifibrahim jarifibrahim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 2 of 2 files at r11.
Reviewable status: all files reviewed, 7 unresolved discussions (waiting on @ashish-goswami, @golangcibot, and @manishrjain)


db_test.go, line 1831 at r11 (raw file):

func TestVerifyChecksum(t *testing.T) {
	// user stream write for writing.

user => use


levels.go, line 1009 at r11 (raw file):

			errChkVerify := t.VerifyChecksum()
			if err := t.DecrRef(); err != nil {
				s.kv.opt.Errorf("unable to descrease reference of table: %s while "+

descrease => decrease.

Copy link
Contributor Author

@ashish-goswami ashish-goswami left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 12 of 14 files reviewed, 7 unresolved discussions (waiting on @golangcibot, @jarifibrahim, and @manishrjain)


db_test.go, line 1831 at r11 (raw file):

Previously, jarifibrahim (Ibrahim Jarif) wrote…

user => use

Done.


levels.go, line 1009 at r11 (raw file):

Previously, jarifibrahim (Ibrahim Jarif) wrote…

descrease => decrease.

Done.

* Change max value size to uint32

* Fix table/builder.go
* Fix skl package
* Fix existing tests
* Add test
@ashish-goswami ashish-goswami changed the base branch from ibrahim/footer-protobuf to master July 9, 2019 06:39
@ashish-goswami ashish-goswami merged commit 1496af9 into master Jul 9, 2019
@ashish-goswami ashish-goswami deleted the ashish/block-level-changes branch July 9, 2019 10:41
danielmai added a commit that referenced this pull request Nov 12, 2019
This was missing when options.ChecksumVerificationMode was added in #880.
danielmai added a commit that referenced this pull request Nov 12, 2019
* Add WithChecksumVerificationMode option builder method.

This was missing when options.ChecksumVerificationMode was added in #880.
matthewmcneely pushed a commit that referenced this pull request Dec 28, 2025
**Description**

Correct an incorrect comment on the value size in skl.node.

Background: the PR #880 changed
the value size from uint16 to uint32, but the did not update the
comment.

**Checklist**

- [X] Code compiles correctly and linting passes locally
- [X] Tests added for new functionality, or regression tests for bug
fixes added as applicable

Signed-off-by: Benjamin Wang <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

Making block size based and implement binary search in it

6 participants