Skip to content

Comments

[breaking/format] Support for max value size of uint32#893

Merged
ashish-goswami merged 8 commits intoashish/block-level-changesfrom
ashish/valuesize-uint32
Jun 28, 2019
Merged

[breaking/format] Support for max value size of uint32#893
ashish-goswami merged 8 commits intoashish/block-level-changesfrom
ashish/valuesize-uint32

Conversation

@ashish-goswami
Copy link
Contributor

@ashish-goswami ashish-goswami commented Jun 24, 2019

Currently, we store value size in uint16(in skiplist and table). This limits us to have value size to 64KB. This PR adds changes to store value size in uint32.
Impacted parts:

  • Value size in skiplist
  • Value size in header of table entries
  • LSMOnlyOptions.ValueThreshold value

Fixes #812


This change is Reviewable

* Fix table/builder.go
* Fix skl package
* Fix existing tests
* Add test
@ashish-goswami ashish-goswami requested review from a team, jarifibrahim and manishrjain June 24, 2019 17:12
Copy link
Contributor

@manishrjain manishrjain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 0 of 10 files reviewed, 4 unresolved discussions (waiting on @ashish-goswami, @jarifibrahim, and @manishrjain)


db.go, line 195 at r2 (raw file):

	opt.maxBatchCount = opt.maxBatchSize / int64(skl.MaxNodeSize)

	if opt.ValueThreshold > math.MaxUint32-100 { // TODO: correct this.

Disallow greater than opt.MaxTableSize or maybe opt.maxBatchSize.


db.go, line 196 at r2 (raw file):

	if opt.ValueThreshold > math.MaxUint32-100 { // TODO: correct this.
		return nil, ErrValueThreshold

The error string can be generated here.


options.go, line 168 at r2 (raw file):

	// TODO: exact value of ValueThreshold ??
	LSMOnlyOptions.ValueThreshold = 4294967195 // Max value length which fits in uint32.

ValueThreshold can be 1MB.

* Set max ValueThreshold size to 1 MB
* Address review comments
* Add/update tests
@ashish-goswami ashish-goswami changed the title [WIP] Support for max value size of uint32 Support for max value size of uint32 Jun 25, 2019
Copy link
Contributor Author

@ashish-goswami ashish-goswami left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 0 of 11 files reviewed, 4 unresolved discussions (waiting on @golangcibot, @jarifibrahim, and @manishrjain)


db.go, line 195 at r2 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

Disallow greater than opt.MaxTableSize or maybe opt.maxBatchSize.

Done.


db.go, line 196 at r2 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

The error string can be generated here.

Done.


options.go, line 168 at r2 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

ValueThreshold can be 1MB.

Done.


table/builder.go, line 135 at r1 (raw file):

Previously, golangcibot (Bot from GolangCI) wrote…

line is 101 characters (from lll)

Done.

@ashish-goswami ashish-goswami changed the title Support for max value size of uint32 [breaking/format] Support for max value size of uint32 Jun 25, 2019
@ashish-goswami ashish-goswami added this to the v2.1.0 milestone Jun 25, 2019
Copy link
Contributor

@martinmr martinmr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm: except for a couple minor comments. Defer to @manishrjain for the final approval.

Reviewed 3 of 8 files at r1, 2 of 2 files at r2, 6 of 6 files at r3.
Reviewable status: all files reviewed, 5 unresolved discussions (waiting on @ashish-goswami, @golangcibot, @jarifibrahim, and @manishrjain)


db.go, line 196 at r2 (raw file):

Previously, ashish-goswami (Ashish Goswami) wrote…

Done.

I think Manish meant calling errors.Errorf here like right below.


errors.go, line 29 at r3 (raw file):

	// ErrValueThreshold is returned when ValueThreshold is set
	// to a value greater than ValueThresholdLimit.

Can this comment be in a single line? If not disregard.

Copy link
Contributor

@manishrjain manishrjain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: all files reviewed, 6 unresolved discussions (waiting on @ashish-goswami, @golangcibot, @jarifibrahim, and @manishrjain)


errors.go, line 31 at r3 (raw file):

	// to a value greater than ValueThresholdLimit.
	ErrValueThreshold = errors.Errorf(
		"Invalid ValueThreshold, must be less than or equal to %d", ValueThresholdLimit)

Let's remove this error here. And inline it in the place of usage.


options.go, line 160 at r3 (raw file):

const (
	// ValueThresholdLimit is the maximum permissible value of opt.ValueThreshold.
	ValueThresholdLimit = (1 << 20) // 1 MB.

Does it have to be public? Can't this be a private var?

Also, is it being used in more than 1 place in the main code? (ignoring test files)

Copy link
Contributor

@manishrjain manishrjain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got comments. Defer to @jarifibrahim to ensure my comments are addressed, and for final approval.

Reviewed 3 of 8 files at r1, 2 of 2 files at r2, 2 of 6 files at r3, 4 of 4 files at r4.
Reviewable status: all files reviewed, 7 unresolved discussions (waiting on @ashish-goswami, @golangcibot, @jarifibrahim, and @manishrjain)


db.go, line 196 at r4 (raw file):

	// We are limiting opt.ValueThreshold to LSMOnlyOptions.ValueThreshold for now.
	if opt.ValueThreshold > LSMOnlyOptions.ValueThreshold {

This doesn't make sense. LSMOnlyOptions is already an opt, which could have been passed to this Open call.


db.go, line 204 at r4 (raw file):

	// the transaction APIs. Transaction batches entries into batches of size opt.maxBatchSize.
	if int64(opt.ValueThreshold) > opt.maxBatchSize {
		return nil, errors.Errorf("Valuethreshold size too big. Either reduce opt.ValueThreshold " +

ValueThreshold greater than max batch size of %d. Either reduce ...


options.go, line 167 at r4 (raw file):

	LSMOnlyOptions = DefaultOptions

	LSMOnlyOptions.ValueThreshold = (1 << 20) // 1 MB for now, can be changed in future.

Remove the "for now, can be changed in future."

Copy link
Contributor

@jarifibrahim jarifibrahim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: all files reviewed, 12 unresolved discussions (waiting on @ashish-goswami, @golangcibot, and @manishrjain)


db_test.go, line 1475 at r4 (raw file):

	dopts.ValueThreshold = 1 << 21
	_, err = Open(dopts)

Close the opened DB. We don't want unnecessary go routines running the background.


db_test.go, line 1482 at r4 (raw file):

	// maxBatchSize is calculated from MaxTableSize.
	dopts.MaxTableSize = int64(LSMOnlyOptions.ValueThreshold)
	_, err = Open(dopts)

Close the opened DB.


db_test.go, line 1483 at r4 (raw file):

	dopts.MaxTableSize = int64(LSMOnlyOptions.ValueThreshold)
	_, err = Open(dopts)
	require.Error(t, err, "db creation should have been failed")

"db creation should have been failed" => "DB creation failed"


skl/skl_test.go, line 126 at r4 (raw file):

	l.Put(y.KeyWithTs([]byte("key4"), 1), y.ValueStruct{Value: val5, Meta: 60, UserMeta: 0})
	v = l.Get(y.KeyWithTs([]byte("key4"), 1))
	require.True(t, v.Value != nil)

You can use require.NotNil(..) https://godoc.org/github.com/stretchr/testify/require#NotNil


skl/skl_test.go, line 188 at r4 (raw file):

			defer wg.Done()
			v := l.Get(key(i))
			require.True(t, v.Value != nil)

You can use require.NotNil(..) https://godoc.org/github.com/stretchr/testify/require#NotNil

Copy link
Contributor Author

@ashish-goswami ashish-goswami left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 7 of 11 files reviewed, 12 unresolved discussions (waiting on @golangcibot, @jarifibrahim, @manishrjain, and @martinmr)


db.go, line 196 at r4 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

This doesn't make sense. LSMOnlyOptions is already an opt, which could have been passed to this Open call.

Fixed it now.


db.go, line 204 at r4 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

ValueThreshold greater than max batch size of %d. Either reduce ...

Done.


db_test.go, line 1475 at r4 (raw file):

Previously, jarifibrahim (Ibrahim Jarif) wrote…

Close the opened DB. We don't want unnecessary go routines running the background.

This test is for for testing error . If there is no error, it should fail.


db_test.go, line 1482 at r4 (raw file):

Previously, jarifibrahim (Ibrahim Jarif) wrote…

Close the opened DB.

Again same reason.


db_test.go, line 1483 at r4 (raw file):

Previously, jarifibrahim (Ibrahim Jarif) wrote…

"db creation should have been failed" => "DB creation failed"

Here we are checking for non nil error. If error is nil, then require should fail test.


errors.go, line 29 at r3 (raw file):

Previously, martinmr (Martin Martinez Rivera) wrote…

Can this comment be in a single line? If not disregard.

Moved the error to be inline.


options.go, line 160 at r3 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

Does it have to be public? Can't this be a private var?

Also, is it being used in more than 1 place in the main code? (ignoring test files)

I have made maxValueThreshold as private. It is getting used in LSMOnlyOptions and error message when opt.ValueThreshold is greater than maxValueThreshold.


options.go, line 167 at r4 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

Remove the "for now, can be changed in future."

Done.


skl/skl_test.go, line 126 at r4 (raw file):

Previously, jarifibrahim (Ibrahim Jarif) wrote…

You can use require.NotNil(..) https://godoc.org/github.com/stretchr/testify/require#NotNil

Done.


skl/skl_test.go, line 188 at r4 (raw file):

Previously, jarifibrahim (Ibrahim Jarif) wrote…

You can use require.NotNil(..) https://godoc.org/github.com/stretchr/testify/require#NotNil

Done.

Copy link
Contributor

@jarifibrahim jarifibrahim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 4 of 4 files at r5.
Reviewable status: all files reviewed, 7 unresolved discussions (waiting on @golangcibot, @manishrjain, and @martinmr)

@ashish-goswami ashish-goswami merged commit fa87b19 into ashish/block-level-changes Jun 28, 2019
@ashish-goswami ashish-goswami deleted the ashish/valuesize-uint32 branch June 28, 2019 09:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

5 participants