Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tidb_query: fix UB causing by from_raw_parts #7635

Merged
merged 7 commits into from
May 2, 2020
Merged

tidb_query: fix UB causing by from_raw_parts #7635

merged 7 commits into from
May 2, 2020

Conversation

zhongzc
Copy link
Contributor

@zhongzc zhongzc commented Apr 23, 2020

Signed-off-by: zhongzc [email protected]

What problem does this PR solve?

Issue Number: close #7613

Problem Summary:

From stdlib doc:

Behavior is undefined if any of the following conditions are violated:

  • data must be valid for reads for len * mem::size_of::() many bytes, and it must be properly aligned.
  • ...

We didn't provide this promise.

What is changed and how it works?

What's Changed:

Introduce a new type LEBytes. Implement binary_search for this type by explicitly calling std::ptr::read_unaligned.

Check List

Tests

  • Unit test

@zhongzc zhongzc added the sig/coprocessor SIG: Coprocessor label Apr 23, 2020
@zhongzc zhongzc added this to the v4.0.0-rc.2 milestone Apr 23, 2020
@zhongzc
Copy link
Contributor Author

zhongzc commented Apr 23, 2020

@breeswish PTAL

breezewish
breezewish reviewed Apr 23, 2020
View reviewed changes
components/tidb_query_datatype/src/codec/row/v2/row_slice.rs Outdated
@@ -69,11 +71,11 @@ impl RowSlice<'_> {
if let Ok(idx) = non_null_ids.binary_search(&(id as u32)) {
let offset = offsets.get(idx).ok_or(Error::ColumnOffset(idx))?;
let start = if idx > 0 {
offsets[idx - 1] as usize
offsets.get_unchecked(idx - 1) as usize
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There might not be sufficient elements in the offsets (if the data is corrupted). We'd better handle it.

Previously this case result in panic. Now it result in memory issue, which is worse. Better to change it to return error.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see this conversation marked as resolved, but what was the resolution? Why is this unchecked indexing ok?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doing an unsafe array access here isn't obviously important to performance. @zhongzc are you sure this needs to be unchecked?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@brson Previous offsets.get(idx) indicates offsets.get_unchecked(idx - 1) is okay.

breezewish
breezewish reviewed Apr 23, 2020
View reviewed changes
Copy link
Member

@breezewish breezewish left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool!

@MyonKeminta MyonKeminta modified the milestones: v4.0.0-rc.1, v4.0.0 Apr 24, 2020
@breezewish
Copy link
Member

/run-all-tests

@breezewish breezewish added the needs-cherry-pick-release-4.0 Type: Need cherry pick to release 4.0 label Apr 26, 2020
breezewish
breezewish approved these changes Apr 26, 2020
View reviewed changes
Copy link
Member

@breezewish breezewish left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job. @brson Would you like to take a look at the improved implementation?

brson
brson requested changes Apr 30, 2020
View reviewed changes
Copy link
Contributor

@brson brson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Per previous comments, get_unchecked should be marked unsafe. I'd also like to see comment about why get_unchecked is used during indexing after the binary search.

components/tidb_query_datatype/src/codec/row/v2/row_slice.rs Outdated
@@ -69,11 +71,11 @@ impl RowSlice<'_> {
if let Ok(idx) = non_null_ids.binary_search(&(id as u32)) {
let offset = offsets.get(idx).ok_or(Error::ColumnOffset(idx))?;
let start = if idx > 0 {
offsets[idx - 1] as usize
offsets.get_unchecked(idx - 1) as usize
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see this conversation marked as resolved, but what was the resolution? Why is this unchecked indexing ok?

components/tidb_query_datatype/src/codec/row/v2/row_slice.rs Outdated
}

#[inline]
fn get_unchecked(&self, index: usize) -> T {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function should be marked as unsafe since it allows the caller to do unsafe memory access.

components/tidb_query_datatype/src/codec/row/v2/row_slice.rs Outdated
@@ -69,11 +71,11 @@ impl RowSlice<'_> {
if let Ok(idx) = non_null_ids.binary_search(&(id as u32)) {
let offset = offsets.get(idx).ok_or(Error::ColumnOffset(idx))?;
let start = if idx > 0 {
offsets[idx - 1] as usize
offsets.get_unchecked(idx - 1) as usize
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doing an unsafe array access here isn't obviously important to performance. @zhongzc are you sure this needs to be unchecked?

zhongzc added 5 commits April 30, 2020 10:50
@zhongzc
fix UB
158931f 
Signed-off-by: zhongzc <[email protected]>
@zhongzc
address comment
a6783aa 
Signed-off-by: zhongzc <[email protected]>
@zhongzc
fix typo
0b08fc3 
Signed-off-by: zhongzc <[email protected]>
@zhongzc
add fuzz
f18e5bc 
Signed-off-by: zhongzc <[email protected]>
@zhongzc
address comment
c7f89f0 
Signed-off-by: zhongzc <[email protected]>
brson
brson approved these changes May 1, 2020
View reviewed changes
Copy link
Contributor

@brson brson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm thanks @zhongzc

@brson
Copy link
Contributor

brson commented May 1, 2020

/merge

@sre-bot sre-bot added the status/can-merge Indicates a PR has been approved by a committer. label May 1, 2020
@sre-bot
Copy link
Contributor

sre-bot commented May 1, 2020

/run-all-tests

@sre-bot
Copy link
Contributor

sre-bot commented May 1, 2020

@zhongzc merge failed.

@zhongzc
Copy link
Contributor Author

zhongzc commented May 2, 2020

/merge

@sre-bot
Copy link
Contributor

sre-bot commented May 2, 2020

/run-all-tests

@sre-bot
Copy link
Contributor

sre-bot commented May 2, 2020

@zhongzc merge failed.

@zhongzc
Copy link
Contributor Author

zhongzc commented May 2, 2020

/merge

@sre-bot
Copy link
Contributor

sre-bot commented May 2, 2020

/run-all-tests

@sre-bot
Copy link
Contributor

sre-bot commented May 2, 2020

@zhongzc merge failed.

@zhongzc zhongzc merged commit 08e8365 into tikv:master May 2, 2020
sre-bot pushed a commit to sre-bot/tikv that referenced this pull request May 2, 2020
@zhongzc @sre-bot
tidb_query: fix UB causing by from_raw_parts (tikv#7635)
c2fc5c6 
Signed-off-by: zhongzc <[email protected]>
@sre-bot sre-bot mentioned this pull request May 2, 2020
Merged
@sre-bot
Copy link
Contributor

sre-bot commented May 2, 2020

cherry pick to release-4.0 in PR #7730

c1ay pushed a commit to c1ay/tikv that referenced this pull request May 9, 2020
@zhongzc @c1ay
tidb_query: fix UB causing by from_raw_parts (tikv#7635)
10825e6 
Signed-off-by: zhongzc <[email protected]>
sre-bot added a commit that referenced this pull request May 14, 2020
@sre-bot
tidb_query: fix UB causing by from_raw_parts (#7635) (#7730)
234cdda 
Signed-off-by: zhongzc <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@brson brson brson approved these changes

@breezewish breezewish breezewish approved these changes

Assignees
No one assigned
Labels
needs-cherry-pick-release-4.0 Type: Need cherry pick to release 4.0 sig/coprocessor SIG: Coprocessor status/can-merge Indicates a PR has been approved by a committer.
Projects
None yet
Milestone
v4.0.0-rc.2
Development

Successfully merging this pull request may close these issues.

Undefined behavior in tidb_query_datatype's RowSlice
5 participants
@zhongzc @breezewish @brson @sre-bot @MyonKeminta