Skip to content

GH-32947: [Go] Implement Concatenate for REE Array - #14126

Merged
zeroshade merged 7 commits into
apache:masterfrom
zeroshade:rle-concatenate
Jan 25, 2023
Merged

zeroshade merged 7 commits into
apache:masterfrom
zeroshade:rle-concatenate

Conversation

@zeroshade

@zeroshade zeroshade commented Sep 14, 2022 •

Copy link
Copy Markdown
Member

@zeroshade

Copy link
Copy Markdown
Member Author

CC @zagto

@github-actions

Copy link
Copy Markdown

@zeroshade
zeroshade force-pushed the rle-concatenate branch 2 times, most recently from fb90e87 to aa29f37 Compare October 21, 2022 16:00
@zeroshade zeroshade changed the title ARROW-17710: [Go] Implement Concatenate for RLE Array GH-32947: [Go] Implement Concatenate for RLE Array Jan 24, 2023
@zeroshade
zeroshade marked this pull request as ready for review January 24, 2023 20:46
@github-actions

Copy link
Copy Markdown

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #32947 has been automatically assigned in GitHub to PR creator.

@zeroshade
zeroshade requested a review from lidavidm January 24, 2023 20:54

@lidavidm lidavidm left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: PR/Issue should be "REE" array right?

Comment thread go/arrow/array/concat_test.go Resolved
Comment thread go/arrow/array/concat.go Outdated

physicalLength, overflow = addOvf(physicalLength, int(plen))
if overflow {
return nil, fmt.Errorf("%w: run length encoded array length must fit into a 32-bit signed integer",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, shouldn't this depend on the offset type?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suppose there are separate checks below, and since lengths are limited to 32-bit...

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so, this is more of a short circuit. We have to add up all of the physical lengths so we know the size to allocate for the full concatenated array. If we end up overflowing when adding up the physical length we don't have to bother calculating the new run ends, we know we're going to overflow no matter what the offset type is. So this doesn't need to depend on the offset type.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough. I suppose you could short circuit here earlier for int16, but as long as it properly fails either way.

@zeroshade zeroshade changed the title GH-32947: [Go] Implement Concatenate for RLE Array GH-32947: [Go] Implement Concatenate for REE Array Jan 25, 2023
@zeroshade

Copy link
Copy Markdown
Member Author

I'll merge this at EOD if no one else has comments

@zeroshade
zeroshade merged commit 3b21e0c into apache:master Jan 25, 2023
@zeroshade
zeroshade deleted the rle-concatenate branch January 25, 2023 22:54
@ursabot

ursabot commented Jan 26, 2023

Copy link
Copy Markdown

Benchmark runs are scheduled for baseline = 4a1229c and contender = 3b21e0c. 3b21e0c is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.89% ⬆️0.0%] test-mac-arm
[Failed ⬇️0.0% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.03% ⬆️0.06%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 3b21e0c4 ec2-t3-xlarge-us-east-2
[Failed] 3b21e0c4 test-mac-arm
[Failed] 3b21e0c4 ursa-i9-9960x
[Finished] 3b21e0c4 ursa-thinkcentre-m75q
[Finished] 4a1229c0 ec2-t3-xlarge-us-east-2
[Failed] 4a1229c0 test-mac-arm
[Failed] 4a1229c0 ursa-i9-9960x
[Finished] 4a1229c0 ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Go] Concatenate Implementation for REE Array

3 participants