ARROW-16772: [C++] Implement encode and decode functions for Run-Length encoding by zagto · Pull Request #13330 · apache/arrow

zagto · 2022-06-07T13:54:06Z

No description provided.

github-actions · 2022-06-07T15:20:29Z

https://issues.apache.org/jira/browse/ARROW-16772

github-actions · 2022-06-07T15:20:31Z

⚠️ Ticket has no components in JIRA, make sure you assign one.

zagto · 2022-06-07T17:59:34Z

    DataMember("null_placement", &RankOptions::null_placement),
    DataMember("tiebreaker", &RankOptions::tiebreaker));
+static auto kRunLengthEncodeOptionsType =
+    GetFunctionOptionsType<RunLengthEncodeOptions>();


These options options currently unnecessary. To me that seems to be the place where we could specify the run length values type, if we decide to support multiple ones

westonpace · 2022-08-11T23:59:20Z

@drin do you want to take a look through this PR

drin

reviewed most of the files.

still need to go over:

vector_run_length_encode_test.cc
and vector_run_length_encode.cc

(not sure how the review interface works, so this is also a note for myself

drin · 2022-08-12T22:51:17Z

+  if (type->id() == Type::RUN_LENGTH_ENCODED) {
+    return child_data[0].MayHaveNulls();
+  } else {
+    // If an ArrayData is slightly malformed it may have kUnknownNullCount set


Suggested change

// If an ArrayData is slightly malformed it may have kUnknownNullCount set

// If an ArraySpan is slightly malformed it may have kUnknownNullCount set

drin · 2022-08-16T16:40:24Z

+    if (type.id() == Type::RUN_LENGTH_ENCODED) {
+      auto& encoding_type = dynamic_cast<const EncodingType&>(type);
+      return encoded_type_matcher->Matches(*encoding_type.encoded_type());
+    } else {
+      return false;
+    }


Suggested change

if (type.id() == Type::RUN_LENGTH_ENCODED) {

auto& encoding_type = dynamic_cast<const EncodingType&>(type);

return encoded_type_matcher->Matches(*encoding_type.encoded_type());

} else {

return false;

}

if (type.id() != Type::RUN_LENGTH_ENCODED) { return false; }

auto& encoding_type = dynamic_cast<const EncodingType&>(type);

return encoded_type_matcher->Matches(*encoding_type.encoded_type());

optional suggestion, but I think it speeds up readability

drin · 2022-08-16T16:42:40Z

+  };
+
+ private:
+  std::shared_ptr<TypeMatcher> encoded_type_matcher;


I don't know the surrounding context, so I'm just curious if this could be a unique_ptr instead and if there is benefit if it were?

drin · 2022-08-16T20:24:31Z

+int64_t FindPhysicalOffset(const int32_t* accumulated_run_lengths,
+                           int64_t physical_length, int64_t logical_offset);
+
+int64_t FindPhysicalIndex(const int32_t* accumulated_run_lengths,
+                          int64_t physical_length, int64_t physical_offset, int64_t logical_offset, int64_t logical_index);


can accumulated_run_lengths be shortened to run_lengths? Otherwise, I'm not sure what the accumulated portion should signify.

It would also be helpful to know what the difference is between offsets and indices. Based on the implementations in rle_util.cc, it seems like FindPhysicalOffset could be implemented as:

FindPhysicalIndex(run_lengths, physical_length, 0, logical_offset, 0);

so, it's not clear to me what the different naming signifies (except maybe that a physical offset and logical index is/isn't provided)

zagto · 2022-08-19T00:05:54Z

@drin I hope this is not too confusing, but I did a lot of work on the rle-util branch in the recent time, which was not merged/pushed here yet, sorry for that. I'm going to push that, so that you can see the lastest version of the encode/decode kernels

Thanks for the feedback so far, I think the majority still applies to the current code. I don't think you have to review all the new rle_util code unless you want to, someone is eventually going to review ARROW-17263 for that.

drin · 2022-08-22T21:20:50Z

          Type::DENSE_UNION,
          Type::SPARSE_UNION,
          Type::DICTIONARY,
+          Type::RUN_LENGTH_ENCODED,


I think this type is duplicated on line 107, so probably only need one.

Suggested change

Type::RUN_LENGTH_ENCODED,

drin · 2022-08-22T21:49:29Z

-  ACTION(DenseUnion);                           \
-  ACTION(Dictionary);                           \
+// all types that can exist as a Scalar
+#define ARROW_GENERATE_FOR_ALL_SCALAR_TYPES(ACTION) \


Can all of these be scalars? If so, I'm not sure why ARROW_GENERATE_FOR_ALL_TYPES is being changed to a wrapper around this macro.

Otherwise, maybe a comment for ARROW_GENERATE_FOR_ALL_TYPES could help with clarity.

I think all types that exist in arrow right now can exist as scalars. I was considering adding an RLE scalar class for consistency. But the encoding can only be applied to arrays, so an RLE scalar would just be a wrapper for another scalar of the actual data type. To me that seems like something you would not actually want to use and therefore it should not exist.

The idea for new macro is code that works on scalar values does not need to generate a case for RLE (and possibly other future array encodings), which would be unnecessary and may also result in using type_traits<RunLengthEncoded>::ScalarType

drin · 2022-08-23T00:07:16Z

@drin ...I did a lot of work on the rle-util branch in the recent time... I'm going to push that, so that you can see the lastest version of the encode/decode kernels

Thanks for the feedback so far, I think the majority still applies to the current code. I don't think you have to review all the new rle_util code unless you want to, someone is eventually going to review ARROW-17263 for that.

just saw this comment, thanks for the heads up! I'll try to review anyways, it hopefully doesn't have too much extra learning curve

drin · 2022-08-23T00:20:53Z

+  data_->child_data.push_back(std::move(run_ends_array->data()));
+  data_->child_data.push_back(std::move(values_array->data()));


Is there design documentation somewhere?

I think this approach makes sense, but I'm just wondering what the trade-offs are compared to having run ends and values in a single Array, in a similar layout as variable-sized binary? I suspect this approach is just more flexible, especially for run length encoding a nested type array; though, there is likely more to it than that.

I don't think I have written that down anywhere, so I'll put it here:

Yes having the values buffer(s) in the parent would makes supporting multiple data types more complicated:

Each type in arrow has a defined buffer layout, RLE would not. It would have different buffers depending on the type it encodes, i.e. a single values buffer for integer types, typeID+offset for dense union..., but also add a run ends buffer somewhere

we could not rely on things like Validate(), unary kernels, array builders, or anything else to work on the values array without modification

While in Arrow C++ the parent array has pointer to the exact DataType object, in the format itself an array has only a type ID, RLE in this case. The type of the values would have to be put into the buffers somewhere, making the layout even more complicated

Having the run ends as a buffer in the parent is less bad, but also causes problems:

When a buffer is passed over the C bridge, we don't really know it's size (or at least how I understand it). It can be determined either from the length field, or a combination of that and other buffers easily for other types, but not for RLE. The RLE format is designed to allow O(log(n)) random access ,using binary search. For Binary search, we need an upper end, which we conveniently get from the length field of a child array (Yes the whole array needs to be sorted).

Also having a child array gives us another type field, which means the format could support RLE arrays with run ends that are not int32 in the future.

In general I like the idea that a user can simply run any analysis on the run ends and values array on thier own, if they find it useful for some reason. They could always construct an array from the buffers but it just seems less messy with separate arrays.

awesome, thanks! I figured out the memory layout as well, so I realized that this approach also allows run values to be a binary type with a contiguous buffer.

drin

I checked out most of vector_run_length_encode.cc. I think I understand the memory layout now, so hopefully finishing the rest of the review goes quick!

All my suggestions are minor, great job so far 🎉 !

drin · 2022-08-23T00:28:49Z

+  ASSERT_ARRAYS_EQUAL(*new_array, *rle_array);
+  // should be the exact same ArrayData object


Maybe we should also add a mutation to one array and assert that they're not equal?

If the intent of this test is just to have coverage of MakeArray, then it's probably fine. If we want to be sure that MakeArray does a proper copy (which I assume is the intended semantics of MakeArray), then the additional mutation and assertion could be useful.

drin · 2022-08-23T00:32:56Z

+  ASSERT_ARRAYS_EQUAL(*new_array, *rle_array);
+  // should be the exact same ArrayData object
+  ASSERT_EQ(new_array->data(), array_data);
+  ASSERT_NE(std::dynamic_pointer_cast<RunLengthEncodedArray>(new_array), NULLPTR);


I'm not sure if this tests some detailed semantics, but I can't think of a scenario where the last assertion, ASSERT_NE, fails but the previous assertion would succeed.

If Make is OK and MakeArray returns a pointer to the same data, won't this always succeed?

drin · 2022-08-23T00:40:05Z

+  // one dummy buffer, since code may assume there is exactly one buffer
+  ASSERT_EQ(rle_array->data()->buffers.size(), 1);


Suggested change

// one dummy buffer, since code may assume there is exactly one buffer

ASSERT_EQ(rle_array->data()->buffers.size(), 1);

// validity buffer must exist and be empty because each RLE array entries must be valid

ASSERT_EQ(rle_array->data()->buffers.size(), 1);

ASSERT_EQ(rle_array->data()->buffers[0], NULLPTR);

I think this would be easier for newer developers to understand the semantics of the RLE array.

(Also, if this is incorrect then I probably need to re-assess my review so far lol)

This is correct since the RLE parent always has to have a null count of 0, and no validity bitmap. It can actually encode arrays with null values, but there are not set in the parent null_count, since we don't want the standard on bit per logical element validity bitmap. I find this a bit confusing, but this is how it works for Unions too. I tried to do it differently first, but Arrow code essentially assumes non-zero null count = has validity bitmap

drin · 2022-08-23T00:49:27Z

+  ASSERT_ARRAYS_EQUAL(*rle_array->values_array(), *string_values);
+  ASSERT_ARRAYS_EQUAL(*rle_array->run_ends_array(), *int32_values);
+  ASSERT_EQ(rle_array->offset(), 1);
+  // explicitly access null count variable so it is not calculated automatically


not sure what so it is not calculated automatically means here.

drin · 2022-08-23T01:06:09Z

+  Status Visit(const RunLengthEncodedArray& array) {
+    return Status::NotImplemented("printing run-length encoded array");
+  }


I just wanted to check if this is planned for this PR?

drin · 2022-08-23T01:30:28Z

+  using typename EncodeDecodeCommonExec<ArrowType, has_validity_buffer>::Element;
+
+  Status Exec() {
+    ArrayData* output_array_data = this->exec_result->array_data().get();


is this->exec_result->array_data().get() guaranteed to be valid/pre-allocated? Looking at VectorExecutor and ScalarExecutor it does seem so, but I just wanted to confirm.

(I actually have code that assumes it isn't, so I probably should revisit that)

drin · 2022-08-23T01:44:51Z

+    // calculate input null count by ourselves. The input span likely got sliced by an
+    // ExecSpanIterator using SetOffset right before this code executes. SetOffset leaves
+    // the null_count value of the ArraySpan as kUnknownNullCount.


is this because slice doesn't adjust the null count (to reduce overhead)?

yes it sets it to kUnknownNullCount, getting null count after that would cause it to interate over the validity buffer once more

drin · 2022-08-23T20:41:31Z

+      Element previous_element = element;
+      element = this->ReadValue();
+      if (element != previous_element) {


would it be better to declare this once before the loop, and then only set previous_element when element != previous_element?

Just a thought, since I'm not sure it's actually better performance.

zagto · 2022-08-24T15:33:58Z

+      output_array_data->length = 0;
+      output_array_data->offset = 0;
+      output_array_data->buffers = {NULLPTR};
+      output_array_data->child_data[0] =


note to self: this doesn't look right

apache#13821) Will fix [ARROW-13763](https://issues.apache.org/jira/browse/ARROW-13763) A separate Jira issue will be made to address closing files in V2 ParquetDataset, which needs to be handled in the C++ layer. Adds context manager to `pq.ParquetFile` to close input file, and ensure reads within `pq.ParquetDataset` and `pq.read_table` are closed. ```python # user opened file-like object will not be closed with open('file.parquet', 'rb') as f: with pq.ParquetFile(f) as p: table = p.read() assert not f.closed # did not inadvertently close the open file assert not p.closed assert not f.closed # parquet context exit didn't close it assert not p.closed # references the input file status assert f.closed # normal context exit close assert p.closed # ... # path-like will be closed upon exit or `ParquetFile.close` with pq.ParquetFile('file.parquet') as p: table = p.read() assert not p.closed assert p.closed ``` Authored-by: Miles Granger <[email protected]> Signed-off-by: Antoine Pitrou <[email protected]>

See https://issues.apache.org/jira/browse/ARROW-17289 Lead-authored-by: Yaron Gvili <[email protected]> Co-authored-by: rtpsw <[email protected]> Co-authored-by: Antoine Pitrou <[email protected]> Signed-off-by: Antoine Pitrou <[email protected]>

…ctor should limit to Integer.MAX_VALUE (apache#13815) We got a IndexOutOfBoundsException: ``` 2022-08-03 09:33:34,076 Error executing query, currentState RUNNING, java.lang.RuntimeException: org.apache.spark.SparkException: Job aborted due to stage failure: Task 3315 in stage 5.0 failed 4 times, most recent failure: Lost task 3315.3 in stage 5.0 (TID 3926) (30.97.116.209 executor 49): java.lang.IndexOutOfBoundsException: index: 2147312542, length: 777713 (expected: range(0, 2147483648)) at org.apache.iceberg.shaded.org.apache.arrow.memory.ArrowBuf.checkIndex(ArrowBuf.java:699) at org.apache.iceberg.shaded.org.apache.arrow.memory.ArrowBuf.setBytes(ArrowBuf.java:826) at org.apache.iceberg.arrow.vectorized.parquet.VectorizedParquetDefinitionLevelReader$VarWidthReader.nextVal(VectorizedParquetDefinitionLevelReader.java:418) at org.apache.iceberg.arrow.vectorized.parquet.VectorizedParquetDefinitionLevelReader$BaseReader.nextBatch(VectorizedParquetDefinitionLevelReader.java:235) at org.apache.iceberg.arrow.vectorized.parquet.VectorizedPageIterator$VarWidthTypePageReader.nextVal(VectorizedPageIterator.java:353) at org.apache.iceberg.arrow.vectorized.parquet.VectorizedPageIterator$BagePageReader.nextBatch(VectorizedPageIterator.java:161) at org.apache.iceberg.arrow.vectorized.parquet.VectorizedColumnIterator$VarWidthTypeBatchReader.nextBatchOf(VectorizedColumnIterator.java:191) at org.apache.iceberg.arrow.vectorized.parquet.VectorizedColumnIterator$BatchReader.nextBatch(VectorizedColumnIterator.java:74) at org.apache.iceberg.arrow.vectorized.VectorizedArrowReader.read(VectorizedArrowReader.java:158) at org.apache.iceberg.spark.data.vectorized.ColumnarBatchReader.read(ColumnarBatchReader.java:51) at org.apache.iceberg.spark.data.vectorized.ColumnarBatchReader.read(ColumnarBatchReader.java:35) at org.apache.iceberg.parquet.VectorizedParquetReader$FileIterator.next(VectorizedParquetReader.java:134) at org.apache.iceberg.spark.source.BaseDataReader.next(BaseDataReader.java:98) at org.apache.spark.sql.execution.datasources.v2.PartitionIterator.hasNext(DataSourceRDD.scala:79) at org.apache.spark.sql.execution.datasources.v2.MetricsIterator.hasNext(DataSourceRDD.scala:112) at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37) at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.columnartorow_nextBatch_0$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source) at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:755) at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458) ``` The root cause is the following code of `BaseVariableWidthVector.handleSafe` could fail to reallocate because of int overflow and then led to `IndexOutOfBoundsException` when we put the data into the vector. ```java protected final void handleSafe(int index, int dataLength) { while (index >= getValueCapacity()) { reallocValidityAndOffsetBuffers(); } final int startOffset = lastSet < 0 ? 0 : getStartOffset(lastSet + 1); // startOffset + dataLength could overflow while (valueBuffer.capacity() < (startOffset + dataLength)) { reallocDataBuffer(); } } ``` The offset width of `BaseVariableWidthVector` is 4, while the maximum memory allocation is Long.MAX_VALUE. This makes the memory allocation check invalid. Authored-by: xianyangliu <[email protected]> Signed-off-by: Antoine Pitrou <[email protected]>

Currently, Java JNI builds on Github Actions can take one hour due to a very long Arrow C++ build phase (example: https://github.com/apache/arrow/runs/7881918943?check_suite_focus=true#step:6:3512). Disable unused Arrow C++ components so as to make the C++ build faster. Authored-by: Antoine Pitrou <[email protected]> Signed-off-by: Antoine Pitrou <[email protected]>

…nt updates (apache#13769) Building on apache#12157 Lead-authored-by: Jacob Wujciak-Jens <[email protected]> Co-authored-by: Jonathan Keane <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>

Add `--validate` option to `archery crossbow status`. If `--validate` is specified and there are any missing artifacts, `archery crossbow status --validate` is existed with non-zero exit code. We can use it for CI to detect missing artifacts. We can't use `@github-actions crossbow submit` for this change because this isn't merged into the master branch yet. See https://github.com/ursacomputing/crossbow/branches/all?query=build-674 that is submitted `nightly-packages` manually. Authored-by: Sutou Kouhei <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>

…and *-glib-devel should have .gir (apache#13876) The current configuration is inverted. *-glib-libs have .gir and *-glib-devel have .typelib. Authored-by: Sutou Kouhei <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>

…pache#13910) I noticed looking at pkg.go.dev that there really isn't anyone who is using the existing `compute` module, which makes sense since it isn't really finished and only provides limited utility currently. This change will mark the `compute` module as a separate sub-module inside of the `arrow` module, allowing us to use `go1.18` in this new code without forcing anyone who *isn't* using the compute module to upgrade. That way I can leverage the generics when writing the new compute code where appropriate. Authored-by: Matt Topol <[email protected]> Signed-off-by: Matt Topol <[email protected]>

… GetSchema (apache#13898) Consistently implements and tests the GetSchema method in Flight SQL. Builds on apache#13897. Authored-by: David Li <[email protected]> Signed-off-by: David Li <[email protected]>

…lization for `LocalFileSystem` (apache#13796) Introduce a specialization of `GetFileInfoGenerator` in the `LocalFileSystem` class. This implementation tries to improves performance by hiding latencies at two levels: 1. Child directories can be readahead so that listing directories entries from disk can be achieved in parallel with other work; 2. Directory entries can be `stat`'ed and yielded in chunks so that the `FileInfoGenerator` consumer can start receiving entries before a large directory is fully processed. Both mechanisms can be tuned using dedicated parameters in `LocalFileSystemOptions`. Signed-off-by: Pavel Solodovnikov <[email protected]> Co-Authored-by: Igor Seliverstov <[email protected]> Lead-authored-by: Pavel Solodovnikov <[email protected]> Co-authored-by: Antoine Pitrou <[email protected]> Signed-off-by: Antoine Pitrou <[email protected]>

It looks like the entries in the truth tables were copy-pasted and the _results_ were updated to match the function, but not the operator. Authored-by: Gil Forsyth <[email protected]> Signed-off-by: Yibo Cai <[email protected]>

In this PR I: - reduced the scope of the automatically generated `GITHUB_TOKEN` as much as possible (technically `contents:none` would be the minimum but it is a bit unintuitive as it does not prevent checkout of public repos, I set `contents:read` in those cases) - update all actions used to the newest version (checking for breaking changes, only case is actions/github-script which remains on v3 for that reason -> follow up) - move the creation of envvars containing secrets as close to their usage as possible (-> the step they are used in), this duplicates them in workflows with multiple jobs but is safer. I have opted **NOT** to pin the different actions by SHA as recommended in some places as the con outweigh the possible protection in my opinion. The main danger with pinning tags or branches is that a malicious actor changes the commit the tag points to and exfiltrates secrets (either repository secrets or in case of private repos code/ip) or takes some other damaging action like deleting branches, rewriting history etc.. We only ever pass actions the `GITHUB_TOKEN` which is ephemeral (deleted after workflow is finished) and scope limited so exfiltration of that token would worst case allow an attacker to create/delete labels and pr comments as well as modify PR branches (if the submitter activated the checkbox for maintainer access). Actions can not access secrets without the workflow author explicitly passing them as input (envvars might reveal them though) The Apache Org limits the actions that can be used in repos, so we only use well known allow-listed actions, while this does of course not prevent malicious actions it reduces the risk substantially. Pinning SHAs would mitigate these risks (provided the action at that sha was audited...) but would also necessitate regularly checking + re-auditing the actions as to not miss security patches in these actions (e.g. [here](https://github.com/matlab-actions/setup-matlab/releases/tag/v1.1.1)). IMHO that would be a considerable effort (+ needing real expertise in typescript/node to spot any malicious additions outside of blatant secret exfiltration or nuking) resulting in a small gain. Lead-authored-by: Jacob Wujciak-Jens <[email protected]> Co-authored-by: assignUser <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>

…resolved (apache#14209) This just skips the test to stop it from obscuring other failures until we diagnose it properly (ARROW-17818). Authored-by: Dewey Dunnington <[email protected]> Signed-off-by: Dewey Dunnington <[email protected]>

This is a follow-up of ARROW-17621 / apache#14155. Authored-by: Sutou Kouhei <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>

…apache#14197) Authored-by: David Li <[email protected]> Signed-off-by: David Li <[email protected]>

Don't document the javadocs to avoid errors when there is nothing documentable in the javadocs Authored-by: David Li <[email protected]> Signed-off-by: David Li <[email protected]>

…apache#14210) I couldn't reproduce it, so I added a suppression instead. In both cases, the error is that the server is uncontactable. That shouldn't happen, but I changed the tests to also bind to port 0 instead of using a potentially flaky free port finder. Authored-by: David Li <[email protected]> Signed-off-by: David Li <[email protected]>

This is a follow-up of apache#14204. Authored-by: Sutou Kouhei <[email protected]> Signed-off-by: David Li <[email protected]>

…tructor with new (apache#14216) Advantage: readabilty, exception safety and efficiency(only for shared_ptr). Cases that don't apply: When calling a private/protected constructor within class member function, make_shared/unique can't work. Authored-by: Jin Shang <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>

…pache#14228) Temporarily pin LLVM version on Appveyor due to a bug in Conda's packaging of LLVM. Authored-by: Jin Shang <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>

) This is a follow-up of apache#14216. We can't use std::make_shared for CUDA related classes because their constructors aren't public. Authored-by: Sutou Kouhei <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>

…th output buffer (apache#14230) When the output type of an expression is of variable length, e.g. string, Gandiva would realloc the output buffer to make space for new outputs for each row. When num of rows is high some memory allocators perform poorly. We can use the std::vector like approach to amortize the allcation cost. First allocate some initial space depending on the input size. Each time we run out of space, double the buffer size. In the end shrink it to fit the actual size. Arrow string builder also uses this approach. Authored-by: Jin Shang <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>

TweakValidityBit returns a new Array so the calling function should use it. https://github.com/apache/arrow/blob/6cc37cf2d1ba72c46b64fbc7ac499bd0d7296d20/cpp/src/arrow/testing/gtest_util.cc#L568-L579 Authored-by: kshitij12345 <[email protected]> Signed-off-by: David Li <[email protected]>

… OpenTelemetry propagation (apache#11920) Adds a client middleware that sends span/trace ID to the server, and a server middleware that gets the span/trace ID and starts a child span. The middleware are available in builds without OpenTelemetry, they simply do nothing. Authored-by: David Li <[email protected]> Signed-off-by: David Li <[email protected]>

Use `File.deleteOnExit` to delete jni lib file on JVM exit. `File.deleteOnExit` actually add a shut down hook to make sure file delte. Authored-by: jackylee-ch <[email protected]> Signed-off-by: David Li <[email protected]>

* Closes #13330 * Closes: #32105 Lead-authored-by: Felipe Oliveira Carvalho <[email protected]> Co-authored-by: Tobias Zagorni <[email protected]> Signed-off-by: Matt Topol <[email protected]>

zagto force-pushed the rle branch from e8fe5ba to f37700e Compare June 7, 2022 14:53

github-actions Bot added the Component: C++ label Jun 7, 2022

zagto commented Jun 7, 2022

View reviewed changes

westonpace self-requested a review June 23, 2022 15:05

drin reviewed Aug 16, 2022

View reviewed changes

github-actions Bot added the Component: Parquet label Aug 19, 2022

zagto force-pushed the rle branch from 463c90b to 6766407 Compare August 19, 2022 01:16

drin reviewed Aug 22, 2022

View reviewed changes

drin reviewed Aug 23, 2022

View reviewed changes

zagto commented Aug 24, 2022

View reviewed changes

milesgranger and others added 12 commits October 7, 2022 22:59

enable compression for rust (apache#13893)

9596839

ARROW-12590: [C++][R] Update copies of Homebrew files to reflect rece…

94c8cca

…nt updates (apache#13769) Building on apache#12157 Lead-authored-by: Jacob Wujciak-Jens <[email protected]> Co-authored-by: Jonathan Keane <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>

ARROW-17254: [C++][Go][Java][FlightRPC] Implement and test Flight SQL…

4937adc

… GetSchema (apache#13898) Consistently implements and tests the GetSchema method in Flight SQL. Builds on apache#13897. Authored-by: David Li <[email protected]> Signed-off-by: David Li <[email protected]>

assignUser and others added 22 commits October 7, 2022 23:00

MINOR: [Dev] Fix actions/labeler's tag (apache#14215)

b364f5f

This is a follow-up of ARROW-17621 / apache#14155. Authored-by: Sutou Kouhei <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>

ARROW-17810: [Java] Use jacoco-maven-plugin 0.8.8 for Java 18 support (…

afe53e3

…apache#14197) Authored-by: David Li <[email protected]> Signed-off-by: David Li <[email protected]>

ARROW-17787: [Java] Fix Javadoc build (apache#14212)

dc3712b

Don't document the javadocs to avoid errors when there is nothing documentable in the javadocs Authored-by: David Li <[email protected]> Signed-off-by: David Li <[email protected]>

ARROW-17814: [C++] Fix style (apache#14218)

69b73d8

This is a follow-up of apache#14204. Authored-by: Sutou Kouhei <[email protected]> Signed-off-by: David Li <[email protected]>

ARROW-17830: [C++][Gandiva] Temporarily pin LLVM version on AppVeyor (a…

2756b9b

…pache#14228) Temporarily pin LLVM version on Appveyor due to a bug in Conda's packaging of LLVM. Authored-by: Jin Shang <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>

type_fwd: C++17 compatibility

c7f2440

rle_util: support different types for run ends array

90b15aa

fix run-ends type detection in GetPhysicalOffset and GetPhysicalLength

a96883e

test mutltple run ends types in rle offset/length test

efe28e2

rle validate: support differrent run ends types

4ef444e

Merge branch 'rle-util' into rle

823105d

Merge branch 'rle-validate' into rle

63c6a78

rle encode/decode: support multiple run ends types

ab6dba8

github-actions Bot added the Component: Python label Oct 10, 2022

make sure to always initialize Element struct

606c7d7

felipecrv mentioned this pull request Jan 17, 2023

GH-32104: [C++] Add support for Run-End encoded data to Arrow #33641

Merged

asfimport mentioned this pull request Nov 24, 2022

[C++] Implement encode and decode functions for Run-Length encoding #32105

Closed

felipecrv mentioned this pull request Mar 7, 2023

GH-32105: [C++] Encode and decode Run-End Encoded vectors #34195

Merged

zeroshade closed this in #34195 Mar 13, 2023

asfimport mentioned this pull request Sep 23, 2026

[C++] Complete RunLengthEncoded type #32114

Closed

	// If an ArrayData is slightly malformed it may have kUnknownNullCount set
	// If an ArraySpan is slightly malformed it may have kUnknownNullCount set

		data_->child_data.push_back(std::move(run_ends_array->data()));
		data_->child_data.push_back(std::move(values_array->data()));

		ASSERT_ARRAYS_EQUAL(new_array, rle_array);
		// should be the exact same ArrayData object

		// one dummy buffer, since code may assume there is exactly one buffer
		ASSERT_EQ(rle_array->data()->buffers.size(), 1);

Conversation

zagto commented Jun 7, 2022

Uh oh!

github-actions Bot commented Jun 7, 2022

Uh oh!

github-actions Bot commented Jun 7, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

westonpace commented Aug 11, 2022

Uh oh!

drin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zagto commented Aug 19, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

drin commented Aug 23, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

drin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

drin Aug 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zagto Aug 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

drin Aug 23, 2022 •

edited

Loading

zagto Aug 24, 2022 •

edited

Loading