[C++] Run-end encoding and decoding: Add support for more types #34382

Activity

1. felipecrv

   added

   Type: enhancement

   on Feb 28, 2023
2. github-actions

   added

   Component: C++

   on Feb 28, 2023
3. dinimar commented on Mar 17, 2023

   

   dinimar

   on Mar 17, 2023

   Contributor

   More actions

   @felipecrv
   As I found out in code there're add_kernel() function calls related to types you mentioned
   

   arrow/cpp/src/arrow/compute/kernels/vector_run_end_encode.cc

   Lines 607 to 612 in f9324b7

   	for (const auto& ty : NumericTypes()) {
   	add_kernel(ty);
   	}
   	add_kernel(boolean());
   	add_kernel(null());
   	// TODO(GH-34195): Add support for more types

   
   

   arrow/cpp/src/arrow/compute/kernels/vector_run_end_encode.cc

   Lines 631 to 636 in f9324b7

   	for (const auto& ty : NumericTypes()) {
   	add_kernel(ty);
   	}
   	add_kernel(boolean());
   	add_kernel(null());
   	// TODO(GH-34195): Add support for more types

   But simple adding add_kernel() function calls doesn't work. Tests start failing after that. Could you please explain how to add types for run_end_encode and run_end_decode properly?

   About types. As I see, in theory, it's possible to add all types using add_kernel (at least, it's easy for functions with empty arguments) listed in:
   

   arrow/cpp/src/arrow/type_fwd.h

   Lines 456 to 590 in f9324b7

   	ARROW_EXPORT const std::shared_ptr<DataType>& utf8();
   	/// \brief Return a LargeStringType instance
   	ARROW_EXPORT const std::shared_ptr<DataType>& large_utf8();
   	/// \brief Return a BinaryType instance
   	ARROW_EXPORT const std::shared_ptr<DataType>& binary();
   	/// \brief Return a LargeBinaryType instance
   	ARROW_EXPORT const std::shared_ptr<DataType>& large_binary();
   	/// \brief Return a Date32Type instance
   	ARROW_EXPORT const std::shared_ptr<DataType>& date32();
   	/// \brief Return a Date64Type instance
   	ARROW_EXPORT const std::shared_ptr<DataType>& date64();
   	/// \brief Create a FixedSizeBinaryType instance.
   	ARROW_EXPORT
   	std::shared_ptr<DataType> fixed_size_binary(int32_t byte_width);
   	/// \brief Create a DecimalType instance depending on the precision
   	///
   	/// If the precision is greater than 38, a Decimal256Type is returned,
   	/// otherwise a Decimal128Type.
   	ARROW_EXPORT
   	std::shared_ptr<DataType> decimal(int32_t precision, int32_t scale);
   	/// \brief Create a Decimal128Type instance
   	ARROW_EXPORT
   	std::shared_ptr<DataType> decimal128(int32_t precision, int32_t scale);
   	/// \brief Create a Decimal256Type instance
   	ARROW_EXPORT
   	std::shared_ptr<DataType> decimal256(int32_t precision, int32_t scale);
   	/// \brief Create a ListType instance from its child Field type
   	ARROW_EXPORT
   	std::shared_ptr<DataType> list(const std::shared_ptr<Field>& value_type);
   	/// \brief Create a ListType instance from its child DataType
   	ARROW_EXPORT
   	std::shared_ptr<DataType> list(const std::shared_ptr<DataType>& value_type);
   	/// \brief Create a LargeListType instance from its child Field type
   	ARROW_EXPORT
   	std::shared_ptr<DataType> large_list(const std::shared_ptr<Field>& value_type);
   	/// \brief Create a LargeListType instance from its child DataType
   	ARROW_EXPORT
   	std::shared_ptr<DataType> large_list(const std::shared_ptr<DataType>& value_type);
   	/// \brief Create a MapType instance from its key and value DataTypes
   	ARROW_EXPORT
   	std::shared_ptr<DataType> map(std::shared_ptr<DataType> key_type,
   	std::shared_ptr<DataType> item_type,
   	bool keys_sorted = false);
   	/// \brief Create a MapType instance from its key DataType and value field.
   	///
   	/// The field override is provided to communicate nullability of the value.
   	ARROW_EXPORT
   	std::shared_ptr<DataType> map(std::shared_ptr<DataType> key_type,
   	std::shared_ptr<Field> item_field,
   	bool keys_sorted = false);
   	/// \brief Create a FixedSizeListType instance from its child Field type
   	ARROW_EXPORT
   	std::shared_ptr<DataType> fixed_size_list(const std::shared_ptr<Field>& value_type,
   	int32_t list_size);
   	/// \brief Create a FixedSizeListType instance from its child DataType
   	ARROW_EXPORT
   	std::shared_ptr<DataType> fixed_size_list(const std::shared_ptr<DataType>& value_type,
   	int32_t list_size);
   	/// \brief Return a Duration instance (naming use _type to avoid namespace conflict with
   	/// built in time classes).
   	ARROW_EXPORT std::shared_ptr<DataType> duration(TimeUnit::type unit);
   	/// \brief Return a DayTimeIntervalType instance
   	ARROW_EXPORT std::shared_ptr<DataType> day_time_interval();
   	/// \brief Return a MonthIntervalType instance
   	ARROW_EXPORT std::shared_ptr<DataType> month_interval();
   	/// \brief Return a MonthDayNanoIntervalType instance
   	ARROW_EXPORT std::shared_ptr<DataType> month_day_nano_interval();
   	/// \brief Create a TimestampType instance from its unit
   	ARROW_EXPORT
   	std::shared_ptr<DataType> timestamp(TimeUnit::type unit);
   	/// \brief Create a TimestampType instance from its unit and timezone
   	ARROW_EXPORT
   	std::shared_ptr<DataType> timestamp(TimeUnit::type unit, const std::string& timezone);
   	/// \brief Create a 32-bit time type instance
   	///
   	/// Unit can be either SECOND or MILLI
   	ARROW_EXPORT std::shared_ptr<DataType> time32(TimeUnit::type unit);
   	/// \brief Create a 64-bit time type instance
   	///
   	/// Unit can be either MICRO or NANO
   	ARROW_EXPORT std::shared_ptr<DataType> time64(TimeUnit::type unit);
   	/// \brief Create a StructType instance
   	ARROW_EXPORT std::shared_ptr<DataType> struct_(
   	const std::vector<std::shared_ptr<Field>>& fields);
   	/// \brief Create a RunEndEncodedType instance
   	ARROW_EXPORT std::shared_ptr<DataType> run_end_encoded(
   	std::shared_ptr<DataType> run_end_type, std::shared_ptr<DataType> value_type);
   	/// \brief Create a SparseUnionType instance
   	ARROW_EXPORT std::shared_ptr<DataType> sparse_union(FieldVector child_fields,
   	std::vector<int8_t> type_codes = {});
   	/// \brief Create a SparseUnionType instance
   	ARROW_EXPORT std::shared_ptr<DataType> sparse_union(
   	const ArrayVector& children, std::vector<std::string> field_names = {},
   	std::vector<int8_t> type_codes = {});
   	/// \brief Create a DenseUnionType instance
   	ARROW_EXPORT std::shared_ptr<DataType> dense_union(FieldVector child_fields,
   	std::vector<int8_t> type_codes = {});
   	/// \brief Create a DenseUnionType instance
   	ARROW_EXPORT std::shared_ptr<DataType> dense_union(
   	const ArrayVector& children, std::vector<std::string> field_names = {},
   	std::vector<int8_t> type_codes = {});
   	/// \brief Create a DictionaryType instance
   	/// \param[in] index_type the type of the dictionary indices (must be
   	/// a signed integer)
   	/// \param[in] dict_type the type of the values in the variable dictionary
   	/// \param[in] ordered true if the order of the dictionary values has
   	/// semantic meaning and should be preserved where possible
   	ARROW_EXPORT
   	std::shared_ptr<DataType> dictionary(const std::shared_ptr<DataType>& index_type,
   	const std::shared_ptr<DataType>& dict_type,
   	bool ordered = false);
4. felipecrv commented on Mar 17, 2023

   

   felipecrv

   on Mar 17, 2023

   ContributorAuthor

   More actions

   To add more types we need more specializations of ReadValueImpl and WriteValueImpl similaryly to how visit_data_inline.h does. The current specialization only works on types that have a ::c_type. There needs to be one for Binary, String and another for fixed-size byte-string types like FixedSizeBinary, Decimal256.

   If we can make equality of values a template parameter, RunEndEncodingLoop could also be re-used for these specializations.
5. felipecrv commented on Mar 28, 2023

   

   felipecrv

   on Mar 28, 2023

   ContributorAuthor

   More actions

   take
6. github-actions

   assigned

   felipecrv

   on Mar 28, 2023
7. westonpace

   closed this as completedin #34761on Apr 4, 2023
8. westonpace

   added a commit that references this issue on Apr 4, 2023

   GH-34382: [C++] Support more types in run_end_encode and run_end_deco…

   ...

   6d109fb
9. westonpace

   added this to the 12.0.0 milestone on Apr 4, 2023
10. ArgusLi

    added a commit that references this issue on May 15, 2023

    apacheGH-34382: [C++] Support more types in run_end_encode and run_en…

    ...

    d99ddd7