Update contribe/orc in .gitmodules by santrancisco · Pull Request #50920 · ClickHouse/ClickHouse

santrancisco · 2023-06-13T09:28:32Z

Description

I want to start clearing some dependabot alerts - Checked with @Felixoid and for these forked modules, the following steps are necessary to make sure nothing breaks:

Created a new branch - https://github.com/ClickHouse/orc/tree/sync_upstream
Synced the branch with upstream https://github.com/apache/orc
Make pull request to ClickHouse/ClickHouse - Checking out submodule and make commits - git 101 lesson from Misha: NOT to change .gitmodule file directly 😆

Changelog category (leave one):

Not for changelog (changelog entry is not required)

robot-ch-test-poll4 · 2023-06-13T09:28:59Z

This is an automated comment for commit f0dcefa with description of existing statuses. It's updated for the latest CI running
The full report is available here
The overall status of the commit is 🟡 pending

Check name	Description	Status
CI running	A meta-check that indicates the running CI. Normally, it's in success or pending state. The failed status indicates some problems with the PR	🟡 pending
Fast test	Normally this is the first check that is ran for a PR. It builds ClickHouse and runs most of stateless functional tests, omitting some. If it fails, further checks are not started until it is fixed. Look at the report to see which tests fail, then reproduce the failure locally as described here	🟢 success
Mergeable Check	Checks if all other necessary checks are successful	🟢 success
Push to Dockerhub	The check for building and pushing the CI related docker images to docker hub	🟢 success
Style Check	Runs a set of checks to keep the code style clean. If some of tests failed, see the related log from the report	🟢 success

nickitat · 2023-06-13T18:37:14Z

apparently you need to fix our custom cmake for this project. do you need help with that?

santrancisco · 2023-06-13T21:14:28Z

apparently you need to fix our custom cmake for this project. do you need help with that?

Yes please - sorry for adding another task to your plate - it's probably quicker for you to resolve this and I can learn from it than me monkey this up 😓

nickitat · 2023-06-13T22:55:29Z

apparently you need to fix our custom cmake for this project. do you need help with that?

Yes please - sorry for adding another task to your plate - it's probably quicker for you to resolve this and I can learn from it than me monkey this up 😓

this patch should be enough to fix:

diff --git a/contrib/arrow-cmake/CMakeLists.txt b/contrib/arrow-cmake/CMakeLists.txt
index 16198887075..5fe942d1cd0 100644
--- a/contrib/arrow-cmake/CMakeLists.txt
+++ b/contrib/arrow-cmake/CMakeLists.txt
@@ -116,43 +116,79 @@ configure_file("${ORC_SOURCE_SRC_DIR}/Adaptor.hh.in" "${ORC_BUILD_INCLUDE_DIR}/A
 # ARROW_ORC + adapters/orc/CMakefiles
 set(ORC_SRCS
         "${CMAKE_CURRENT_BINARY_DIR}/orc_proto.pb.h"
-        "${ORC_SOURCE_SRC_DIR}/sargs/ExpressionTree.cc"
-        "${ORC_SOURCE_SRC_DIR}/sargs/Literal.cc"
-        "${ORC_SOURCE_SRC_DIR}/sargs/PredicateLeaf.cc"
-        "${ORC_SOURCE_SRC_DIR}/sargs/SargsApplier.cc"
-        "${ORC_SOURCE_SRC_DIR}/sargs/SearchArgument.cc"
-        "${ORC_SOURCE_SRC_DIR}/sargs/TruthValue.cc"
-        "${ORC_SOURCE_SRC_DIR}/Exceptions.cc"
-        "${ORC_SOURCE_SRC_DIR}/OrcFile.cc"
-        "${ORC_SOURCE_SRC_DIR}/Reader.cc"
+        "${ORC_ADDITION_SOURCE_DIR}/orc_proto.pb.cc"
+        "${ORC_SOURCE_SRC_DIR}/Adaptor.cc"
+        "${ORC_SOURCE_SRC_DIR}/Adaptor.hh.in"
+        "${ORC_SOURCE_SRC_DIR}/BlockBuffer.cc"
+        "${ORC_SOURCE_SRC_DIR}/BlockBuffer.hh"
+        "${ORC_SOURCE_SRC_DIR}/BloomFilter.cc"
+        "${ORC_SOURCE_SRC_DIR}/BloomFilter.hh"
+        "${ORC_SOURCE_SRC_DIR}/Bpacking.hh"
+        "${ORC_SOURCE_SRC_DIR}/BpackingDefault.cc"
+        "${ORC_SOURCE_SRC_DIR}/BpackingDefault.hh"
         "${ORC_SOURCE_SRC_DIR}/ByteRLE.cc"
+        "${ORC_SOURCE_SRC_DIR}/ByteRLE.hh"
+        "${ORC_SOURCE_SRC_DIR}/CMakeLists.txt"
         "${ORC_SOURCE_SRC_DIR}/ColumnPrinter.cc"
         "${ORC_SOURCE_SRC_DIR}/ColumnReader.cc"
+        "${ORC_SOURCE_SRC_DIR}/ColumnReader.hh"
         "${ORC_SOURCE_SRC_DIR}/ColumnWriter.cc"
+        "${ORC_SOURCE_SRC_DIR}/ColumnWriter.hh"
         "${ORC_SOURCE_SRC_DIR}/Common.cc"
         "${ORC_SOURCE_SRC_DIR}/Compression.cc"
+        "${ORC_SOURCE_SRC_DIR}/Compression.hh"
+        "${ORC_SOURCE_SRC_DIR}/ConvertColumnReader.cc"
+        "${ORC_SOURCE_SRC_DIR}/ConvertColumnReader.hh"
+        "${ORC_SOURCE_SRC_DIR}/CpuInfoUtil.cc"
+        "${ORC_SOURCE_SRC_DIR}/CpuInfoUtil.hh"
+        "${ORC_SOURCE_SRC_DIR}/Dispatch.hh"
+        "${ORC_SOURCE_SRC_DIR}/Exceptions.cc"
         "${ORC_SOURCE_SRC_DIR}/Int128.cc"
         "${ORC_SOURCE_SRC_DIR}/LzoDecompressor.cc"
+        "${ORC_SOURCE_SRC_DIR}/LzoDecompressor.hh"
         "${ORC_SOURCE_SRC_DIR}/MemoryPool.cc"
+        "${ORC_SOURCE_SRC_DIR}/Murmur3.cc"
+        "${ORC_SOURCE_SRC_DIR}/Murmur3.hh"
+        "${ORC_SOURCE_SRC_DIR}/Options.hh"
+        "${ORC_SOURCE_SRC_DIR}/OrcFile.cc"
         "${ORC_SOURCE_SRC_DIR}/RLE.cc"
+        "${ORC_SOURCE_SRC_DIR}/RLE.hh"
+        "${ORC_SOURCE_SRC_DIR}/RLEV2Util.cc"
+        "${ORC_SOURCE_SRC_DIR}/RLEV2Util.hh"
         "${ORC_SOURCE_SRC_DIR}/RLEv1.cc"
+        "${ORC_SOURCE_SRC_DIR}/RLEv1.hh"
+        "${ORC_SOURCE_SRC_DIR}/RLEv2.hh"
+        "${ORC_SOURCE_SRC_DIR}/Reader.cc"
+        "${ORC_SOURCE_SRC_DIR}/Reader.hh"
         "${ORC_SOURCE_SRC_DIR}/RleDecoderV2.cc"
         "${ORC_SOURCE_SRC_DIR}/RleEncoderV2.cc"
-        "${ORC_SOURCE_SRC_DIR}/RLEV2Util.cc"
+        "${ORC_SOURCE_SRC_DIR}/SchemaEvolution.cc"
+        "${ORC_SOURCE_SRC_DIR}/SchemaEvolution.hh"
         "${ORC_SOURCE_SRC_DIR}/Statistics.cc"
+        "${ORC_SOURCE_SRC_DIR}/Statistics.hh"
         "${ORC_SOURCE_SRC_DIR}/StripeStream.cc"
+        "${ORC_SOURCE_SRC_DIR}/StripeStream.hh"
         "${ORC_SOURCE_SRC_DIR}/Timezone.cc"
+        "${ORC_SOURCE_SRC_DIR}/Timezone.hh"
         "${ORC_SOURCE_SRC_DIR}/TypeImpl.cc"
+        "${ORC_SOURCE_SRC_DIR}/TypeImpl.hh"
+        "${ORC_SOURCE_SRC_DIR}/Utils.hh"
         "${ORC_SOURCE_SRC_DIR}/Vector.cc"
         "${ORC_SOURCE_SRC_DIR}/Writer.cc"
-        "${ORC_SOURCE_SRC_DIR}/Adaptor.cc"
-        "${ORC_SOURCE_SRC_DIR}/BloomFilter.cc"
-        "${ORC_SOURCE_SRC_DIR}/Murmur3.cc"
-        "${ORC_SOURCE_SRC_DIR}/BlockBuffer.cc"
-        "${ORC_SOURCE_SRC_DIR}/wrap/orc-proto-wrapper.cc"
         "${ORC_SOURCE_SRC_DIR}/io/InputStream.cc"
+        "${ORC_SOURCE_SRC_DIR}/io/InputStream.hh"
         "${ORC_SOURCE_SRC_DIR}/io/OutputStream.cc"
-        "${ORC_ADDITION_SOURCE_DIR}/orc_proto.pb.cc"
+        "${ORC_SOURCE_SRC_DIR}/io/OutputStream.hh"
+        "${ORC_SOURCE_SRC_DIR}/sargs/ExpressionTree.cc"
+        "${ORC_SOURCE_SRC_DIR}/sargs/ExpressionTree.hh"
+        "${ORC_SOURCE_SRC_DIR}/sargs/Literal.cc"
+        "${ORC_SOURCE_SRC_DIR}/sargs/PredicateLeaf.cc"
+        "${ORC_SOURCE_SRC_DIR}/sargs/PredicateLeaf.hh"
+        "${ORC_SOURCE_SRC_DIR}/sargs/SargsApplier.cc"
+        "${ORC_SOURCE_SRC_DIR}/sargs/SargsApplier.hh"
+        "${ORC_SOURCE_SRC_DIR}/sargs/SearchArgument.cc"
+        "${ORC_SOURCE_SRC_DIR}/sargs/SearchArgument.hh"
+        "${ORC_SOURCE_SRC_DIR}/sargs/TruthValue.cc"
         )

santrancisco · 2023-06-14T03:19:52Z

Hmmm it's still broken but looking at the log and apache orc code it seems they have a problem here...

https://github.com/apache/orc/blob/f2c191f9653a5ddbca016e024ca0fb61508f5eeb/c%2B%2B/src/CpuInfoUtil.cc#L475

ArchParseUserSimdLevel is only defined when CPUINFO_ARCH_X86 is set. :-/

~~With that said, do we need/use "CpuInfoUtil.cc" ? or it's ok to remove them from the above~~? 🤔 scratch that - it is needed

santrancisco · 2023-06-21T03:21:45Z

Almost there - the folks looking after apache orc added support for ARMs which cleaned up most of the failed workflow except for powerpc build. i'm asking them for help to add this architect to their build as well. Once they merge the fix, i will update our local fork and merge this pull.

…is now merged into apache/orc repository

* Bump protobuf to v3.18.x (ref: ClickHouse/ClickHouse#50400) * Bump protobuf to v3.19.6 (ref: ClickHouse/ClickHouse#55963) * Bump protobuf to v3.19.6 (ref: ClickHouse/ClickHouse#50920) * Update contribe/orc (ref: ClickHouse/ClickHouse#53324) * Bump protobuf to v21.9 (ref: ClickHouse/ClickHouse#56014) * Removed unused code in google-protobuf-cmake/CMakeLists.txt (ref: ClickHouse/ClickHouse#56102) * Bump gRPC to v1.55.4 and protobuf to v22.5 (ref: ClickHouse/ClickHouse#56664) * Bump gRPC to v1.59 and protobuf to v24.4 (ref: ClickHouse/ClickHouse#56853) * enable functions in readonly mode (ref: ClickHouse/ClickHouse#42708) * Introduce more fields for in-source function documentation (ref: ClickHouse/ClickHouse#49300) * Add ability to convert timeplusd table structure to CapnProto/Protobuf format schema (ref: ClickHouse/ClickHouse#52278) * Well known protos (ref: ClickHouse/ClickHouse#56741) * Update protobuf to v25.1 (ref: ClickHouse/ClickHouse#58020) * fixed unit_tests_dbms build * Avoid ignoring errors of execute_process() (set COMMAND_ERROR_IS_FATAL=ANY) (ref: ClickHouse/ClickHouse#68267) * fixed reigsterFunction * fixed error after rebase

santrancisco requested a review from Felixoid June 13, 2023 09:28

robot-ch-test-poll4 added the pr-not-for-changelog This PR should not be mentioned in the changelog label Jun 13, 2023

Update orc submodule

a01056f

santrancisco force-pushed the update_contrib_ocr_module branch from a1ad0a5 to a01056f Compare June 13, 2023 16:35

robot-clickhouse-ci-1 added the submodule changed At least one submodule changed in this PR. label Jun 13, 2023

santrancisco added the can be tested Allows running workflows for external contributors label Jun 13, 2023

nickitat self-assigned this Jun 13, 2023

alexey-milovidov approved these changes Jun 13, 2023

View reviewed changes

santrancisco mentioned this pull request Jun 13, 2023

Sync upstream ClickHouse/orc#6

Merged

Update CMakeLists.txt with help from Nikita

29b9cba

santrancisco added 5 commits June 14, 2023 13:29

test removing CpuInfoUtil.cc and see if build breaks :p

6ffdfb8

Re-added CpuInfoUtil to CMakeList.txt

197c761

Temporary check fix PR apache/orc#1542 (comment) with our build

3365a8f

Fixed wrong contrib orc submodule url i meant to test *facepalm*

c47e2a1

ran git checkout inside contrib/orc

26fc9ee

santrancisco added 2 commits June 21, 2023 21:44

Testing fix for PPC platform apache/orc#1542

a413722

Fixing git submodule back to our own fork as this pull apache/orc#1542 …

f0dcefa

…is now merged into apache/orc repository

Felixoid approved these changes Jun 22, 2023

View reviewed changes

santrancisco merged commit 6288a17 into master Jun 22, 2023

santrancisco deleted the update_contrib_ocr_module branch June 22, 2023 09:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update contribe/orc in .gitmodules#50920

Update contribe/orc in .gitmodules#50920
santrancisco merged 9 commits intomasterfrom
update_contrib_ocr_module

santrancisco commented Jun 13, 2023 •

edited

Loading

Uh oh!

robot-ch-test-poll4 commented Jun 13, 2023 •

edited by robot-ch-test-poll2

Loading

Uh oh!

nickitat commented Jun 13, 2023

Uh oh!

santrancisco commented Jun 13, 2023

Uh oh!

nickitat commented Jun 13, 2023

Uh oh!

santrancisco commented Jun 14, 2023 •

edited

Loading

Uh oh!

santrancisco commented Jun 21, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

santrancisco commented Jun 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changelog category (leave one):

Uh oh!

robot-ch-test-poll4 commented Jun 13, 2023 • edited by robot-ch-test-poll2 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nickitat commented Jun 13, 2023

Uh oh!

santrancisco commented Jun 13, 2023

Uh oh!

nickitat commented Jun 13, 2023

Uh oh!

santrancisco commented Jun 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

santrancisco commented Jun 21, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

santrancisco commented Jun 13, 2023 •

edited

Loading

robot-ch-test-poll4 commented Jun 13, 2023 •

edited by robot-ch-test-poll2

Loading

santrancisco commented Jun 14, 2023 •

edited

Loading