Skip to content

[DRAFT] Build: remove hadoop 2 support#10932

Closed
steveloughran wants to merge 1 commit intoapache:mainfrom
steveloughran:hadoop3-only
Closed

[DRAFT] Build: remove hadoop 2 support#10932
steveloughran wants to merge 1 commit intoapache:mainfrom
steveloughran:hadoop3-only

Conversation

@steveloughran
Copy link
Copy Markdown
Contributor

@steveloughran steveloughran commented Aug 13, 2024

Now that the minimum java version is 11, it's impossible for Iceberg to work on a Hadoop release less than 3.3.0.a

Removing the hadoop2 version and libraries forces all building and testing onto a compatible version, and permits followup work using modern hadoop APIs.

@steveloughran
Move Iceberg to hadoop 3
76767df 
Now that the minimum java version is 11, it's impossible for
Iceberg to work on a Hadoop release less than 3.3.0.

Removing the hadoop2 version and libraries forces all building
and testing onto a compatible version, and permits followup
work using modern hadoop APIs.
@steveloughran steveloughran marked this pull request as draft August 13, 2024 17:03
@steveloughran
Copy link
Copy Markdown
Contributor Author

steveloughran commented Aug 13, 2024

hive2 tests are failing with what looks like guava version/access issues.

TestHiveIcebergStorageHandlerWithEngine > testDescribeTable() > fileFormat=PARQUET, engine=mr, catalog=CUSTOM_CATALOG, isVectorized=false FAILED
    java.lang.IllegalArgumentException: Failed to execute Hive query 'DESCRIBE default.customers': Error running query: java.lang.IllegalAccessError: class org.apache.hadoop.hive.ql.exec.FetchOperator tried to access method 'com.google.common.collect.UnmodifiableIterator com.google.common.collect.Iterators.emptyIterator()' (org.apache.hadoop.hive.ql.exec.FetchOperator and com.google.common.collect.Iterators are in unnamed module of loader 'app')
        at org.apache.iceberg.mr.hive.TestHiveShell.executeStatement(TestHiveShell.java:164)
        at org.apache.iceberg.mr.hive.TestHiveIcebergStorageHandlerWithEngine.testDescribeTable(TestHiveIcebergStorageHandlerWithEngine.java:253)

        Caused by:
        org.apache.hive.service.cli.HiveSQLException: Error running query: java.lang.IllegalAccessError: class org.apache.hadoop.hive.ql.exec.FetchOperator tried to access method 'com.google.common.collect.UnmodifiableIterator com.google.common.collect.Iterators.emptyIterator()' (org.apache.hadoop.hive.ql.exec.FetchOperator and com.google.common.collect.Iterators are in unnamed module of loader 'app')
            at app//org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:238)
            at app//org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:290)
            at app//org.apache.hive.service.cli.operation.Operation.run(Operation.java:320)
            at app//org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:530)
            at app//org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:500)
            at app//org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:265)
            at app//org.apache.iceberg.mr.hive.TestHiveShell.executeStatement(TestHiveShell.java:150)
            ... 1 more

            Caused by:
            java.lang.IllegalAccessError: class org.apache.hadoop.hive.ql.exec.FetchOperator tried to access method 'com.google.common.collect.UnmodifiableIterator com.google.common.collect.Iterators.emptyIterator()' (org.apache.hadoop.hive.ql.exec.FetchOperator and com.google.common.collect.Iterators are in unnamed module of loader 'app')
                at org.apache.hadoop.hive.ql.exec.FetchOperator.<init>(FetchOperator.java:108)
                at org.apache.hadoop.hive.ql.exec.FetchTask.initialize(FetchTask.java:87)
                at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:541)
                at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
                at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1295)
                at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:204)
                ... 7 more

apparently Hive 2 also needs to be cut as it is only a hadoop 2.x runtime.

https://hive.apache.org/general/downloads/

@steveloughran steveloughran mentioned this pull request Aug 13, 2024
Merged
@Fokko Fokko mentioned this pull request Aug 14, 2024
Closed
3 tasks
@Fokko
Copy link
Copy Markdown
Contributor

Fokko commented Aug 14, 2024

@steveloughran Thanks for taking a stab at this. I've created an issue here #10940 to track this. I'm also a bit constrained on time, so I'm open for someone else to take this on.

@manuzhang
Copy link
Copy Markdown
Member

manuzhang commented Aug 15, 2024

@Fokko @steveloughran I'd love to take this on if you guys are time constrained.

@Fokko
Copy link
Copy Markdown
Contributor

Fokko commented Aug 15, 2024

@manuzhang That would be great, thank you! 🙌

@steveloughran
Copy link
Copy Markdown
Contributor Author

steveloughran commented Aug 15, 2024

@manuzhang a key issue is hive2 has to go -which is not just a build profile, its the CI test setup.

That can be done in the same PR as this, or as a PR done in advance.

@github-actions
Copy link
Copy Markdown

github-actions bot commented Nov 11, 2024

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.

@github-actions github-actions bot added the stale label Nov 11, 2024
@github-actions
Copy link
Copy Markdown

github-actions bot commented Nov 18, 2024

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

@github-actions github-actions bot closed this Nov 18, 2024
This was referenced Feb 20, 2025
Draft
Merged
@xxubai xxubai mentioned this pull request Jun 13, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

ALIYUN build flink MR spark stale

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@steveloughran @Fokko @manuzhang