Skip to content

[Bug] [connector-file-s3] Lack of dependencies #7270

@Xuzhengz

Description

@Xuzhengz

Search before asking

  • I had searched in the issues and found no similar issues.

What happened

After the dev branch finishes packaging, run the s3 file plugin with an error message

SeaTunnel Version

dev

SeaTunnel Config

{
    "env": {
        "job.name": "文本文件输出测试",
        "job.mode": "batch"
    }, 
    "source": [
        {
            "plugin_name": "FakeSource",
            "resourceType": "nosql",
            "row.num": 100,
            "schema": {
                "table": "ottomi",
                "fields": {
                    "id": "string",
                    "name": "string"
                }
            },
            "resources": [
                "ottomi"
            ],
            "result_table_name": "ot_547a5338dd494638a8cfc2157f4fd06f",
            "ottomi_node_type": "FakeSource"
        }
    ],
    "transform": [

    ],
    "sink": [
        {
            "file_format_type": "text",
            "field_delimiter": ",",
            "custom_filename": true,
            "file_name_expression": "text_test",
            "is_enable_transaction": false,
            "batch_size": 1000000000,
            "plugin_name": "S3File",
            "bucket": "s3a://ottomi",
            "fs.s3a.endpoint": "******",
            "fs.s3a.aws.credentials.provider": "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider",
            "access_key": "P7HpCVa0Eth8X3dM",
            "secret_key": "xxM9cJL5QKqq9M7jvwkQ5LJgsQSVvIay",
            "tmp_path": "/tmp/ottomi",
            "path": "/file-node/download/1793520703906430978/text/",
            "result_table_name": "ot_8e1b8b2321db40cc8b62667b0ac794f1",
            "source_table_name": [
                "ot_547a5338dd494638a8cfc2157f4fd06f"
            ]
        }
    ]
}

Running Command

bin/seatunnel.sh -c demo.json

Error Exception

Caused by: java.io.IOException: org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider instantiation exception: java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;Ljava/lang/Object;)V
        at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:678)
        at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:566)
        at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:256)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3303)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:476)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:227)
        at org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopFileSystemProxy.initialize(HadoopFileSystemProxy.java:222)
        at org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopFileSystemProxy.getFileSystem(HadoopFileSystemProxy.java:171)
        at org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopFileSystemProxy.fileExist(HadoopFileSystemProxy.java:61)
        at org.apache.seatunnel.connectors.seatunnel.file.s3.catalog.S3FileCatalog.tableExists(S3FileCatalog.java:90)
        at org.apache.seatunnel.api.sink.DefaultSaveModeHandler.tableExists(DefaultSaveModeHandler.java:138)
        at org.apache.seatunnel.api.sink.DefaultSaveModeHandler.createSchemaWhenNotExist(DefaultSaveModeHandler.java:107)
        at org.apache.seatunnel.api.sink.DefaultSaveModeHandler.handleSchemaSaveMode(DefaultSaveModeHandler.java:69)
        at org.apache.seatunnel.api.sink.SaveModeHandler.handleSaveMode(SaveModeHandler.java:38)
        at org.apache.seatunnel.api.sink.SaveModeExecuteWrapper.execute(SaveModeExecuteWrapper.java:36)
        at org.apache.seatunnel.engine.server.master.JobMaster.handleSaveMode(JobMaster.java:373)
        ... 20 more
Caused by: java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;Ljava/lang/Object;)V
        at org.apache.hadoop.fs.s3a.S3AUtils.lookupPassword(S3AUtils.java:742)
        at org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider.<init>(SimpleAWSCredentialsProvider.java:59)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:635)
        ... 36 more

        at org.apache.seatunnel.engine.server.CoordinatorService.lambda$submitJob$4(CoordinatorService.java:506)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Zeta or Flink or Spark Version

No response

Java or Scala Version

1.8

Screenshots

image
image

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions