Background

as of 0.2.10, Seafowl is able to read external table with http(s) protocol, e.g.

• files in public S3 bucket
• pre-signed file in private S3 bucket

However, it fails to read private S3 files without pre-signed URL. E.g. register a S3 folder, which contains many files. File path could follow hive partitioned conventions.

Assume the seaflow config already use S3 object store in bucketA and the IAM user also have read access to another private bucket bucketB

[object_store]
type = "s3"
access_key_id = "******"
secret_access_key = "*"
region = "us-east-1"
bucket = "bucketA"

Expect

curl -i -H "Content-Type: application/json" localhost:8080/q -H "Authorization: Bearer ******" -d@- <<EOF
{"query": "CREATE EXTERNAL TABLE tripdata \
STORED AS PARQUET \
LOCATION 's3://bucketB/yellow_tripdata_2022-10.parquet'
"}
EOF

HTTP/1.1 200

Actual

HTTP/1.1 400 Bad Request

Internal error: No suitable object store found for s3://bucketB/yellow_tripdata_2022-10.parquet