Skip to content

support gcp oauth for s3 external table and iceberg#1088

Merged
yuzifeng1984 merged 6 commits intodevelopfrom
feat/issue-1082-gcp-auth
Jan 29, 2026
Merged

support gcp oauth for s3 external table and iceberg#1088
yuzifeng1984 merged 6 commits intodevelopfrom
feat/issue-1082-gcp-auth

Conversation

@yuzifeng1984
Copy link
Copy Markdown
Collaborator

@yuzifeng1984 yuzifeng1984 commented Jan 19, 2026

#1082 Experimentally support GCP OAuth in S3 external table and Iceberg.

Fetch bearer token from 'http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token' (configurable) in sending HTTP requests for both rest and storage requests.

-- Iceberg  database: auth_type='gcp_oauth'
CREATE DATABASE demo
SETTINGS
    type = 'iceberg',
    catalog_uri = 'https://biglake.googleapis.com/iceberg/v1/restcatalog',
    catalog_type = 'rest',
    project_id = '<PROJECT_ID>',
    warehouse = 'gs://zifeng-temp',
    storage_endpoint = 'gs://zifeng-temp/demo',
    auth_type='gcp_oauth';

-- S3 external table: http_client='gcp_oauth'
CREATE EXTERNAL TABLE IF NOT EXISTS s3_out_csv
(
  id uint64,
  key string,
  payload string,
  ts datetime
)
SETTINGS
  type = 's3',
  endpoint = 'gs://zifeng-temp',
  bucket = 'zifeng-temp',
  write_to = 'example/s3_csv.csv',
  http_client='gcp_oauth',
  data_format = 'CSV';

@yokofly
Copy link
Copy Markdown
Collaborator

yokofly commented Jan 19, 2026

@codex review

Copy link
Copy Markdown
Contributor

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4e09d78da3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chenziliang
Copy link
Copy Markdown
Collaborator

@yuzifeng1984 , let's rebase and get it merged

@yuzifeng1984 yuzifeng1984 force-pushed the feat/issue-1082-gcp-auth branch from df6ae21 to 06a48e7 Compare January 29, 2026 08:07
@yuzifeng1984 yuzifeng1984 merged commit 1823242 into develop Jan 29, 2026
19 of 20 checks passed
@yuzifeng1984 yuzifeng1984 deleted the feat/issue-1082-gcp-auth branch January 29, 2026 12:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants