Skip to content

Commit f7e2667

Browse files
fix(catalog/rest): sign SigV4 with credentials from catalog properties (#1999)
* fix(catalog/rest): sign SigV4 with credentials from catalog properties The REST catalog's SigV4 signer only used the AWS default credential chain (config.LoadDefaultConfig), ignoring the s3.* credential properties passed to the catalog. This forced callers targeting AWS SigV4 REST catalogs (S3 Tables, Glue) to export AWS_* environment variables even when they supplied credentials via catalog properties. Build a static credentials provider from s3.access-key-id / s3.secret-access-key / s3.session-token when present, and only fall back to the default chain otherwise. An explicitly provided aws.Config (WithAwsConfig) still wins. This mirrors the same gap tracked for the Python client in apache/iceberg-python#2070. Signed-off-by: iremcaginyurtturk <[email protected]> * test(catalog/rest): pin SigV4 props-cred signing; error on partial creds Address review feedback: - Add TestSigV4SignsWithPropsCredentials, which asserts the SigV4 Authorization header is signed with the credentials from catalog properties. This pins the wiring: removing the cfg.Credentials assignment makes the test fail. - staticCredsFromProps now returns an error when only one of the access-key / secret-access-key pair is set, instead of silently falling back to the ambient default identity. Neither set still falls back to the default chain. Signed-off-by: iremcaginyurtturk <[email protected]> * docs(catalog/rest): document SigV4 signing credential precedence Address review feedback (minor): the SigV4 signing identity resolves as WithAwsConfig > s3.* catalog properties > AWS default credential chain. Document this on WithSigV4 and in the rest.sigv4-enabled configuration reference. Signed-off-by: iremcaginyurtturk <[email protected]> * fix(catalog/rest): error on incomplete SigV4 creds via ValidateStaticCredentials Delegate staticCredsFromProps to internal/awsconfig.ValidateStaticCredentials so a token-only s3.session-token (no access/secret key) returns the ErrIncompleteStaticCredentials sentinel instead of silently dropping to the default credential chain. Add a token-only test case and assert the sentinel via errors.Is. Document the s3.* vs Java rest.* divergence in the WithSigV4 godoc and configuration.md. * feat(catalog/rest): accept Java rest.* SigV4 credential aliases Accept rest.access-key-id / rest.secret-access-key / rest.session-token as aliases for the s3.* signing-credential properties, resolved per field with the s3.* keys taking precedence. This lets operators migrating from the Java client configure SigV4 signing with the property names they already use instead of silently falling through to the AWS default credential chain. * fix(catalog/rest): harden SigV4 credential boundaries Two credential-boundary fixes for the props-based SigV4 signing path: - Resolve the credential tuple atomically per namespace. Resolving each field independently let a partial s3.* pair be completed with a rest.* field (or inherit an unrelated rest.* token), signing as a hybrid identity that still passed validation. Now if any s3.* field is set we validate and use only the s3.* tuple, otherwise only the rest.* tuple; never backfill across namespaces. - Do not re-sign a cross-origin redirect. sessionTransport signed every hop, so a redirect to a different origin exposed the SigV4 Authorization header and session token to an unconfigured host. Signing is now skipped when the request origin (scheme/host/effective port) differs from the configured catalog origin. Tests cover the mixed-partial and s3-pair-plus-rest-token cases and assert a cross-origin redirect target receives neither the Authorization header nor the session token. Also closes a response body left open in an existing test. Signed-off-by: iremcaginyurtturk <[email protected]> --------- Signed-off-by: iremcaginyurtturk <[email protected]>
1 parent dd935d8 commit f7e2667

4 files changed

Lines changed: 237 additions & 3 deletions

File tree

‎catalog/rest/options.go‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,14 @@ func WithMetadataLocation(loc string) Option {
8888
}
8989
}
9090

91+
// WithSigV4 enables AWS SigV4 request signing for the REST catalog. The signing
92+
// identity is resolved in order: an explicit WithAwsConfig, then the s3.* catalog
93+
// credential properties (s3.access-key-id / s3.secret-access-key / s3.session-token),
94+
// then the AWS default credential chain.
95+
//
96+
// The Java-client property names (rest.access-key-id / rest.secret-access-key /
97+
// rest.session-token) are accepted as aliases, resolved per field with the s3.*
98+
// keys taking precedence when both are set.
9199
func WithSigV4() Option {
92100
return func(o *options) {
93101
o.enableSigv4 = true

‎catalog/rest/rest.go‎

Lines changed: 73 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,7 @@ import (
4141

4242
"github.com/apache/iceberg-go"
4343
"github.com/apache/iceberg-go/catalog"
44+
internalaws "github.com/apache/iceberg-go/internal/awsconfig"
4445
iceio "github.com/apache/iceberg-go/io"
4546
"github.com/apache/iceberg-go/metrics"
4647
"github.com/apache/iceberg-go/table"
@@ -49,6 +50,7 @@ import (
4950
"github.com/aws/aws-sdk-go-v2/aws"
5051
v4 "github.com/aws/aws-sdk-go-v2/aws/signer/v4"
5152
"github.com/aws/aws-sdk-go-v2/config"
53+
"github.com/aws/aws-sdk-go-v2/credentials"
5254
"golang.org/x/oauth2"
5355
"golang.org/x/oauth2/clientcredentials"
5456
"golang.org/x/sync/semaphore"
@@ -92,6 +94,12 @@ const (
9294
keyRestSigV4Region = "rest.signing-region"
9395
keyRestSigV4Service = "rest.signing-name"
9496
keyAuthUrl = "rest.authorization-url"
97+
// keyRestAccessKeyID and friends are the Java-client property names for the
98+
// SigV4 signing credentials. They are accepted as aliases for the s3.*
99+
// properties; the s3.* keys take precedence when both are set.
100+
keyRestAccessKeyID = "rest.access-key-id"
101+
keyRestSecretAccessKey = "rest.secret-access-key"
102+
keyRestSessionToken = "rest.session-token"
95103
// keyOAuth2ServerURI is the portable, spec-aligned property for the OAuth2
96104
// token endpoint used by Java, PyIceberg and iceberg-rust. It is the
97105
// preferred key; keyAuthUrl is retained as a compatibility alias. When both
@@ -243,6 +251,31 @@ type sessionTransport struct {
243251
cfg aws.Config
244252
service string
245253
newHash func() hash.Hash
254+
// signingOrigin is the configured catalog origin. Requests to a different
255+
// origin (e.g. a redirect hop) are not signed, so the SigV4 Authorization
256+
// header and session token never reach an unconfigured host.
257+
signingOrigin *url.URL
258+
}
259+
260+
// sameOrigin reports whether two URLs share scheme, host, and effective port.
261+
func sameOrigin(a, b *url.URL) bool {
262+
return strings.EqualFold(a.Scheme, b.Scheme) &&
263+
strings.EqualFold(a.Hostname(), b.Hostname()) &&
264+
defaultedPort(a) == defaultedPort(b)
265+
}
266+
267+
func defaultedPort(u *url.URL) string {
268+
if p := u.Port(); p != "" {
269+
return p
270+
}
271+
switch strings.ToLower(u.Scheme) {
272+
case "https":
273+
return "443"
274+
case "http":
275+
return "80"
276+
default:
277+
return ""
278+
}
246279
}
247280

248281
// from https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/aws/signer/v4#Signer.SignHTTP
@@ -292,7 +325,7 @@ func (s *sessionTransport) RoundTrip(r *http.Request) (*http.Response, error) {
292325
r.Header.Set(k, v)
293326
}
294327

295-
if s.signer != nil {
328+
if s.signer != nil && (s.signingOrigin == nil || sameOrigin(s.signingOrigin, r.URL)) {
296329
var payloadHash string
297330
if r.Body == nil {
298331
payloadHash = emptyStringHash
@@ -1107,26 +1140,64 @@ func (r *Catalog) createSession(ctx context.Context, opts *options) (*http.Clien
11071140
if opts.enableSigv4 {
11081141
cfg := opts.awsConfig
11091142
if !opts.awsConfigSet {
1143+
creds, err := staticCredsFromProps(opts.additionalProps)
1144+
if err != nil {
1145+
cleanup()
1146+
1147+
return nil, nil, err
1148+
}
11101149
// If no config provided, load defaults from environment.
1111-
var err error
11121150
cfg, err = config.LoadDefaultConfig(ctx)
11131151
if err != nil {
11141152
cleanup()
11151153

11161154
return nil, nil, err
11171155
}
1156+
// Sign with the S3 credentials carried in the catalog properties when
1157+
// present, rather than only the AWS default credential chain.
1158+
if creds != nil {
1159+
cfg.Credentials = creds
1160+
}
11181161
}
11191162
if opts.sigv4Region != "" {
11201163
cfg.Region = opts.sigv4Region
11211164
}
11221165

11231166
session.cfg, session.service = cfg, opts.sigv4Service
11241167
session.signer, session.newHash = v4.NewSigner(), sha256.New
1168+
session.signingOrigin = r.baseURI
11251169
}
11261170

11271171
return cl, cleanup, nil
11281172
}
11291173

1174+
// staticCredsFromProps returns a static credentials provider built from the
1175+
// signing-credential properties. It prefers the s3.* keys and falls back to the
1176+
// Java-compatible rest.* aliases, resolving the tuple atomically from a single
1177+
// namespace so a partial pair is never completed with fields from the other one.
1178+
// It returns (nil, nil) when neither namespace sets any credential property, so
1179+
// the caller falls back to the default credential chain, and an
1180+
// ErrIncompleteStaticCredentials error when the chosen namespace is incomplete.
1181+
func staticCredsFromProps(props iceberg.Properties) (aws.CredentialsProvider, error) {
1182+
namespaces := [][3]string{
1183+
{iceio.S3AccessKeyID, iceio.S3SecretAccessKey, iceio.S3SessionToken},
1184+
{keyRestAccessKeyID, keyRestSecretAccessKey, keyRestSessionToken},
1185+
}
1186+
for _, ns := range namespaces {
1187+
accessKey, secretKey, token := props[ns[0]], props[ns[1]], props[ns[2]]
1188+
if accessKey == "" && secretKey == "" && token == "" {
1189+
continue
1190+
}
1191+
if err := internalaws.ValidateStaticCredentials(ns[0], ns[1], ns[2], accessKey, secretKey, token); err != nil {
1192+
return nil, err
1193+
}
1194+
1195+
return credentials.NewStaticCredentialsProvider(accessKey, secretKey, token), nil
1196+
}
1197+
1198+
return nil, nil
1199+
}
1200+
11301201
func (r *Catalog) fetchConfig(ctx context.Context, opts *options) (*options, error) {
11311202
params := url.Values{}
11321203
if opts.warehouseLocation != "" {

‎catalog/rest/rest_internal_test.go‎

Lines changed: 155 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,8 @@ import (
4242

4343
"github.com/apache/iceberg-go"
4444
"github.com/apache/iceberg-go/catalog"
45+
internalaws "github.com/apache/iceberg-go/internal/awsconfig"
46+
iceio "github.com/apache/iceberg-go/io"
4547
"github.com/apache/iceberg-go/table"
4648
"github.com/aws/aws-sdk-go-v2/aws"
4749
v4 "github.com/aws/aws-sdk-go-v2/aws/signer/v4"
@@ -52,6 +54,159 @@ import (
5254
"golang.org/x/sync/errgroup"
5355
)
5456

57+
func TestStaticCredsFromProps(t *testing.T) {
58+
creds, err := staticCredsFromProps(iceberg.Properties{
59+
iceio.S3AccessKeyID: "AK",
60+
iceio.S3SecretAccessKey: "SK",
61+
iceio.S3SessionToken: "ST",
62+
})
63+
require.NoError(t, err)
64+
require.NotNil(t, creds)
65+
got, err := creds.Retrieve(context.Background())
66+
require.NoError(t, err)
67+
require.Equal(t, "AK", got.AccessKeyID)
68+
require.Equal(t, "SK", got.SecretAccessKey)
69+
require.Equal(t, "ST", got.SessionToken)
70+
71+
creds, err = staticCredsFromProps(iceberg.Properties{})
72+
require.NoError(t, err, "no creds must fall back to the default chain")
73+
require.Nil(t, creds)
74+
75+
_, err = staticCredsFromProps(iceberg.Properties{iceio.S3AccessKeyID: "AK"})
76+
require.ErrorIs(t, err, internalaws.ErrIncompleteStaticCredentials, "a lone access key must be an error, not the ambient identity")
77+
78+
_, err = staticCredsFromProps(iceberg.Properties{iceio.S3SecretAccessKey: "SK"})
79+
require.ErrorIs(t, err, internalaws.ErrIncompleteStaticCredentials, "a lone secret key must be an error, not the ambient identity")
80+
81+
_, err = staticCredsFromProps(iceberg.Properties{iceio.S3SessionToken: "ST"})
82+
require.ErrorIs(t, err, internalaws.ErrIncompleteStaticCredentials, "a lone session token must be an error, not the ambient identity")
83+
84+
creds, err = staticCredsFromProps(iceberg.Properties{
85+
keyRestAccessKeyID: "RAK",
86+
keyRestSecretAccessKey: "RSK",
87+
keyRestSessionToken: "RST",
88+
})
89+
require.NoError(t, err)
90+
require.NotNil(t, creds)
91+
got, err = creds.Retrieve(context.Background())
92+
require.NoError(t, err)
93+
require.Equal(t, "RAK", got.AccessKeyID)
94+
require.Equal(t, "RSK", got.SecretAccessKey)
95+
require.Equal(t, "RST", got.SessionToken)
96+
97+
creds, err = staticCredsFromProps(iceberg.Properties{
98+
iceio.S3AccessKeyID: "AK",
99+
iceio.S3SecretAccessKey: "SK",
100+
keyRestAccessKeyID: "RAK",
101+
keyRestSecretAccessKey: "RSK",
102+
})
103+
require.NoError(t, err)
104+
got, err = creds.Retrieve(context.Background())
105+
require.NoError(t, err)
106+
require.Equal(t, "AK", got.AccessKeyID, "s3.* keys take precedence over rest.* aliases")
107+
require.Equal(t, "SK", got.SecretAccessKey)
108+
109+
_, err = staticCredsFromProps(iceberg.Properties{keyRestAccessKeyID: "RAK"})
110+
require.ErrorIs(t, err, internalaws.ErrIncompleteStaticCredentials, "a lone rest.* access key must be an error")
111+
112+
_, err = staticCredsFromProps(iceberg.Properties{
113+
iceio.S3AccessKeyID: "AK",
114+
keyRestSecretAccessKey: "RSK",
115+
})
116+
require.ErrorIs(t, err, internalaws.ErrIncompleteStaticCredentials, "a partial pair must not be completed with a field from the other namespace")
117+
118+
creds, err = staticCredsFromProps(iceberg.Properties{
119+
iceio.S3AccessKeyID: "AK",
120+
iceio.S3SecretAccessKey: "SK",
121+
keyRestSessionToken: "RST",
122+
})
123+
require.NoError(t, err)
124+
got, err = creds.Retrieve(context.Background())
125+
require.NoError(t, err)
126+
require.Equal(t, "AK", got.AccessKeyID)
127+
require.Empty(t, got.SessionToken, "a complete s3.* pair must not inherit an unrelated rest.* session token")
128+
}
129+
130+
// TestSigV4SignsWithPropsCredentials pins the wiring: the SigV4 Authorization
131+
// header must be signed with the credentials carried in the catalog properties.
132+
func TestSigV4SignsWithPropsCredentials(t *testing.T) {
133+
var authHeader string
134+
mux := http.NewServeMux()
135+
mux.HandleFunc("/v1/config", func(w http.ResponseWriter, r *http.Request) {
136+
json.NewEncoder(w).Encode(map[string]any{"defaults": map[string]any{}, "overrides": map[string]any{}})
137+
})
138+
mux.HandleFunc("/test", func(w http.ResponseWriter, r *http.Request) {
139+
authHeader = r.Header.Get("Authorization")
140+
w.WriteHeader(http.StatusOK)
141+
})
142+
srv := httptest.NewServer(mux)
143+
defer srv.Close()
144+
145+
cat, err := NewCatalog(context.Background(), "rest", srv.URL,
146+
WithSigV4RegionSvc("us-east-1", "s3"),
147+
WithAdditionalProps(iceberg.Properties{
148+
iceio.S3AccessKeyID: "AKIDEXAMPLEPROPS",
149+
iceio.S3SecretAccessKey: "secretexample",
150+
}))
151+
require.NoError(t, err)
152+
153+
req, err := http.NewRequestWithContext(context.Background(), http.MethodGet, srv.URL+"/test", nil)
154+
require.NoError(t, err)
155+
resp, err := cat.cl.Do(req)
156+
require.NoError(t, err)
157+
require.NoError(t, resp.Body.Close())
158+
159+
require.Contains(t, authHeader, "Credential=AKIDEXAMPLEPROPS/",
160+
"SigV4 must sign with the credentials from catalog properties, not the default chain")
161+
}
162+
163+
// TestSigv4DoesNotSignCrossOriginRedirect pins that a redirect to a different
164+
// origin is not re-signed, so the SigV4 Authorization header and session token
165+
// never reach an unconfigured host.
166+
func TestSigv4DoesNotSignCrossOriginRedirect(t *testing.T) {
167+
var secondHit bool
168+
var gotAuth, gotToken string
169+
second := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
170+
secondHit = true
171+
gotAuth = r.Header.Get("Authorization")
172+
gotToken = r.Header.Get("X-Amz-Security-Token")
173+
w.WriteHeader(http.StatusOK)
174+
}))
175+
defer second.Close()
176+
177+
var firstAuth string
178+
mux := http.NewServeMux()
179+
mux.HandleFunc("/v1/config", func(w http.ResponseWriter, r *http.Request) {
180+
json.NewEncoder(w).Encode(map[string]any{"defaults": map[string]any{}, "overrides": map[string]any{}})
181+
})
182+
mux.HandleFunc("/redirect", func(w http.ResponseWriter, r *http.Request) {
183+
firstAuth = r.Header.Get("Authorization")
184+
http.Redirect(w, r, second.URL+"/landing", http.StatusTemporaryRedirect)
185+
})
186+
first := httptest.NewServer(mux)
187+
defer first.Close()
188+
189+
cat, err := NewCatalog(context.Background(), "rest", first.URL,
190+
WithSigV4RegionSvc("us-east-1", "s3"),
191+
WithAdditionalProps(iceberg.Properties{
192+
iceio.S3AccessKeyID: "AKIDEXAMPLEPROPS",
193+
iceio.S3SecretAccessKey: "secretexample",
194+
iceio.S3SessionToken: "SESSIONTOKENEXAMPLE",
195+
}))
196+
require.NoError(t, err)
197+
198+
req, err := http.NewRequestWithContext(context.Background(), http.MethodGet, first.URL+"/redirect", nil)
199+
require.NoError(t, err)
200+
resp, err := cat.cl.Do(req)
201+
require.NoError(t, err)
202+
require.NoError(t, resp.Body.Close())
203+
204+
require.Contains(t, firstAuth, "Credential=AKIDEXAMPLEPROPS/", "the configured origin must still be signed")
205+
require.True(t, secondHit, "the redirect target must be reached")
206+
require.Empty(t, gotAuth, "the redirect target must not receive the SigV4 Authorization header")
207+
require.Empty(t, gotToken, "the redirect target must not receive the session token")
208+
}
209+
55210
func TestSplitIdentForPathRequiresNamespaceAndName(t *testing.T) {
56211
cat := &Catalog{}
57212

‎website/src/configuration.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -56,7 +56,7 @@ catalog:
5656
| `catalog.<name>.aws-profile` | AWS named profile for the Glue catalog. When unset, the AWS SDK default credential chain is used. |
5757
| `catalog.<name>.sql-driver` | `database/sql` driver name for the SQL catalog. Maps to the `sql.driver` property. The default CLI binary only compiles in `sqliteshim`; other drivers require a custom build. |
5858
| `catalog.<name>.sql-dialect` | SQL dialect for the SQL catalog (`postgres`, `mysql`, `sqlite`, `mssql`, `oracle`). Maps to the `sql.dialect` property. The default CLI binary only ships `sqlite` via `sqliteshim`; other dialects need a custom build with their drivers. |
59-
| `catalog.<name>.rest.sigv4-enabled` | Enable AWS SigV4 signing for REST. |
59+
| `catalog.<name>.rest.sigv4-enabled` | Enable AWS SigV4 signing for REST. When enabled, requests are signed with the `s3.*` credential properties if set (`s3.access-key-id` / `s3.secret-access-key` / `s3.session-token`), otherwise with the AWS default credential chain. The Java-client names (`rest.access-key-id` / `rest.secret-access-key` / `rest.session-token`) are accepted as aliases, resolved per field with the `s3.*` keys taking precedence. |
6060
| `catalog.<name>.rest.signing-name` | SigV4 service name. |
6161
| `catalog.<name>.rest.signing-region` | SigV4 region. |
6262

0 commit comments

Comments
 (0)