Repository navigation
[Tracking] Split #67470: route Azure data access through native Azure credentials #68103
Description
Search before asking
- I had searched in the issues and found no similar issues.
Description
#67470 ([fix](iceberg) Route Azure data files through native credentials) moves Azure Iceberg data access off the Hadoop ABFS compatibility path:
- FE preserves Azure account/container/object URIs instead of rewriting them to
s3://container/path, and emits provider-owned native Azure credentials (AZURE_AUTH_TYPE,AZURE_ENDPOINT,AZURE_ACCOUNT_NAME,AZURE_SAS_TOKEN, ...) to BE. - BE routes
FILE_S3ranges withprovider=azurethrough the native Azure Blob client (SharedKey, SAS with expiry validation, OAuth2 client secret), with a separate bounded client cache and SAS-redacted diagnostics. - Iceberg REST vended SAS credentials replace the static authentication group per request, for both the native data path (BE) and the Iceberg FileIO metadata path (FE,
ADLSFileIO). - Serialized metadata-table tasks carry a fixed credential expiry so the BE JNI scanner fails before opening an expired FileIO.
That PR touches 135 files (+13179 / -946) across BE, the FE filesystem SPI, fe-core, the Iceberg connector and the BE Java metadata scanner, which is too large to review as one change. It is being split into a series of independently reviewable PRs. This issue tracks the series; every PR in the series references this issue.
Split plan and merge order
Rules for the series:
- Consumers land before producers. BE accepts both the legacy
AWS_*SharedKey map and the newAZURE_*map before FE starts emitting the new one. - All SPI additions land in one PR with default implementations, so the plugin API version is bumped once (filesystem
1.0 -> 2.0, connector9.0 -> 10.0); later PRs do not touch the frozen surface. - Each PR compiles and passes its own unit tests on its own; no PR leaves half-wired behavior on master.
| # | PR | Scope | Depends on |
|---|---|---|---|
| BE-1 | [feature](be) Native Azure credentials, SAS expiry and client lifetime in S3ClientFactory |
AzureCredentialOptions (SharedKey/SAS/OAuth2), AzureAuthFactory validation, S3ClientFactory native AZURE_* protocol + legacy AWS_* compatibility, separate expiring Azure client cache, S3URI for abfs[s]:// / wasb[s]:// / Azure HTTP(S), SAS redaction in logs |
- |
| BE-2 | [feature](be) Validate Azure credentials at reader/writer open and keep Azure display paths |
access-time SAS validation in S3FileReader / S3FileSystem, Azure URI container/account checks in get_key, original URI in diagnostics |
BE-1 |
| FE-1 | [feature](fe) Storage/connector SPI: access-time validation, backend view selection and request-local storage access |
StorageProperties / FileSystemProperties / FileSystemProvider / ConnectorStorageContext / ConnectorScanRange default methods, ConnectorStorageAccess, ConnectorStorageAccessResolver, API version bump |
- |
| FE-2 | [fix](fe) Record the synthetic default HDFS binding at creation instead of re-guessing it |
bindDefault / isSyntheticDefault, StorageAdapter.isExplicitlyConfigured, Iceberg selectEffectiveStorages drops the synthetic HDFS binding for object-storage-only catalogs |
FE-1 |
| FE-3 | [fix](fe) Preserve Azure account authority in AzureUri and fix Azure listing pagination |
AzureAccountHost, AzureUri literal ABFS object names + original authority, AzureFileSystem pagination fixes |
- |
| FE-4 | [feature](fe) Emit native Azure credentials (SharedKey/SAS/OAuth2) and route Azure locations by provider |
AzureAuthType, AzureSasToken, AzureFileSystemProperties native AZURE_* backend map + NATIVE kind + URI preservation, fe-core StorageAdapter / LocationPath provider routing, StorageUriUtils Azure normalization removed, ObjectInfoAdapter SAS, S3SourceOffsetProvider object keys |
FE-1, FE-3 (end-to-end: BE-1, BE-2) |
| FE-5 | [feature](fe) Request-local storage bindings: vended credentials replace the static authentication group |
AzureVendedSas, AzureFileIOSharedKey, bindVended, Iceberg FileIO views in the Azure provider; FileSystemPluginManager.bindVended/bindAllExcept, StorageAdapter.ofVended/ofAllExcept, DefaultConnectorContext.newStorageAccessResolver/resolveStorageProperties, PluginDrivenSplit backend file type |
FE-1, FE-2, FE-4 |
| ICE-1 | [feature](iceberg) Select provider-owned Azure FileIO credentials in the REST catalog load path |
IcebergRestFileIOClient / IcebergRestFileIOProperties, IcebergPostSuccessFileIOInitializationException, iceberg-azure dependency (JDK HTTP transport) |
FE-1, FE-5 |
| ICE-2 | [feature](iceberg) Route Iceberg scan and write data access through request-local storage bindings |
IcebergScanPlanProvider ReadStorageAccess, IcebergWritePlanProvider single resolved access per sink, IcebergStatementScope.readStorageAccess, IcebergScanRange.backendFileType |
FE-5, ICE-1 |
| ICE-3 | [feature](iceberg) Carry a fixed FileIO credential expiry with serialized metadata tasks |
TIcebergFileDesc.file_io_expiry_ms (optional), both BE JNI readers, IcebergSysTableJniScanner expiry check, scanner plugin Azure dependencies, IcebergMetadataTaskProperties |
ICE-1 (ICE-2 first to avoid conflicts) |
| REG | [test](regression) Native Azure Iceberg acceptance suite |
opt-in external_table_p2 suite against a pre-provisioned Azure REST catalog |
all |
Suggested serial order: BE-1, BE-2, FE-1, FE-3, FE-2, FE-4, FE-5, ICE-1, ICE-2, ICE-3, REG. BE-1, FE-1 and FE-3 are independent and can be opened in parallel.
Progress
- BE-1 — [feature](be) Native Azure credentials, SAS expiry and client lifetime in S3ClientFactory #68117
- BE-2
- FE-1
- FE-2
- FE-3
- FE-4
- FE-5
- ICE-1
- ICE-2
- ICE-3
- REG
- Close [fix](iceberg) Route Azure data files through native credentials #67470 once the series is merged
Compatibility and rolling upgrade
- SharedKey: a new FE emits both the
AZURE_*fields and the matching legacyAWS_*group, so an old BE keeps working during a rolling upgrade; a new BE verifies that the two groups agree and rejects conflicting values. - SAS and OAuth2 have no legacy wire form. Upgrade all BEs before enabling native SAS/OAuth2 data access from a new FE. To roll back, stop native SAS/OAuth2 use first, then roll back the FE/BE pair.
- Genuine Fabric OneLake locations keep their explicit Hadoop (
FILE_HDFS) path.
Carved out of the series
- The
$filesmetadata-table projection fix (keepfile_sizewhensplit_offsetsis requested) that was bundled into [fix](iceberg) Route Azure data files through native credentials #67470 is unrelated to Azure and will be submitted as a standalone fix. - The two design notes under
docs/in [fix](iceberg) Route Azure data files through native credentials #67470 are not merged; the acceptance fixture and rolling-upgrade notes move into the PR descriptions.
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
[feature](be) Native Azure credentials, SAS expiry and client lifetim…
[fix](be) Keep legacy Azure endpoints literal in S3ClientFactory
[fix](be) Guard azure_transport_endpoint() with USE_AZURE
[fix](be) Keep ADLS names literal and canonicalize Azure hosts in val…
Doris probe: 4.1.4.1 as FE + BE containers, full BE crash log