Skip to content

Commit 5e3e896

Browse files
morningmanclaude
andcommitted
[P3b-T01] commit 3: unify HadoopAuthenticator onto fe-kerberos + delete fe-filesystem-hdfs copies
Final step of P3b kerberos consolidation. Two competing HadoopAuthenticator interfaces existed: fe-kerberos's (getUGI + doAs(PrivilegedExceptionAction), the single source of truth used by fe-core/HMS) and a second fe-filesystem-spi one (doAs(IOCallable)) with its own KerberosHadoopAuthenticator/ SimpleHadoopAuthenticator copies in fe-filesystem-hdfs. The hdfs copies had drifted from the canonical impls (different login/refresh + a first-writer-wins setConfiguration guard). Solution (user-approved "pure consolidation" — adopt fe-kerberos behavior, restoring parity with the legacy fe-common/HMS HDFS path): - Delete fe-filesystem-spi HadoopAuthenticator + IOCallable (used only by fe-filesystem-hdfs; grep-confirmed no other consumer). - Delete fe-filesystem-hdfs KerberosHadoopAuthenticator + SimpleHadoopAuthenticator. - Retarget the 4 consumers (DFSFileSystem, HdfsInputFile, HdfsOutputFile, HdfsFileIterator) to org.apache.doris.kerberos.HadoopAuthenticator; the doAs(() -> ...) lambdas bind unchanged to doAs(PrivilegedExceptionAction). - DFSFileSystem.buildAuthenticator() seam keeps the kerberos-vs-simple SELECTION decision byte-identical (HdfsConfigBuilder.isKerberosEnabled = principal+keytab present), NOT fe-kerberos's hadoop.security.authentication gate — only the authenticator behavior changes. - Add fe-kerberos dep to fe-filesystem-hdfs/pom.xml (no cycle: fe-kerberos is a zero-FE-dep leaf). Accepted behavior changes (gated by docker kerberos e2e): simple/no-username now runs as remote user "hadoop" via doAs (was: FE process user, direct); kerberos uses fe-kerberos LoginContext+80%-refresh+unconditional setConfiguration. Adversarial review (wf_b1a4e7e4-b51, 3 lenses + verify) confirmed 3 MINOR and all are fixed here: - Empty-string hadoop.username regression: fe-kerberos HadoopSimpleAuthenticator defaults to "hadoop" only on null, so "" reached createRemoteUser("") which throws IllegalArgumentException at FS construction (the removed copy guarded !isEmpty()). buildAuthenticator now normalizes absent/empty uniformly to the "hadoop" default. RED proven (IllegalArgumentException: Null user) -> GREEN. - Restored the dropped empty-string test coverage (DFSAuthenticatorTest). - Scrubbed stale docs that still claimed fe-filesystem-spi contains HadoopAuthenticator (fe-filesystem README + fe-filesystem-spi pom description; README is aggregator-level, a transparent consequence of the spi deletion). Tests: new DFSAuthenticatorTest pins the wiring + accepted no-username->"hadoop" + empty-string->"hadoop"; HdfsFileIteratorTest passthrough retargeted; obsolete SimpleHadoopAuthenticatorTest deleted; KerberosHadoopAuthenticatorEnvTest retargeted (lazy getUGI login). fe-filesystem-hdfs 79/0/0 (+fe-kerberos/spi via -am) BUILD SUCCESS; checkstyle 0; import-gate exit 0; whole-repo grep filesystem.spi.HadoopAuthenticator/IOCallable = 0; reactor test-compile clean (sole failure = pre-existing fe-connector-paimon HiveConf shade quirk, unrelated). docker kerberos e2e (HDFS kerberized + HMS) NOT run — the real gate; UGI login can't be exercised in unit tests. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_011mTrPcvMZtFjsxWJM5TRnG
1 parent 237c08a commit 5e3e896

15 files changed

Lines changed: 162 additions & 352 deletions

File tree

‎fe/fe-filesystem/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -56,7 +56,7 @@ fe-filesystem/ (aggregator POM — no Java code)
5656
dependencies). Defines `FileSystem`, `Location`, `FileEntry`, `DorisInputFile`,
5757
`DorisOutputFile`, etc.
5858
* **SPI** — The `FileSystemProvider` interface (extends `PluginFactory` from `fe-extension-spi`)
59-
plus the object-storage layer (`ObjStorage`, `ObjFileSystem`, `HadoopAuthenticator`). Also
59+
plus the object-storage layer (`ObjStorage`, `ObjFileSystem`). Also
6060
compiled into `fe-core`.
6161
* **IMPL** — Concrete backends. Each one depends on `fe-filesystem-spi` (and transitively on
6262
`fe-filesystem-api`). S3-delegating backends (OSS, COS, OBS) also depend on `fe-filesystem-s3`

‎fe/fe-filesystem/fe-filesystem-hdfs/pom.xml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,13 @@ under the License.
4848
<version>${revision}</version>
4949
</dependency>
5050

51+
<!-- Shared Kerberos/Simple HadoopAuthenticator (single source of truth; P3b consolidation). -->
52+
<dependency>
53+
<groupId>org.apache.doris</groupId>
54+
<artifactId>fe-kerberos</artifactId>
55+
<version>${revision}</version>
56+
</dependency>
57+
5158
<dependency>
5259
<groupId>org.apache.commons</groupId>
5360
<artifactId>commons-lang3</artifactId>

‎fe/fe-filesystem/fe-filesystem-hdfs/src/main/java/org/apache/doris/filesystem/hdfs/DFSFileSystem.java‎

Lines changed: 34 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,12 @@
2323
import org.apache.doris.filesystem.FileIterator;
2424
import org.apache.doris.filesystem.GlobListing;
2525
import org.apache.doris.filesystem.Location;
26-
import org.apache.doris.filesystem.spi.HadoopAuthenticator;
26+
import org.apache.doris.kerberos.AuthenticationConfig;
27+
import org.apache.doris.kerberos.HadoopAuthenticator;
28+
import org.apache.doris.kerberos.HadoopKerberosAuthenticator;
29+
import org.apache.doris.kerberos.HadoopSimpleAuthenticator;
30+
import org.apache.doris.kerberos.KerberosAuthenticationConfig;
31+
import org.apache.doris.kerberos.SimpleAuthenticationConfig;
2732

2833
import org.apache.hadoop.conf.Configuration;
2934
import org.apache.hadoop.fs.FileStatus;
@@ -64,15 +69,38 @@ public class DFSFileSystem implements org.apache.doris.filesystem.FileSystem {
6469
public DFSFileSystem(Map<String, String> properties) {
6570
this.properties = Collections.unmodifiableMap(new HashMap<>(properties));
6671
this.conf = HdfsConfigBuilder.build(properties);
72+
this.authenticator = buildAuthenticator(properties, conf);
73+
}
74+
75+
/**
76+
* Builds the {@link HadoopAuthenticator} for the given HDFS catalog properties using the
77+
* shared {@code fe-kerberos} authenticators (single source of truth; P3b consolidation —
78+
* the former fe-filesystem-hdfs copies were removed). The kerberos-vs-simple decision is
79+
* unchanged: principal + keytab present selects {@link HadoopKerberosAuthenticator},
80+
* otherwise {@link HadoopSimpleAuthenticator}. Kerberos login happens lazily on first
81+
* {@code getUGI()}; the simple path executes as {@code hadoop.username} (defaulting to
82+
* remote user {@code "hadoop"}).
83+
*/
84+
static HadoopAuthenticator buildAuthenticator(Map<String, String> properties, Configuration conf) {
6785
if (HdfsConfigBuilder.isKerberosEnabled(properties)) {
68-
this.authenticator = new KerberosHadoopAuthenticator(
86+
boolean debug = Boolean.parseBoolean(
87+
properties.getOrDefault(AuthenticationConfig.DORIS_KRB5_DEBUG, "false"));
88+
return new HadoopKerberosAuthenticator(new KerberosAuthenticationConfig(
6989
properties.get(HdfsConfigBuilder.KEY_PRINCIPAL),
7090
properties.get(HdfsConfigBuilder.KEY_KEYTAB),
71-
conf);
72-
} else {
73-
this.authenticator = new SimpleHadoopAuthenticator(
74-
properties.get("hadoop.username"));
91+
conf,
92+
debug));
93+
}
94+
SimpleAuthenticationConfig simpleConfig = new SimpleAuthenticationConfig();
95+
// Treat absent OR empty hadoop.username uniformly: leave it null so the shared
96+
// authenticator defaults to remote user "hadoop". Passing "" straight through would
97+
// reach UserGroupInformation.createRemoteUser(""), which rejects an empty user with
98+
// IllegalArgumentException (the removed hdfs copy guarded this with !isEmpty()).
99+
String hadoopUsername = properties.get("hadoop.username");
100+
if (hadoopUsername != null && !hadoopUsername.isEmpty()) {
101+
simpleConfig.setUsername(hadoopUsername);
75102
}
103+
return new HadoopSimpleAuthenticator(simpleConfig);
76104
}
77105

78106
private org.apache.hadoop.fs.FileSystem getHadoopFs(Path path) throws IOException {

‎fe/fe-filesystem/fe-filesystem-hdfs/src/main/java/org/apache/doris/filesystem/hdfs/HdfsFileIterator.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@
2020
import org.apache.doris.filesystem.FileEntry;
2121
import org.apache.doris.filesystem.FileIterator;
2222
import org.apache.doris.filesystem.Location;
23-
import org.apache.doris.filesystem.spi.HadoopAuthenticator;
23+
import org.apache.doris.kerberos.HadoopAuthenticator;
2424

2525
import org.apache.hadoop.fs.FileStatus;
2626
import org.apache.hadoop.fs.RemoteIterator;

‎fe/fe-filesystem/fe-filesystem-hdfs/src/main/java/org/apache/doris/filesystem/hdfs/HdfsInputFile.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@
2020
import org.apache.doris.filesystem.DorisInputFile;
2121
import org.apache.doris.filesystem.DorisInputStream;
2222
import org.apache.doris.filesystem.Location;
23-
import org.apache.doris.filesystem.spi.HadoopAuthenticator;
23+
import org.apache.doris.kerberos.HadoopAuthenticator;
2424

2525
import org.apache.hadoop.fs.FSDataInputStream;
2626
import org.apache.hadoop.fs.Path;

‎fe/fe-filesystem/fe-filesystem-hdfs/src/main/java/org/apache/doris/filesystem/hdfs/HdfsOutputFile.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@
1919

2020
import org.apache.doris.filesystem.DorisOutputFile;
2121
import org.apache.doris.filesystem.Location;
22-
import org.apache.doris.filesystem.spi.HadoopAuthenticator;
22+
import org.apache.doris.kerberos.HadoopAuthenticator;
2323

2424
import org.apache.hadoop.fs.Path;
2525

‎fe/fe-filesystem/fe-filesystem-hdfs/src/main/java/org/apache/doris/filesystem/hdfs/KerberosHadoopAuthenticator.java‎

Lines changed: 0 additions & 117 deletions
This file was deleted.

‎fe/fe-filesystem/fe-filesystem-hdfs/src/main/java/org/apache/doris/filesystem/hdfs/SimpleHadoopAuthenticator.java‎

Lines changed: 0 additions & 63 deletions
This file was deleted.
Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
package org.apache.doris.filesystem.hdfs;
19+
20+
import org.apache.doris.kerberos.HadoopAuthenticator;
21+
import org.apache.doris.kerberos.HadoopKerberosAuthenticator;
22+
import org.apache.doris.kerberos.HadoopSimpleAuthenticator;
23+
24+
import org.apache.hadoop.conf.Configuration;
25+
import org.junit.jupiter.api.Assertions;
26+
import org.junit.jupiter.api.Test;
27+
28+
import java.io.IOException;
29+
import java.util.HashMap;
30+
import java.util.Map;
31+
32+
/**
33+
* Verifies that {@link DFSFileSystem} wires the shared {@code fe-kerberos} authenticators
34+
* (P3b consolidation): the duplicate fe-filesystem-hdfs {@code KerberosHadoopAuthenticator}/
35+
* {@code SimpleHadoopAuthenticator} copies are gone.
36+
*
37+
* <p>It also pins the deliberately-adopted behavior change: the simple/no-username path now
38+
* executes as remote user {@code "hadoop"} (legacy fe-common/HMS parity) instead of the FE
39+
* process user. The duplicate copy used to run such actions directly as the FE process user.
40+
*/
41+
class DFSAuthenticatorTest {
42+
43+
@Test
44+
void simpleNoUsernameRunsAsHadoop() throws IOException {
45+
HadoopAuthenticator auth = DFSFileSystem.buildAuthenticator(new HashMap<>(), new Configuration());
46+
Assertions.assertTrue(auth instanceof HadoopSimpleAuthenticator,
47+
"non-kerberos properties must select the shared simple authenticator");
48+
Assertions.assertEquals("hadoop", auth.getUGI().getUserName(),
49+
"no hadoop.username must default to remote user 'hadoop' (fe-common/HMS parity)");
50+
}
51+
52+
@Test
53+
void simpleWithUsernameRunsAsThatUser() throws IOException {
54+
Map<String, String> props = new HashMap<>();
55+
props.put("hadoop.username", "testuser");
56+
HadoopAuthenticator auth = DFSFileSystem.buildAuthenticator(props, new Configuration());
57+
Assertions.assertTrue(auth instanceof HadoopSimpleAuthenticator);
58+
Assertions.assertEquals("testuser", auth.getUGI().getUserName(),
59+
"hadoop.username must drive the simple authenticator's UGI identity");
60+
}
61+
62+
@Test
63+
void simpleEmptyUsernameRunsAsHadoop() throws IOException {
64+
Map<String, String> props = new HashMap<>();
65+
props.put("hadoop.username", "");
66+
HadoopAuthenticator auth = DFSFileSystem.buildAuthenticator(props, new Configuration());
67+
Assertions.assertTrue(auth instanceof HadoopSimpleAuthenticator);
68+
// Empty must be treated like absent (-> remote user "hadoop"), not passed to
69+
// UserGroupInformation.createRemoteUser("") which would throw IllegalArgumentException.
70+
Assertions.assertEquals("hadoop", auth.getUGI().getUserName(),
71+
"empty hadoop.username must default to remote user 'hadoop', not crash construction");
72+
}
73+
74+
@Test
75+
void principalAndKeytabSelectKerberos() {
76+
Map<String, String> props = new HashMap<>();
77+
props.put("hadoop.kerberos.principal", "doris/[email protected]");
78+
props.put("hadoop.kerberos.keytab", "/etc/doris.keytab");
79+
HadoopAuthenticator auth = DFSFileSystem.buildAuthenticator(props, new Configuration());
80+
// Type only: getUGI() would trigger a real KDC login (covered by docker kerberos e2e).
81+
Assertions.assertTrue(auth instanceof HadoopKerberosAuthenticator,
82+
"principal + keytab presence must select the shared kerberos authenticator");
83+
}
84+
}

0 commit comments

Comments
 (0)