Skip to content

Commit 39e02de

Browse files
committed
fix: also decline Unicode HDFS write destinations
The raw/decoded URI comparison only catches what java.net.URI had to escape, and java.net.URI leaves non-ASCII path characters alone, so an hdfs://ns/cafe<U+0301>/output destination was admitted. percent_encoding's should_percent_encode is !byte.is_ascii() || set.contains(byte), so the native parser escapes every non-ASCII byte regardless of the encode set and the writer creates caf%C3%A9 outside Spark's staging directory. The guard now also declines any character the native parser rewrites. The ASCII half of that set was determined against the locked url 2.5 crate by parsing hdfs://ns/pre<c>post/output for every printable ASCII c: space, ", #, <, >, ?, backtick, { and } are rewritten and the rest survive, so partition directories and Spark's _temporary attempt paths still qualify. The comment no longer claims the Java comparison detects the divergence on its own; both conditions are kept because the Java one still catches a literal % that the native parser leaves alone. Tests add accented (precomposed and combining), CJK, emoji and nested non-ASCII cases plus the remaining escaped ASCII characters, built from code points since scalastyle forbids non-ASCII source. Disabling the new condition makes the accented case fail, so the Java comparison alone demonstrably does not cover it.
1 parent 4095ea6 commit 39e02de

2 files changed

Lines changed: 83 additions & 16 deletions

File tree

‎spark/src/main/scala/org/apache/comet/serde/operator/NativeWriteUtils.scala‎

Lines changed: 47 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -54,30 +54,65 @@ object NativeWriteUtils {
5454
}
5555

5656
/**
57-
* A fallback reason when `outputPath` is an HDFS destination whose path needs percent-escaping,
58-
* or `None` when the write can proceed.
57+
* ASCII characters the native URL parser percent-encodes inside a path. Determined against the
58+
* locked `url` 2.5 crate by parsing `hdfs://ns/pre<c>post/output` for every printable ASCII `c`
59+
* and comparing `url.path()` with the input: these nine are rewritten and the rest survive,
60+
* including `%`, `[`, `\`, `]`, `^` and `|`. Control characters and DEL are handled separately
61+
* in [[needsNativeUrlEscaping]] rather than listed here.
62+
*/
63+
private val nativeUrlEscapedAscii: Set[Char] =
64+
Set(' ', '"', '#', '<', '>', '?', '`', '{', '}')
65+
66+
/**
67+
* Whether the native URL parser would rewrite `path`, so that the name it creates on HDFS
68+
* differs from the Hadoop filename Spark commits.
69+
*
70+
* `percent_encoding`'s `should_percent_encode` is `!byte.is_ascii() || set.contains(byte)`, so
71+
* every non-ASCII byte is escaped regardless of the encode set. That is the case Java's URI
72+
* comparison cannot see, because `java.net.URI` leaves non-ASCII path characters alone: a name
73+
* holding U+00E9 comes back identically from `getRawPath` and `getPath`, while the native
74+
* parser produces `caf%C3%A9`.
75+
*/
76+
private def needsNativeUrlEscaping(path: String): Boolean =
77+
path.exists(c => c < ' ' || c > '~' || nativeUrlEscapedAscii.contains(c))
78+
79+
/**
80+
* A fallback reason when `outputPath` is an HDFS destination whose path the native writer and
81+
* Spark would spell differently, or `None` when the write can proceed.
82+
*
83+
* Comet and Spark disagree about what such a path names. The native side receives
84+
* `Path.toString`, which is not a URI string, and reaches HDFS through
85+
* `create_hdfs_object_store`, which hands `url.path()` -- now escaped by the Rust parser -- to
86+
* `object_store::path::Path::parse`. So the native writer creates a directory literally called
87+
* `dir%20with%20space`, or `caf%C3%A9` for a name holding U+00E9. Spark's committer, meanwhile,
88+
* works with the unescaped Hadoop `Path` and commits `dir with space`, or that same U+00E9 name
89+
* unescaped. Job commit then succeeds while the data sits somewhere else, which is worse than
90+
* not accelerating the write.
91+
*
92+
* Two conditions, because neither covers the other:
5993
*
60-
* Comet and Spark disagree about what such a path names. The native side reaches HDFS through
61-
* `create_hdfs_object_store`, which hands `url.path()` -- still escaped -- to
62-
* `object_store::path::Path::parse`, so the native writer creates a directory literally called
63-
* `dir%20with%20space`. Spark's committer, meanwhile, works with the unescaped Hadoop `Path`
64-
* and commits `dir with space`. Job commit then succeeds while the data sits somewhere else,
65-
* which is worse than not accelerating the write.
94+
* - the native parser escapes a character, which is the direct statement of the divergence
95+
* and the only condition that catches non-ASCII names;
96+
* - `java.net.URI` had to escape something, which catches a literal `%` in the Hadoop name.
97+
* The native parser leaves `%` alone, so `50%off` reaches `Path::parse` as an invalid
98+
* escape rather than as a rewritten name.
6699
*
67100
* Local `file:` destinations are unaffected and deliberately not gated here: they go through a
68101
* different object-store constructor that does not retain the escaping.
69102
*/
70103
def escapedHdfsDestination(outputPath: String): Option[String] = {
71104
if (!outputPath.startsWith("hdfs:")) return None
72105
val uri = new Path(outputPath).toUri
73-
// `getPath` decodes, `getRawPath` does not. They differ exactly when the path contains
74-
// something the URI form had to escape.
75106
val raw = uri.getRawPath
76107
val decoded = uri.getPath
77-
if (raw != null && decoded != null && raw != decoded) {
108+
val javaEscaped = raw != null && decoded != null && raw != decoded
109+
// Checked against the string handed to the native writer, which is `outputPath` itself.
110+
val nativeEscaped = needsNativeUrlEscaping(outputPath)
111+
if (javaEscaped || nativeEscaped) {
112+
val shown = if (decoded != null) decoded else outputPath
78113
Some(
79114
"HDFS output paths needing URI escaping are not supported: the native writer would " +
80-
s"write to the escaped path while Spark commits the unescaped one ($decoded)")
115+
s"write to the escaped path while Spark commits the unescaped one ($shown)")
81116
} else {
82117
None
83118
}

‎spark/src/test/scala/org/apache/comet/parquet/CometParquetWriterSuite.scala‎

Lines changed: 36 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -889,24 +889,56 @@ class CometParquetWriterSuite extends CometTestBase {
889889

890890
test("HDFS output paths needing URI escaping are declined at planning") {
891891
// The local case above writes natively, but HDFS cannot: `create_hdfs_object_store` hands the
892-
// still-escaped `url.path()` to `object_store::path::Path::parse`, so the native writer would
892+
// now-escaped `url.path()` to `object_store::path::Path::parse`, so the native writer would
893893
// create `dir%20with%20space` while Spark's committer commits `dir with space`. Job commit
894894
// would succeed with the data somewhere else, so the write has to stay on Spark until the
895895
// native path handling preserves Hadoop filenames.
896+
//
897+
// Non-ASCII names are the case a `getRawPath != getPath` comparison alone cannot see:
898+
// `java.net.URI` leaves non-ASCII path characters alone, so both accessors agree, while
899+
// `percent_encoding` escapes every non-ASCII byte regardless of the encode set and the native
900+
// writer creates `caf%C3%A9`. Built from code points because scalastyle forbids non-ASCII
901+
// source characters.
902+
def cp(codePoints: Int*): String = codePoints.map(Character.toChars(_).mkString).mkString
903+
val eAcute = cp(0x00e9) // precomposed LATIN SMALL LETTER E WITH ACUTE
904+
val combiningAcute = cp(0x0301) // COMBINING ACUTE ACCENT, applied to a plain "e"
905+
val cjk = cp(0x65e5, 0x672c, 0x8a9e) // "nihongo"
906+
val emoji = cp(0x1f642) // astral plane, so a surrogate pair on the JVM
907+
val uUmlaut = cp(0x00fc)
908+
896909
Seq(
897910
"hdfs://ns/dir with space/output.parquet",
898911
"hdfs://ns/dir%with%percent/output.parquet",
899-
"hdfs://ns/nested/dir with space/output.parquet").foreach { path =>
912+
"hdfs://ns/nested/dir with space/output.parquet",
913+
s"hdfs://ns/caf$eAcute/output.parquet",
914+
s"hdfs://ns/cafe$combiningAcute/output.parquet",
915+
s"hdfs://ns/$cjk/output.parquet",
916+
s"hdfs://ns/$emoji/output.parquet",
917+
// Non-ASCII in a nested segment rather than the leaf.
918+
s"hdfs://ns/${uUmlaut}ber/nested/output.parquet",
919+
// The remaining ASCII characters the native parser escapes.
920+
"hdfs://ns/quote\"here/output.parquet",
921+
"hdfs://ns/hash#here/output.parquet",
922+
"hdfs://ns/angle<here>/output.parquet",
923+
"hdfs://ns/question?here/output.parquet",
924+
"hdfs://ns/back`tick/output.parquet",
925+
"hdfs://ns/brace{here}/output.parquet").foreach { path =>
900926
assert(
901927
NativeWriteUtils.escapedHdfsDestination(path).isDefined,
902928
s"expected $path to be declined")
903929
}
904930

905-
// Unescaped HDFS paths, and local paths of any shape, are unaffected.
931+
// Ordinary HDFS paths, and local paths of any shape, are unaffected. Keeping these passing is
932+
// the point of gating on the exact set the native parser rewrites rather than on "not plain
933+
// ASCII alphanumerics", which would decline the partition directories Spark actually writes.
906934
Seq(
907935
"hdfs://ns/plain/output.parquet",
936+
"hdfs://ns/part-00000-a1b2.c3d4-c000.snappy.parquet",
937+
"hdfs://ns/dt=2026-09-09/hour=17/output.parquet",
938+
"hdfs://ns/_temporary/0/_temporary/attempt_202609091700_0001_m_000000_0/part-0.parquet",
908939
"file:///tmp/dir with space/output.parquet",
909-
"file:///tmp/dir%with%percent/output.parquet").foreach { path =>
940+
"file:///tmp/dir%with%percent/output.parquet",
941+
s"file:///tmp/caf$eAcute/output.parquet").foreach { path =>
910942
assert(
911943
NativeWriteUtils.escapedHdfsDestination(path).isEmpty,
912944
s"expected $path to be accepted")

0 commit comments

Comments
 (0)