Skip to content

Commit c97b60e

Browse files
schenksjclaude
andcommitted
fix: use Hadoop Path when parsing input file URIs in native Delta scan
CometScanRule.nativeDeltaScan validates filesystem schemes by constructing `new java.net.URI(f)` over raw `inputFiles` strings. Any path containing characters invalid in a raw URI (unescaped `%`, spaces, etc.) threw URISyntaxException during plan rewrite, silently degrading Comet's native Delta scan. Surfaced by running Delta's own test suite with Comet enabled: Delta injects `test%file%prefix-` into test filenames, but the same class of failure would hit real users with `%` or spaces in their S3 object keys. Use `new org.apache.hadoop.fs.Path(f).toUri` instead — Path handles URI escaping correctly. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
1 parent 64e1f3e commit c97b60e

1 file changed

Lines changed: 6 additions & 1 deletion

File tree

‎spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@ import scala.collection.mutable.ListBuffer
2626
import scala.jdk.CollectionConverters._
2727

2828
import org.apache.hadoop.conf.Configuration
29+
import org.apache.hadoop.fs.Path
2930
import org.apache.spark.internal.Logging
3031
import org.apache.spark.sql.SparkSession
3132
import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeReference, DynamicPruningExpression, EqualTo, Expression, GenericInternalRow, InputFileBlockLength, InputFileBlockStart, InputFileName, Literal, PlanExpression}
@@ -402,8 +403,12 @@ case class CometScanRule(session: SparkSession)
402403
Set("file", "s3", "s3a", "gs", "gcs", "abfss", "abfs", "wasbs", "wasb", "oss")
403404
val inputFiles = scanExec.relation.location.inputFiles
404405
if (inputFiles.nonEmpty) {
406+
// Use Hadoop Path to safely handle paths containing characters that are
407+
// invalid in a raw URI (e.g. unescaped '%' in Delta's test-only filename
408+
// prefixes). Parsing them with `new URI(f)` directly throws
409+
// URISyntaxException on such paths.
405410
val schemes = inputFiles
406-
.map(f => new java.net.URI(f).getScheme)
411+
.map(f => new Path(f).toUri.getScheme)
407412
.filter(_ != null)
408413
.toSet
409414
val unsupported = schemes -- supportedSchemes

0 commit comments

Comments
 (0)