Repository navigation
Commit 39e02de
committed
fix: also decline Unicode HDFS write destinations
The raw/decoded URI comparison only catches what java.net.URI had to
escape, and java.net.URI leaves non-ASCII path characters alone, so an
hdfs://ns/cafe<U+0301>/output destination was admitted. percent_encoding's
should_percent_encode is !byte.is_ascii() || set.contains(byte), so the
native parser escapes every non-ASCII byte regardless of the encode set
and the writer creates caf%C3%A9 outside Spark's staging directory.
The guard now also declines any character the native parser rewrites. The
ASCII half of that set was determined against the locked url 2.5 crate by
parsing hdfs://ns/pre<c>post/output for every printable ASCII c: space, ",
#, <, >, ?, backtick, { and } are rewritten and the rest survive, so
partition directories and Spark's _temporary attempt paths still qualify.
The comment no longer claims the Java comparison detects the divergence on
its own; both conditions are kept because the Java one still catches a
literal % that the native parser leaves alone.
Tests add accented (precomposed and combining), CJK, emoji and nested
non-ASCII cases plus the remaining escaped ASCII characters, built from
code points since scalastyle forbids non-ASCII source. Disabling the new
condition makes the accented case fail, so the Java comparison alone
demonstrably does not cover it.1 parent 4095ea6 commit 39e02de
2 files changed
Lines changed: 83 additions & 16 deletions
File tree
- spark/src
- main/scala/org/apache/comet/serde/operator
- test/scala/org/apache/comet/parquet
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
54 | 54 | | |
55 | 55 | | |
56 | 56 | | |
57 | | - | |
58 | | - | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
59 | 93 | | |
60 | | - | |
61 | | - | |
62 | | - | |
63 | | - | |
64 | | - | |
65 | | - | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
66 | 99 | | |
67 | 100 | | |
68 | 101 | | |
69 | 102 | | |
70 | 103 | | |
71 | 104 | | |
72 | 105 | | |
73 | | - | |
74 | | - | |
75 | 106 | | |
76 | 107 | | |
77 | | - | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
78 | 113 | | |
79 | 114 | | |
80 | | - | |
| 115 | + | |
81 | 116 | | |
82 | 117 | | |
83 | 118 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
889 | 889 | | |
890 | 890 | | |
891 | 891 | | |
892 | | - | |
| 892 | + | |
893 | 893 | | |
894 | 894 | | |
895 | 895 | | |
| 896 | + | |
| 897 | + | |
| 898 | + | |
| 899 | + | |
| 900 | + | |
| 901 | + | |
| 902 | + | |
| 903 | + | |
| 904 | + | |
| 905 | + | |
| 906 | + | |
| 907 | + | |
| 908 | + | |
896 | 909 | | |
897 | 910 | | |
898 | 911 | | |
899 | | - | |
| 912 | + | |
| 913 | + | |
| 914 | + | |
| 915 | + | |
| 916 | + | |
| 917 | + | |
| 918 | + | |
| 919 | + | |
| 920 | + | |
| 921 | + | |
| 922 | + | |
| 923 | + | |
| 924 | + | |
| 925 | + | |
900 | 926 | | |
901 | 927 | | |
902 | 928 | | |
903 | 929 | | |
904 | 930 | | |
905 | | - | |
| 931 | + | |
| 932 | + | |
| 933 | + | |
906 | 934 | | |
907 | 935 | | |
| 936 | + | |
| 937 | + | |
| 938 | + | |
908 | 939 | | |
909 | | - | |
| 940 | + | |
| 941 | + | |
910 | 942 | | |
911 | 943 | | |
912 | 944 | | |
| |||
0 commit comments