Repository navigation
Commit eb15d53
committed
tools: keep a BOM through the fatal decode and report the right bad-byte offset
Two things the fatal decode got wrong and both are mine, from c5556a5 and d5c31a1:
* The BOM. `new TextDecoder("utf-8")` consumes a leading byte-order mark by default, so switching from
`fs.readFile(path, "utf-8")` - which keeps it - to a decoder dropped it, and every path below republishes
the decoded string: the approval baseline, `saveChanges`, `saveDirectly`. Measured on pinned v22.23.4
against a file whose bytes are EF BB BD 68 69 0D 0A 74 68 65 72 65 0D 0A: `readFile` with utf-8 returns
a 12-character string beginning U+FEFF, the decoder with default options returns 11 characters without
it, and `{ fatal: true, ignoreBOM: true }` returns 12 characters whose re-encoding is byte-identical to
the file. The fix is `ignoreBOM: true`, which restores what the tool did before my change rather than
inventing new behaviour.
* The offset. A BOM is three bytes that decode to one character, so counting through the decoded string
reported a bad byte three positions early (measured: offset 3 where the offending byte is at 6), and a
U+FFFD that the file itself contains is indistinguishable in the decoder's output from a replacement the
decoder produced, so a real one earlier in the file was read as the failure position (measured: offset 1
where the offending byte is at 5). The offset is now walked in the original bytes, stepping over a
replacement only when those bytes really are EF BB BD.
Both are checked against byte positions read off a hex dump rather than against anything derived from the
decoded string: BOM plus invalid -> 4, literal U+FFFD plus invalid -> 4, both together -> 6, bad
continuation -> 1, truncated sequence -> 1, invalid restricted continuation (ED A0 80) -> 1, multibyte
then invalid -> 2, and the two rows already in the spec -> 3 and -> 1. Nine cases, all agreeing with the
tool's own output on the pinned interpreter.
Verification, pinned node v22.23.4 with stdio on file descriptors:
- src/core/tools/__tests__/applyDiffTool.failureReporting.spec.ts: 44 passed (1 file), four of them new.
- tsc --noEmit -p tsconfig.json in src: 0 errors.
- eslint --prune-suppressions --max-warnings=0: 0 on both files; its rewrite of
src/eslint-suppressions.json was reverted, so no suppression churn rides along.
- prettier --check clean on both, formatted before measuring.
- The merge candidate (this head onto upstream/main e487cd7, which is what the pull_request CI
actually checks out) is run locally before the push, because main's own
applyDiffTool.outsideWorkspace.spec.ts runs against this file there.
Gate: 355 of 500 counted executable lines at base 73931fe with base...head and
--unified=0 on the formatted tree, against 347 at d5c31a1 - a +8-line move, measured with v22.23.4.
Spec rows cost nothing: the four new rows are in __tests__, which the gate excludes.1 parent d5c31a1 commit eb15d53
2 files changed
Lines changed: 67 additions & 9 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
91 | 91 | | |
92 | 92 | | |
93 | 93 | | |
94 | | - | |
| 94 | + | |
95 | 95 | | |
96 | | - | |
97 | | - | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
98 | 116 | | |
99 | 117 | | |
100 | 118 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
154 | 154 | | |
155 | 155 | | |
156 | 156 | | |
157 | | - | |
| 157 | + | |
158 | 158 | | |
159 | 159 | | |
160 | 160 | | |
| |||
166 | 166 | | |
167 | 167 | | |
168 | 168 | | |
169 | | - | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
170 | 173 | | |
171 | 174 | | |
172 | 175 | | |
173 | 176 | | |
174 | 177 | | |
175 | 178 | | |
176 | 179 | | |
177 | | - | |
| 180 | + | |
178 | 181 | | |
179 | 182 | | |
180 | 183 | | |
| |||
196 | 199 | | |
197 | 200 | | |
198 | 201 | | |
199 | | - | |
| 202 | + | |
200 | 203 | | |
201 | 204 | | |
202 | 205 | | |
| |||
209 | 212 | | |
210 | 213 | | |
211 | 214 | | |
212 | | - | |
| 215 | + | |
213 | 216 | | |
214 | 217 | | |
215 | 218 | | |
216 | 219 | | |
217 | 220 | | |
218 | 221 | | |
219 | 222 | | |
220 | | - | |
| 223 | + | |
221 | 224 | | |
222 | 225 | | |
223 | 226 | | |
224 | 227 | | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
225 | 265 | | |
226 | 266 | | |
227 | 267 | | |
| |||
0 commit comments