Repository navigation
Commit 48cd014
committed
fix(tools): normalise CR and CRLF line endings before parsing
`remove_blank_lines` only stripped trailing \r/\n bytes, leaving
internal \r\n or lone \r sequences intact. tree-sitter 0.26.8 only
advances its row counter on \n, so CR-only files collapsed all content
onto row 0, producing incorrect sloc/cloc/blank metrics.
Rewrite as `normalize_line_endings`: an in-place write-pointer pass
that converts every \r\n → \n and every lone \r → \n before trimming
trailing newlines. Also normalise CRLF in `check_func_space` (test
helper) so test strings with \r\n behave identically to real files.
Adds four unit tests for normalize_line_endings and three LOC regression
canaries (python_cloc_crlf_matches_lf, python_blank_crlf_matches_lf,
rust_cloc_crlf_matches_lf) that will catch any future tree-sitter
row-counting regression with \r bytes.
Fixes the CR-only case of mozilla/rust-code-analysis#994.1 parent 680dcc5 commit 48cd014
2 files changed
Lines changed: 155 additions & 14 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4927 | 4927 | | |
4928 | 4928 | | |
4929 | 4929 | | |
| 4930 | + | |
| 4931 | + | |
| 4932 | + | |
| 4933 | + | |
| 4934 | + | |
| 4935 | + | |
| 4936 | + | |
| 4937 | + | |
| 4938 | + | |
| 4939 | + | |
| 4940 | + | |
| 4941 | + | |
| 4942 | + | |
| 4943 | + | |
| 4944 | + | |
| 4945 | + | |
| 4946 | + | |
| 4947 | + | |
| 4948 | + | |
| 4949 | + | |
| 4950 | + | |
| 4951 | + | |
| 4952 | + | |
| 4953 | + | |
| 4954 | + | |
| 4955 | + | |
| 4956 | + | |
| 4957 | + | |
| 4958 | + | |
| 4959 | + | |
| 4960 | + | |
| 4961 | + | |
| 4962 | + | |
| 4963 | + | |
| 4964 | + | |
| 4965 | + | |
| 4966 | + | |
| 4967 | + | |
| 4968 | + | |
| 4969 | + | |
| 4970 | + | |
| 4971 | + | |
| 4972 | + | |
| 4973 | + | |
| 4974 | + | |
| 4975 | + | |
| 4976 | + | |
| 4977 | + | |
| 4978 | + | |
| 4979 | + | |
| 4980 | + | |
| 4981 | + | |
| 4982 | + | |
| 4983 | + | |
| 4984 | + | |
| 4985 | + | |
| 4986 | + | |
| 4987 | + | |
| 4988 | + | |
| 4989 | + | |
| 4990 | + | |
| 4991 | + | |
| 4992 | + | |
| 4993 | + | |
| 4994 | + | |
| 4995 | + | |
| 4996 | + | |
| 4997 | + | |
| 4998 | + | |
| 4999 | + | |
| 5000 | + | |
| 5001 | + | |
4930 | 5002 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
11 | 11 | | |
12 | 12 | | |
13 | 13 | | |
14 | | - | |
| 14 | + | |
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
| |||
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
31 | | - | |
| 31 | + | |
32 | 32 | | |
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
36 | | - | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
37 | 39 | | |
38 | 40 | | |
39 | 41 | | |
| |||
82 | 84 | | |
83 | 85 | | |
84 | 86 | | |
85 | | - | |
| 87 | + | |
86 | 88 | | |
87 | 89 | | |
88 | 90 | | |
| |||
245 | 247 | | |
246 | 248 | | |
247 | 249 | | |
248 | | - | |
249 | | - | |
250 | | - | |
251 | | - | |
252 | | - | |
253 | | - | |
254 | | - | |
255 | | - | |
256 | | - | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
257 | 270 | | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
258 | 274 | | |
259 | 275 | | |
260 | 276 | | |
| |||
390 | 406 | | |
391 | 407 | | |
392 | 408 | | |
393 | | - | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
394 | 412 | | |
395 | 413 | | |
396 | 414 | | |
| |||
425 | 443 | | |
426 | 444 | | |
427 | 445 | | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
| 451 | + | |
| 452 | + | |
428 | 453 | | |
429 | 454 | | |
430 | 455 | | |
| |||
492 | 517 | | |
493 | 518 | | |
494 | 519 | | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
495 | 564 | | |
0 commit comments