cksum/hashsum: Support for non-UTF-8 input in checksum files #6793

RenjiSann · 2024-10-18T13:59:40Z

Second part of splitting #6603.

This PR adds support for non-UTF-8 checksum file content for cksum and hashsum validation tools.

In practice, given a CHECKSUM file with the following content, non-UTF-8 characters can appear in 2 places:

in the filename
in a comment
All the other places are either algorithm name, which should be ASCII, or hexa/base64 digest, which should be ASCII as well.

$ cat CHECKSUM
SHA1 (filename-might-have-non-utf-8) = 058ab38dd3603703b3a7063cf95dc51a4286b6fe
# comment-might-have-non-utf-8

The most important step is to get rid of the call to BufRead::lines() to iterate on the lines of the CHECKSUM file, otherwise, it panics on non-UTF-8 characters.

This change makes heavy use of String/OsString/Vec<u8> conversion, which happen to not be equally treated on all platforms.

github-actions · 2024-10-18T14:25:45Z

GNU testsuite comparison:

Congrats! The gnu test tests/timeout/timeout is no longer failing!

github-actions · 2024-10-19T00:05:59Z

GNU testsuite comparison:

Skip an intermittent issue tests/tail/inotify-dir-recreate (fails in this run but passes in the 'main' branch)

RenjiSann · 2024-10-19T15:46:38Z

I closed the PR by mistake, wanted to mark it as ready :/

github-actions · 2024-10-19T16:12:00Z

GNU testsuite comparison:

Congrats! The gnu test tests/timeout/timeout is no longer failing!

RenjiSann · 2024-10-19T23:25:04Z

@cakebaker @BenWiederhake
PR is ready for review if one of you can take a look :)

github-actions · 2024-10-19T23:49:09Z

GNU testsuite comparison:

Congrats! The gnu test tests/tail/assert is no longer failing!
Skipping an intermittent issue tests/tail/inotify-dir-recreate (passes in this run but fails in the 'main' branch)

cakebaker · 2024-10-23T07:19:28Z

@RenjiSann sorry, there are some conflicts :|

RenjiSann · 2024-10-23T09:13:02Z

@RenjiSann sorry, there are some conflicts :|

Fixed ! 😁

github-actions · 2024-10-23T09:38:23Z

GNU testsuite comparison:

Skip an intermittent issue tests/tail/inotify-dir-recreate (fails in this run but passes in the 'main' branch)
Congrats! The gnu test tests/misc/usage_vs_getopt is no longer failing!

src/uucore/src/lib/features/checksum.rs

tests/by-util/test_cksum.rs

github-actions · 2024-10-23T13:02:33Z

GNU testsuite comparison:

Congrats! The gnu test tests/misc/usage_vs_getopt is no longer failing!

src/uucore/src/lib/lib.rs

github-actions · 2024-10-23T15:31:27Z

GNU testsuite comparison:

Congrats! The gnu test tests/misc/usage_vs_getopt is no longer failing!
Congrats! The gnu test tests/timeout/timeout is no longer failing!

src/uucore/src/lib/features/checksum.rs

github-actions · 2024-10-23T22:01:38Z

GNU testsuite comparison:

Congrats! The gnu test tests/misc/usage_vs_getopt is no longer failing!
Congrats! The gnu test tests/timeout/timeout is no longer failing!

github-actions · 2024-10-24T08:42:11Z

GNU testsuite comparison:

Congrats! The gnu test tests/timeout/timeout is no longer failing!

The full line will be matched against the selected regex anyway

github-actions · 2024-10-24T10:10:58Z

GNU testsuite comparison:

Skip an intermittent issue tests/timeout/timeout (fails in this run but passes in the 'main' branch)

RenjiSann · 2024-10-24T10:58:45Z

Ready for what should be a final review :)

cakebaker · 2024-10-24T12:58:48Z

Thanks :)

RenjiSann changed the title ~~cksum/hashsum: Support for non-UTF-8 inputs~~ cksum/hashsum: Support for non-UTF-8 input in checksum files Oct 18, 2024

RenjiSann force-pushed the checksum-utf8 branch from febab10 to 71bc35b Compare October 18, 2024 23:34

RenjiSann mentioned this pull request Oct 18, 2024

cksum: Accept non UTF-8 inputs #6603

Closed

RenjiSann force-pushed the checksum-utf8 branch from 71bc35b to 7fa39e4 Compare October 18, 2024 23:39

RenjiSann force-pushed the checksum-utf8 branch 3 times, most recently from 050ad32 to 91c1cd8 Compare October 19, 2024 15:45

RenjiSann closed this Oct 19, 2024

RenjiSann reopened this Oct 19, 2024

RenjiSann marked this pull request as ready for review October 19, 2024 15:46

RenjiSann force-pushed the checksum-utf8 branch from 91c1cd8 to 57ac761 Compare October 19, 2024 16:11

RenjiSann force-pushed the checksum-utf8 branch 3 times, most recently from 3285e74 to 953fa37 Compare October 19, 2024 23:23

RenjiSann force-pushed the checksum-utf8 branch from 953fa37 to 838d433 Compare October 23, 2024 09:12

RenjiSann force-pushed the checksum-utf8 branch from 838d433 to 61ca177 Compare October 23, 2024 12:36

sylvestre reviewed Oct 23, 2024

View reviewed changes

src/uucore/src/lib/features/checksum.rs Outdated Show resolved Hide resolved

cakebaker reviewed Oct 23, 2024

View reviewed changes

tests/by-util/test_cksum.rs Outdated Show resolved Hide resolved

cakebaker reviewed Oct 23, 2024

View reviewed changes

src/uucore/src/lib/lib.rs Outdated Show resolved Hide resolved

RenjiSann force-pushed the checksum-utf8 branch from 61ca177 to 7e45735 Compare October 23, 2024 14:43

RenjiSann force-pushed the checksum-utf8 branch 3 times, most recently from 23d0c71 to ebe53e0 Compare October 23, 2024 15:05

cakebaker reviewed Oct 23, 2024

View reviewed changes

src/uucore/src/lib/features/checksum.rs Outdated Show resolved Hide resolved

cakebaker reviewed Oct 23, 2024

View reviewed changes

src/uucore/src/lib/features/checksum.rs Outdated Show resolved Hide resolved

RenjiSann force-pushed the checksum-utf8 branch 3 times, most recently from 178cc03 to 50bfa3d Compare October 23, 2024 21:35

RenjiSann added 6 commits October 24, 2024 11:41

uucore(lib): add u8/OsString conversion utils

3031874

uucore(checksum): determine regex from full line, not trimmed

97b81a6

The full line will be matched against the selected regex anyway

test(cksum): add test for trailing spaces making an error

1c41a12

uucore(checksum): handle non-utf-8 text in checksum files

8490461

uucore(checksum): fix non-utf-8 filename printing

99cddd1

test(cksum): add tests for non-utf-8 handling in checksum files

6030ad9

RenjiSann force-pushed the checksum-utf8 branch from 463c959 to 6030ad9 Compare October 24, 2024 09:44

cakebaker merged commit cb8711f into uutils:main Oct 24, 2024
67 of 68 checks passed

RenjiSann deleted the checksum-utf8 branch October 24, 2024 13:48

RenjiSann mentioned this pull request Oct 25, 2024

checksum: prepare further behavior fix with a rework #6822

Merged

BrewTestBot mentioned this pull request Nov 16, 2024

uutils-coreutils 0.0.28 Homebrew/homebrew-core#197947

Merged

RenjiSann mentioned this pull request Nov 17, 2024

cksum: --check can't handle non-UTF-8 in comments #6573

Closed

moonfruit mentioned this pull request Nov 22, 2024

uutils-selected 0.0.28 moonfruit/homebrew-tap#178

Closed

Uh oh!

cksum/hashsum: Support for non-UTF-8 input in checksum files #6793

cksum/hashsum: Support for non-UTF-8 input in checksum files #6793

Uh oh!

Conversation

RenjiSann commented Oct 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 18, 2024

Uh oh!

github-actions bot commented Oct 19, 2024

Uh oh!

RenjiSann commented Oct 19, 2024

Uh oh!

github-actions bot commented Oct 19, 2024

Uh oh!

RenjiSann commented Oct 19, 2024

Uh oh!

github-actions bot commented Oct 19, 2024

Uh oh!

cakebaker commented Oct 23, 2024

Uh oh!

RenjiSann commented Oct 23, 2024

Uh oh!

github-actions bot commented Oct 23, 2024

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Oct 23, 2024

Uh oh!

Uh oh!

github-actions bot commented Oct 23, 2024

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Oct 23, 2024

Uh oh!

github-actions bot commented Oct 24, 2024

Uh oh!

github-actions bot commented Oct 24, 2024

Uh oh!

RenjiSann commented Oct 24, 2024

Uh oh!

Uh oh!

cakebaker commented Oct 24, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

RenjiSann commented Oct 18, 2024 •

edited

Loading