First draft of utf-8 to utf-16 conversion by lemire · Pull Request #5 · simdutf/simdutf

lemire · 2021-02-18T00:56:22Z

This makes it so that UTF16 data is on char16_t pointers, while UTF8 data is on char pointers. Technically, we should use char8_t but I believe it is a C++17 feature.
It includes an ugly copy-paste of a fast utf8-to-utf16 for SSE.
It brings back runtime dispatch on x64 hardware (hopefully)
It changes some classes name from CamelCase to under_score style.

…mire/simdutf into dlemire/portingsseutf8decoder

…dutf into dlemire/portingsseutf8decoder

lemire · 2021-02-19T02:14:42Z

@WojciechMula This is ready for review, we could merge this. I will open related issues about what I do not like...

This is not meant to be good... just a first step.

WojciechMula

Apart from minor comments, everything looks nice. Good job.

WojciechMula · 2021-02-19T17:32:56Z

+   * transcode_utf8_to_utf16_test_base can be used to test UTF8 => UTF16 transcoding.
+   */
+
+  transcode_utf8_to_utf16_test_base::transcode_utf8_to_utf16_test_base(GenerateCodepoint generate,


Just a naming convention remark: I personally use CamelCase for classes and enums, lowercase_with_undescore for methods/variables.Thus this class may be named: TranscodeUTF8toUTF16TestBase. Do we want to have a consistent naming style?

Yes. I do not care much, but as you may notice much_of_the_code_follows_the_lower_case_with_underscore_convention. :-)

It is common in C++... The standard library has from_chars and to_string and unordered_set.

The convention you seem to favour seems more common in Java/C#.

Ok, I'll follow this lower-case convention.

lemire · 2021-02-19T18:54:24Z

@WojciechMula If you are happy with this PR, we can merge. I have opened issue regarding the components that need work.

I am not in a hurry to merge per se, as I am working now on the NEON port and algorithmic issue, but if you want to do some work, we would prefer to avoid conflicts.

WojciechMula · 2021-02-19T19:00:00Z

Please merge the PR. Everything is fine.

lemire · 2021-02-19T19:07:58Z

Merging.

Please do not hesitate to criticize my code!

==17876==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x517000000307 at pc 0x561dd0419320 bp 0x7ffd11808360 sp 0x7ffd11808358 WRITE of size 16 at 0x517000000307 thread T0 #0 0x561dd041931f in simdutf::haswell::(anonymous namespace)::convert_masked_utf8_to_latin1(char const*, unsigned long, char*&) /home/pauldreik/code/delaktig/simdutf/src/haswell/avx2_convert_utf8_to_latin1.cpp:28:5 #1 0x561dd037a5d7 in simdutf::haswell::(anonymous namespace)::utf8_to_latin1::validating_transcoder::convert(char const*, unsigned long, char*) /home/pauldreik/code/delaktig/simdutf/src/generic/utf8_to_latin1/utf8_to_latin1.h:178:29 #2 0x561dd037a06b in simdutf::haswell::implementation::convert_utf8_to_latin1(char const*, unsigned long, char*) const /home/pauldreik/code/delaktig/simdutf/src/haswell/implementation.cpp:309:20 #3 0x561dd0286e12 in test_impl_ossfuzz_372067232(simdutf::implementation const&) /home/pauldreik/code/delaktig/simdutf/tests/convert_utf8_to_latin1_tests.cpp:93:33 #4 0x561dd028516f in ossfuzz_372067232(simdutf::implementation const&) /home/pauldreik/code/delaktig/simdutf/tests/convert_utf8_to_latin1_tests.cpp:20:1 #5 0x561dd0299782 in simdutf::test::test_entry::operator()(simdutf::implementation const&) /home/pauldreik/code/delaktig/simdutf/tests/helpers/test.h:18:58 #6 0x561dd02926ca in (anonymous namespace)::run((anonymous namespace)::CommandLine const&) /home/pauldreik/code/delaktig/simdutf/tests/helpers/test.cpp:179:9 #7 0x561dd0290fc5 in simdutf::test::main(int, char**) /home/pauldreik/code/delaktig/simdutf/tests/helpers/test.cpp:207:3 #8 0x561dd028d611 in main /home/pauldreik/code/delaktig/simdutf/tests/convert_utf8_to_latin1_tests.cpp:293:1 #9 0x7f31b415adb9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 #10 0x7f31b415ae74 in __libc_start_main csu/../csu/libc-start.c:360:3 #11 0x561dd01a9650 in _start (/home/pauldreik/code/delaktig/simdutf/build/paul_clang_18-Debug/tests/convert_utf8_to_latin1_tests+0x35650) (BuildId: ab54037bea01a246225d9f551b72ba81eb4f6416) 0x517000000307 is located 0 bytes after 647-byte region [0x517000000080,0x517000000307) allocated by thread T0 here: #0 0x561dd0282be1 in operator new(unsigned long) (/home/pauldreik/code/delaktig/simdutf/build/paul_clang_18-Debug/tests/convert_utf8_to_latin1_tests+0x10ebe1) (BuildId: ab54037bea01a246225d9f551b72ba81eb4f6416) #1 0x561dd028f28e in std::__new_allocator<char>::allocate(unsigned long, void const*) /usr/bin/../lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/new_allocator.h:151:27 #2 0x561dd0290023 in std::allocator<char>::allocate(unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/allocator.h:196:32 #3 0x561dd0290023 in std::allocator_traits<std::allocator<char>>::allocate(std::allocator<char>&, unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/alloc_traits.h:478:20 #4 0x561dd0290023 in std::_Vector_base<char, std::allocator<char>>::_M_allocate(unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/stl_vector.h:380:20 #5 0x561dd028fe20 in std::_Vector_base<char, std::allocator<char>>::_M_create_storage(unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/stl_vector.h:398:33 #6 0x561dd028f9d1 in std::_Vector_base<char, std::allocator<char>>::_Vector_base(unsigned long, std::allocator<char> const&) /usr/bin/../lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/stl_vector.h:334:9 #7 0x561dd028e4d8 in std::vector<char, std::allocator<char>>::vector(unsigned long, std::allocator<char> const&) /usr/bin/../lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/stl_vector.h:557:9 #8 0x561dd0286d70 in test_impl_ossfuzz_372067232(simdutf::implementation const&) /home/pauldreik/code/delaktig/simdutf/tests/convert_utf8_to_latin1_tests.cpp:92:21 #9 0x561dd028516f in ossfuzz_372067232(simdutf::implementation const&) /home/pauldreik/code/delaktig/simdutf/tests/convert_utf8_to_latin1_tests.cpp:20:1 #10 0x561dd0299782 in simdutf::test::test_entry::operator()(simdutf::implementation const&) /home/pauldreik/code/delaktig/simdutf/tests/helpers/test.h:18:58 #11 0x561dd02926ca in (anonymous namespace)::run((anonymous namespace)::CommandLine const&) /home/pauldreik/code/delaktig/simdutf/tests/helpers/test.cpp:179:9 #12 0x561dd0290fc5 in simdutf::test::main(int, char**) /home/pauldreik/code/delaktig/simdutf/tests/helpers/test.cpp:207:3 #13 0x561dd028d611 in main /home/pauldreik/code/delaktig/simdutf/tests/convert_utf8_to_latin1_tests.cpp:293:1 #14 0x7f31b415adb9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 SUMMARY: AddressSanitizer: heap-buffer-overflow /home/pauldreik/code/delaktig/simdutf/src/haswell/avx2_convert_utf8_to_latin1.cpp:28:5 in simdutf::haswell::(anonymous namespace)::convert_masked_utf8_to_latin1(char const*, unsigned long, char*&) Shadow bytes around the buggy address: 0x517000000080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x517000000100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x517000000180: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x517000000200: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x517000000280: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =>0x517000000300:[07]fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x517000000380: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x517000000400: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x517000000480: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x517000000500: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x517000000580: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb ==17876==ABORTING

) ==17876==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x517000000307 at pc 0x561dd0419320 bp 0x7ffd11808360 sp 0x7ffd11808358 WRITE of size 16 at 0x517000000307 thread T0 #0 0x561dd041931f in simdutf::haswell::(anonymous namespace)::convert_masked_utf8_to_latin1(char const*, unsigned long, char*&) /home/pauldreik/code/delaktig/simdutf/src/haswell/avx2_convert_utf8_to_latin1.cpp:28:5 #1 0x561dd037a5d7 in simdutf::haswell::(anonymous namespace)::utf8_to_latin1::validating_transcoder::convert(char const*, unsigned long, char*) /home/pauldreik/code/delaktig/simdutf/src/generic/utf8_to_latin1/utf8_to_latin1.h:178:29 #2 0x561dd037a06b in simdutf::haswell::implementation::convert_utf8_to_latin1(char const*, unsigned long, char*) const /home/pauldreik/code/delaktig/simdutf/src/haswell/implementation.cpp:309:20 #3 0x561dd0286e12 in test_impl_ossfuzz_372067232(simdutf::implementation const&) /home/pauldreik/code/delaktig/simdutf/tests/convert_utf8_to_latin1_tests.cpp:93:33 #4 0x561dd028516f in ossfuzz_372067232(simdutf::implementation const&) /home/pauldreik/code/delaktig/simdutf/tests/convert_utf8_to_latin1_tests.cpp:20:1 #5 0x561dd0299782 in simdutf::test::test_entry::operator()(simdutf::implementation const&) /home/pauldreik/code/delaktig/simdutf/tests/helpers/test.h:18:58 #6 0x561dd02926ca in (anonymous namespace)::run((anonymous namespace)::CommandLine const&) /home/pauldreik/code/delaktig/simdutf/tests/helpers/test.cpp:179:9 #7 0x561dd0290fc5 in simdutf::test::main(int, char**) /home/pauldreik/code/delaktig/simdutf/tests/helpers/test.cpp:207:3 #8 0x561dd028d611 in main /home/pauldreik/code/delaktig/simdutf/tests/convert_utf8_to_latin1_tests.cpp:293:1 #9 0x7f31b415adb9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 #10 0x7f31b415ae74 in __libc_start_main csu/../csu/libc-start.c:360:3 #11 0x561dd01a9650 in _start (/home/pauldreik/code/delaktig/simdutf/build/paul_clang_18-Debug/tests/convert_utf8_to_latin1_tests+0x35650) (BuildId: ab54037bea01a246225d9f551b72ba81eb4f6416) 0x517000000307 is located 0 bytes after 647-byte region [0x517000000080,0x517000000307) allocated by thread T0 here: #0 0x561dd0282be1 in operator new(unsigned long) (/home/pauldreik/code/delaktig/simdutf/build/paul_clang_18-Debug/tests/convert_utf8_to_latin1_tests+0x10ebe1) (BuildId: ab54037bea01a246225d9f551b72ba81eb4f6416) #1 0x561dd028f28e in std::__new_allocator<char>::allocate(unsigned long, void const*) /usr/bin/../lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/new_allocator.h:151:27 #2 0x561dd0290023 in std::allocator<char>::allocate(unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/allocator.h:196:32 #3 0x561dd0290023 in std::allocator_traits<std::allocator<char>>::allocate(std::allocator<char>&, unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/alloc_traits.h:478:20 #4 0x561dd0290023 in std::_Vector_base<char, std::allocator<char>>::_M_allocate(unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/stl_vector.h:380:20 #5 0x561dd028fe20 in std::_Vector_base<char, std::allocator<char>>::_M_create_storage(unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/stl_vector.h:398:33 #6 0x561dd028f9d1 in std::_Vector_base<char, std::allocator<char>>::_Vector_base(unsigned long, std::allocator<char> const&) /usr/bin/../lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/stl_vector.h:334:9 #7 0x561dd028e4d8 in std::vector<char, std::allocator<char>>::vector(unsigned long, std::allocator<char> const&) /usr/bin/../lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/stl_vector.h:557:9 #8 0x561dd0286d70 in test_impl_ossfuzz_372067232(simdutf::implementation const&) /home/pauldreik/code/delaktig/simdutf/tests/convert_utf8_to_latin1_tests.cpp:92:21 #9 0x561dd028516f in ossfuzz_372067232(simdutf::implementation const&) /home/pauldreik/code/delaktig/simdutf/tests/convert_utf8_to_latin1_tests.cpp:20:1 #10 0x561dd0299782 in simdutf::test::test_entry::operator()(simdutf::implementation const&) /home/pauldreik/code/delaktig/simdutf/tests/helpers/test.h:18:58 #11 0x561dd02926ca in (anonymous namespace)::run((anonymous namespace)::CommandLine const&) /home/pauldreik/code/delaktig/simdutf/tests/helpers/test.cpp:179:9 #12 0x561dd0290fc5 in simdutf::test::main(int, char**) /home/pauldreik/code/delaktig/simdutf/tests/helpers/test.cpp:207:3 #13 0x561dd028d611 in main /home/pauldreik/code/delaktig/simdutf/tests/convert_utf8_to_latin1_tests.cpp:293:1 #14 0x7f31b415adb9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 SUMMARY: AddressSanitizer: heap-buffer-overflow /home/pauldreik/code/delaktig/simdutf/src/haswell/avx2_convert_utf8_to_latin1.cpp:28:5 in simdutf::haswell::(anonymous namespace)::convert_masked_utf8_to_latin1(char const*, unsigned long, char*&) Shadow bytes around the buggy address: 0x517000000080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x517000000100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x517000000180: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x517000000200: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x517000000280: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =>0x517000000300:[07]fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x517000000380: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x517000000400: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x517000000480: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x517000000500: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x517000000580: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb ==17876==ABORTING

lemire and others added 8 commits February 17, 2021 15:45

Adding conversion placeholder files.

0ccb320

Tweaking.

27b2f2c

Tweaking.

9b23761

Tweaking Power

44cf39d

Tweaks.

0399f08

Looks good.

7bf26c8

Commenting.

433cae1

Merge branch 'dlemire/portingsseutf8decoder' of https://github.com/le…

643d45a

…mire/simdutf into dlemire/portingsseutf8decoder

WojciechMula reviewed Feb 18, 2021

View reviewed changes

Comment thread benchmarks/CMakeLists.txt Outdated

Comment thread src/haswell/implementation.cpp

Comment thread tests/helpers/transcode_test_base.h Outdated

lemire added 3 commits February 18, 2021 19:46

More reengineering.

9677ccc

Merge branch 'dlemire/portingsseutf8decoder' of github.com:lemire/sim…

c1ba307

…dutf into dlemire/portingsseutf8decoder

Ok.

a8d2164

lemire changed the title ~~(draft) Upcoming changes~~ First draft of utf-8 to utf-16 conversion Feb 19, 2021

lemire requested review from WojciechMula February 19, 2021 02:14

WojciechMula reviewed Feb 19, 2021

View reviewed changes

lemire added 3 commits February 19, 2021 13:34

Removing space.

f3da26e

This removes some of the duplication.

1341b45

Removing obsolete comment.

f7c0d91

lemire mentioned this pull request Feb 19, 2021

Functions in namespace finisher_functions need to be redone from scratch (PR 5) #13

Closed

lemire merged commit 2c9bf3a into master Feb 19, 2021

lemire deleted the dlemire/portingsseutf8decoder branch July 7, 2021 19:34

pauldreik mentioned this pull request Jul 31, 2024

fix: correct AVX-512 UTF-32 validator #438

Merged

pauldreik mentioned this pull request Aug 7, 2024

out of bounds write in utf16_length_from_utf8 (icelake) #483

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

First draft of utf-8 to utf-16 conversion#5

First draft of utf-8 to utf-16 conversion#5
lemire merged 14 commits intomasterfrom
dlemire/portingsseutf8decoder

lemire commented Feb 18, 2021 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lemire commented Feb 19, 2021

Uh oh!

WojciechMula left a comment

Uh oh!

Uh oh!

Uh oh!

WojciechMula Feb 19, 2021

Uh oh!

lemire Feb 19, 2021

Uh oh!

WojciechMula Feb 19, 2021

Uh oh!

lemire commented Feb 19, 2021

Uh oh!

WojciechMula commented Feb 19, 2021

Uh oh!

lemire commented Feb 19, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

lemire commented Feb 18, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lemire commented Feb 19, 2021

Uh oh!

WojciechMula left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

WojciechMula Feb 19, 2021

Choose a reason for hiding this comment

Uh oh!

lemire Feb 19, 2021

Choose a reason for hiding this comment

Uh oh!

WojciechMula Feb 19, 2021

Choose a reason for hiding this comment

Uh oh!

lemire commented Feb 19, 2021

Uh oh!

WojciechMula commented Feb 19, 2021

Uh oh!

lemire commented Feb 19, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

lemire commented Feb 18, 2021 •

edited

Loading