I can see a few problems here.

1. process_slice is only called in the main search loop (while ptr < body_end_ptr { ... }) if the chunk contains non-ASCII characters. So line breaks won't be recorded unless they're at the very start or end of the source text, or they happen to have a non-ASCII char close to them.

The chunk.contains_unicode() check in that loop needs to be replaced with a check for "chunk contains unicode OR \r / \n".

Or maybe better: Check chunk for non-ASCII bytes and call process_slice if any are found (as now). Then do a 2nd check for any \n / \r bytes in the chunk, and call a different function if any are found. That 2nd function can be simpler and faster, as it doesn't have to deal with PS / LS (because they'd be caught by the non-ASCII check).

2. process_slice is passed a small subset of the source text (in the main loop, 32 bytes). If 0xE2 is the 31st byte, then PS/LS won't be identified, because the last 2 bytes are after the end of the slice.

3. Irregular line breaks are non-ASCII, so need a Translation record as well as LineTranslation. Simplest way to do that is skip the continue statement if line_break_len == 3.

Also, a small optimization: No need for 2 vars is_line_break and line_break_len. Just line_break_len would do, with 0 representing "not a line break".

Looks like this change was introduced while resolving merge conflicts from #13339. This PR has reverted the change #13339 made, which isn't right. Ditto other changes in this file. convert_offset_back and convert_span_back methods definitely shouldn't be removed!

thanks, I edited

This can be simplified.

1. Always check against full source text - to remove a branch.
2. Don't check if 2 more bytes. source_text is a &str, which is guaranteed valid UTF-8. So if a byte is 0xE2, it's guaranteed there are 2 more bytes after it. So just assert!(full_offset + 2 < source_text.len()) and then check the bytes.

That assertion can never fail. Later on, once we're completely confident of the implementation, we can remove that assert! and use unsafe indexing to get the next 2 bytes without bounds checks. But for now, safer to keep the assertion.

Note: I'm suggesting an assert! as opposed to relying on the implicit bounds checks in source_bytes[full_offset + 1] and source_bytes[full_offset + 2] because the assertion will do 1 bounds check instead of 2.

thanks for optimize, I edited

I think same problem here as with LS / PS. What if \r is the last byte in the chunk, and \n is in the next chunk? Need to check full source text to avoid that problem.

(but unlike PS / LS, you can't guarantee that \r isn't last byte in the file, so it shouldn't be asserted)

thanks, I edited

Move this into the if has_unicode || has_line_breaks { ... } block. If no line breaks or unicode chars in the block, we don't use this value, so avoid calculating it in that case.

thanks, I edited