TL;DR

This is an issue related to "escape sequences" in JavaScript string literals.

When using the RegExp constructor instead of /regex/ literals
And if an escape sequence is included in the string literal

The corresponding position might not be correctly retrieved from the parsed Span.

Simplified

If there is a "file" containing:

"O\"X\"C"

and you want to slice each O, X, and C, you need to calculate the slice position like this.

fn main() {
    let source_text = std::fs::read_to_string("./test.js").unwrap();
    assert_eq!("O", &source_text[1..2]);
                 // &source_text[2..3]; \ 👈🏻
                 // &source_text[3..4]; "
    assert_eq!("X", &source_text[4..5]);
                 // &source_text[5..6]; \ 👈🏻
                 // &source_text[6..7]; "
    assert_eq!("C", &source_text[7..8]);
}

You need to include \ as part of the slicing index.

However, if you're just handling the "string":

fn main() {
    let source_text = "O\"X\"C";
    assert_eq!("O", &source_text[0..1]);
                 // &source_text[1..2]; "
    assert_eq!("X", &source_text[2..3]);
                 // &source_text[3..4]; "
    assert_eq!("C", &source_text[4..5]);
}

There's no need to consider the \.

This difference is causing problems when a diagnostic tries to slice the file.

Current status

As an implementation detail, oxc_regular_expression is not aware of these escape sequences.

In most cases, the current code works with an AST with oxc_parser's StringLiteral and its value is passed to oxc_regular_expression.

At this point from oxc_regular_expression, /O"X"C/ and new RegExp("O\"X\"C") seems equivalent, and new RegExp('O"X"C') is also the same. (Both "\"'`" and '"\'`' are normalized as "\"'`" in StringLiteral.value.)

Thus, the regular expression can be parsed without needing to be aware of these differences.

However, when attempting to slice the original code string using the parsed Span, issues may arise.

As seen in the original PR:

new RegExp('\\u{1111}*\\x1F', 'u')
            ^         ^

These 2 \ cause a misalignment as a result.

At that time, I was aware of this issue but was only focused on parsing.

#3824 (comment)
#5106

I was satisfied with oxc_parser resolving the escapes.

How to fix

In my view, the only solution is to update oxc_regular_expression to be aware of escape sequences.

When parsing a regular expression from the RegExp constructor, use string_literal.span.source_text(&self.source_text) instead of string_literal.value
- This is the entire string, including quotes
- Then pass inner_text, quote_type, span_offset?
In oxc_regular_expression:
- Introduce a lexer layer in front of the current Reader implementation
- Tokenize the text based on the quote type, similar to oxc_parser/lexer
  - oxc/crates/oxc_parser/src/lexer/string.rs
    
    Line 155 in 21b08ba
    
    impl<'a> Lexer<'a> {
- Should consider template literals surrounded by ` too?
  - oxc/crates/oxc_parser/src/lexer/template.rs
    
    Line 17 in 21b08ba
    
    impl<'a> Lexer<'a> {
- Provide a mode toggle to use the lexer or not?
  - When parsing a regex literal, lexing is unnecessary
  - Also unnecessary when using as a standalone regex parser

Are there any other approaches to consider...?

Uh oh!

regular_expression: report character span offsets not byte offsets #6141

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions