Skip to content

Conversation

@Nerixyz
Copy link
Contributor

@Nerixyz Nerixyz commented Jun 8, 2024

This PR advances the Typst lexer. It's extended from #2656 by @jonaspleyer (see #2656 (comment)).

It adds a few snippets and most importantly, the example is now tokenized correctly. The example was fixed too (#2596 (comment)).

I described all commits I made in #2656 (comment) (which mainly worked its way towards correctly handling the example).

It might be nice to add an option for the lexer to start in code (similar to the startinline option for PHP) - I'm not sure what the best implementation of that would be, though.
I've added the start option, which specifies the starting state of the lexer (one of 'markup', 'code', 'math') - this can be useful when highlighting Typst code inline.


Original description of #2656:

This PR adds multiple improvements to the existing typst lexer and some tests

  • One of these tests is currently failing. This is by design.

I cannot guarantee that I will be working on this problem too much but wanted to get these commits out here.
Please tell me if the failing test prohibits merging. I can comment it out such that future development can reference it.

Happy about any feedback.

@jonaspleyer
Copy link
Contributor

Thanks @Nerixyz for your continued efforts on this topic!

Copy link
Contributor

@oliverpool oliverpool left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the work, it looks good!

(note: not a maintainer, just a random user)

@Nerixyz
Copy link
Contributor Author

Nerixyz commented Jul 8, 2024

Thanks for the work, it looks good!

(note: not a maintainer, just a random user)

Oopsie, I should've read the docs better. Thank you for pointing that out!

@oliverpool
Copy link
Contributor

That was quick 😄

I imported your lexer into chroma and tested it with the following snippet:

#let song(
  title: none,
  title_index: none,
  singer: none,
  singer_index: none,
  references: (),
  line_color: rgb(0xd0, 0xd0, 0xd0),
  header_display: (number, title, singer) => grid(
    columns: (auto,1fr),
    inset: 4pt,
    align: horizon,
    grid.cell(fill: rgb(0xd0, 0xd0, 0xd0),align: top, inset:(rest:4pt, /*right: 10pt*/), heading(outlined: false,str(number))),
    [
      #heading(title)
      #if singer != none {
        strong(singer) + "\n"
      }
    ],
  ),
  // comment
  doc,
)

the \n looks off (and the nearby #if is not styled):

image

This may be due to the conversion to chroma's xml (I don't know how to use pygment to quickly render a snippet, so maybe this issue is not present here), but maybe you can reproduce it/test it as well?

@Nerixyz Nerixyz force-pushed the fix/typst-advance branch from e39e73d to 1d48b45 Compare July 8, 2024 10:55
@Nerixyz
Copy link
Contributor Author

Nerixyz commented Jul 8, 2024

the \n looks off (and the nearby #if is not styled):

I missed that. Should be supported now (although with some quirks):

#if foo {
  bar
} else {
  baz
} qox

In this case, qox would become a variable because the "inline_code" context lasts until the next newline or ;. But as you can see by the current output from GitHub, the textmate grammer doesn't even support inline #if {} else {}.

@Nerixyz Nerixyz force-pushed the fix/typst-advance branch from 1d48b45 to d3549cb Compare July 8, 2024 10:59
@Nerixyz Nerixyz force-pushed the fix/typst-advance branch from d3549cb to 2fcb426 Compare July 8, 2024 11:01
Copy link
Contributor

@oliverpool oliverpool left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the latest changes, my own typst examples look very good 👍 (I don't use math, but a good mix of markup, content and functions).

@Nerixyz
Copy link
Contributor Author

Nerixyz commented Jul 8, 2024

With the latest changes, my own typst examples look very good 👍 (I don't use math, but a good mix of markup, content and functions).

Thank you for providing feedback! I also use it on https://nerixyz.github.io/icu-typ/ (mostly code). If there are any further issues, feel free to @ me.

@Anteru Anteru merged commit a67a515 into pygments:master Oct 6, 2024
@Anteru
Copy link
Collaborator

Anteru commented Oct 6, 2024

I'll trust you guys you know what you're doing here, the lexing looks fine.

@Anteru Anteru added this to the 2.19.0 milestone Oct 6, 2024
@Anteru Anteru added the A-lexing area: changes to individual lexers label Oct 6, 2024
@Nerixyz Nerixyz deleted the fix/typst-advance branch October 6, 2024 16:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-lexing area: changes to individual lexers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants