Skip to content

Conversation

@jonaspleyer
Copy link
Contributor

@jonaspleyer jonaspleyer commented Feb 27, 2024

This PR adds multiple improvements to the existing typst lexer and some tests

  • One of these tests is currently failing. This is by design.

I cannot guarantee that I will be working on this problem too much but wanted to get these commits out here.
Please tell me if the failing test prohibits merging. I can comment it out such that future development can reference it.

Happy about any feedback.

@miliog
Copy link

miliog commented Feb 28, 2024

I'm not sure if you saw, but the Typst test file which was previously introduced is invalid, see #2596 (comment) (or that is what you meant by failing by design idk)

@jonaspleyer
Copy link
Contributor Author

I'm not sure if you saw, but the Typst test file which was previously introduced is invalid, see #2596 (comment) (or that is what you meant by failing by design idk)

I did not see the original typst file to be honest. I added my own tests which are much more specific and only parse individual snippets. See commit 93e9eea. There one of the tests is failing. Of course, this is due to the lexer not picking it up correctly. Think of it as TDD ;-) where the correct implementation is still missing.

Copy link
Contributor

@jeanas jeanas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the delayed review. Please do the testing via our normal infrastructure with a file in tests/snippets/typst/, not with extra Python code, see https://pygments.org/docs/lexerdevelopment/#how-to-test-your-lexer.

@Nerixyz
Copy link
Contributor

Nerixyz commented Jun 8, 2024

Hi, since this wasn't updated in a while, I did some work on it.

  • I refactored the tests from test_typst.py into snippets in Nerixyz@c229ee7 (as mentioned in advance development of typst lexer #2656 (review))
  • The current example code isn't valid Typst, so I fixed it in Nerixyz@37077fa (this can't be compiled, but that's because darkgrey and section_date don't exist which the lexer shouldn't have to care about)
  • Code can't contain text, so I moved the text to markup (you can't have raw links in code for example), moved the literal case introduced in this PR down, and introduced a case for variables in Nerixyz@8e301d3
  • Inline code was broken on linebreaks - fixed in Nerixyz@c1d6691
  • Typst has the convention to use kebab-case and - wasn't supported in variables and function names - fixed in Nerixyz@342f65f
  • #{ was treated as inline code, which meant it would pop back to markup after a linebreak - fixed in Nerixyz@2230465
  • Special characters like --- or \ weren't recognized, because they used the \b suffix - fixed in Nerixyz@5746cbe
  • The text regex in markup would capture too much - most importantly, it would capture functions - fixed in Nerixyz@06dc2f0
  • To ensure nesting of #{ } code works correctly, I added another snippet in Nerixyz@9fb2aba
  • Finally, I added support for #import and #include in Nerixyz@1c9fd95

All commits: https://github.com/Nerixyz/pygments/commits/fix/typst-advance/
Full diff to current state: Nerixyz/pygments@93e9eea...1c9fd95

@jonaspleyer let me know if I should PR the fixes to your branch or if I should open another PR here.

@jonaspleyer
Copy link
Contributor Author

jonaspleyer commented Jun 8, 2024

@jonaspleyer let me know if I should PR the fixes to your branch or if I should open another PR here.

To be honest I had kind of forgotten about this PR until today. I have had a brief glance on the individual commits (without too much depth) and they seem reasonable to me.

I currently do not have the time to further contribute to this PR. Thus it would make sense to open a new one instead. Feel free to tag me once you have done so. Afterwards I will close this PR.
I am only concerned about my contributions showing up correctly but I see that you have simply pulled my repository and committed on top. This should probably be enough.

Thanks for your effort.

@Nerixyz Nerixyz mentioned this pull request Jun 8, 2024
@jonaspleyer
Copy link
Contributor Author

closing in favor of #2724

@jonaspleyer jonaspleyer closed this Jun 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants