This PR is ready to be reviewed and merged, even if the CI job on 3.12-dev is failing. See: #2445

Thanks! I'm a bit worried that the examples in the doc string will actually work with the lexer .. I'd prefer them to be also committed as test snippets. At that point however, we probably shouldn't duplicate them from the test snippets into the documentation. This could be something to solve via a private class field for the documentation, maybe something like _example = '/path/to/example' which is then parsed by the lexer page generator. This would have to go here: https://github.com/pygments/pygments/blob/master/pygments/sphinxext.py#L148

This may sound like overkill, but I think that's the best long term solution to make sure those things don't go out of sync, and it's probably (long term) a good idea to have some example file linked to every lexer.

Thanks @Anteru for the detailed feedback!

I have these kind of reflexion too with my Click Extra project, in which I have to maintain some code to keep the document and code in sync.

I have several strategies for that:

• dedicated Sphinx extension to produce Click code and invokation example
• stringent unittests to check the metadata and content of third-party dependencies (including Pygments itself)
• regular runs via GitHub workflows to automatically produce dynamic documentation and push a PR

It's a little bit all over the place but does the job of reducing the cognitive work of maintenance. I'm just pointing to my stuff out as inspiration: it might not be adapted to Pygments. And I guess I know enough about Pygments and Sphinx to attempt an implementation of the long term solution you described.

But should this be done in this PR? I don't think so because of the bigger scope of that feature: it would introduce new code while this PR is only introducing rST content. So what's missing for this PR to be considered for a merge upstream?

Honestly, I'm not sure if the added cost of having to convert this later is worth to merge this now (I'm really afraid of code rot quickly.) I'll see if I can spare some cycles to convert this, maybe it's not as complicated as I'm afraid it is. I'm not a huge fan of having this extra field on the lexer class to point to a document in the test tree, so the other alternative is to have tests somehow marked up as examples that should show up in the doc for that particular lexer. Let me think a bit more about this, and at the very least try to prototype the _example thing ... I did some work on the Sphinx extension and it feels like this shouldn't be that hard other than having to somehow pipe it through Pygments while building the docs and then insert it back into the doctree.

Since your last comment @Anteru, I realized you went ahead with the _example attribute and implemented it upstream in #2451.

So I updated this PR to use it to document all REPL sessions. All tests are passing, this PR is ready to be merged upstream! :)

@Anteru I just updated the PR with your requested changes, and squashed the commits. All tests are passing, this PR is ready to be merged upstream.

Thanks!