[ty] First cut at semantic token provider by UnboundVariable · Pull Request #19108 · astral-sh/ruff

UnboundVariable · 2025-07-02T22:45:03Z

This PR implements a basic semantic token provider for ty's language server. This allows for more accurate semantic highlighting / coloring within editors that support this LSP functionality.

Here are screen shots that show how code appears in VS Code using the "rainbow" theme both before and after this change.

The token types and modifier tags in this implementation largely mirror those used in Microsoft's default language server for Python.

The implementation supports two LSP interfaces. The first provides semantic tokens for an entire document, and the second returns semantic tokens for a requested range within a document.

The PR includes unit tests. It also includes comments that document known limitations and areas for future improvements.

github-actions · 2025-07-02T22:52:36Z

`mypy_primer` results

No ecosystem changes detected ✅
No memory usage changes detected ✅

codspeed-hq · 2025-07-02T22:57:49Z

CodSpeed WallTime Performance Report

Merging #19108 will not alter performance

_{Comparing UnboundVariable:semantic_tokens (3be32e1) with main (4dd2c03)}

Summary

✅ 7 untouched benchmarks

carljm

Thanks for the PR! I'd prefer for Micha or Dhruv (who know the LSP and our plans in that area better) to review this; just one initial comment.

crates/ty_server/src/server/api/requests/semantic_tokens.rs

dhruvmanila

This is a great start, thank you for doing this!

I've been reviewing this for an hour and I need to take a lunch break but here are my initial comments. I'll look at the remaining parts after the lunch break

crates/ty_ide/src/semantic_tokens.rs

dhruvmanila

Ok, my review is finally done, this is an awesome PR!

The test cases are extremely thorough, thank you for writing them!

I think the only required change would be to make sure that we convert the ty location values back to the LSP values in the common function for the semantic tokens handler. And, around the visitor implementation where some of the calls that might lead to duplicate tokens could be removed.

A lot of my review comments could be considered as suggestions so feel free to either apply them or discard them. It's totally fine if those are done as follow-up. I can also own them as follow-up to this PR if you prefer.

crates/ty_ide/src/semantic_tokens.rs

crates/ty_server/src/server/api/requests/semantic_tokens.rs

crates/ty_ide/src/semantic_tokens.rs

crates/ty_server/src/server/api/requests/semantic_tokens.rs

…ory allocation.

…None` means "entire document".

…er than `Option<SemanticTokens>`.

…g `tokens` as a public field.

…ters.

…als.

UnboundVariable · 2025-07-04T22:08:01Z

@dhruvmanila, thanks for the thorough and insightful code review comments! I think I've addressed them all.

dhruvmanila

This is great! Thank you for taking this on and addressing all the review comments!

crates/ty_ide/src/semantic_tokens.rs

crates/ty_server/src/server/api/semantic_tokens.rs

MichaReiser · 2025-07-07T12:11:38Z

This can be done in a sepreate PR, but it would be great to add semantic highlighting to the playground too. It should be "as easy as" implementing the necessary provider and registering them here:

ruff/playground/ty/src/Editor/Editor.tsx

Lines 143 to 180 in a663796

    
           class PlaygroundServer 
        
             implements 
        
               languages.TypeDefinitionProvider, 
        
               editor.ICodeEditorOpener, 
        
               languages.HoverProvider, 
        
               languages.InlayHintsProvider, 
        
               languages.DocumentFormattingEditProvider, 
        
               languages.CompletionItemProvider 
        
           { 
        
             private typeDefinitionProviderDisposable: IDisposable; 
        
             private editorOpenerDisposable: IDisposable; 
        
             private hoverDisposable: IDisposable; 
        
             private inlayHintsDisposable: IDisposable; 
        
             private formatDisposable: IDisposable; 
        
             private completionDisposable: IDisposable; 
        
             constructor( 
        
               private monaco: Monaco, 
        
               private props: PlaygroundServerProps, 
        
             ) { 
        
               this.typeDefinitionProviderDisposable = 
        
                 monaco.languages.registerTypeDefinitionProvider("python", this); 
        
               this.hoverDisposable = monaco.languages.registerHoverProvider( 
        
                 "python", 
        
                 this, 
        
               ); 
        
               this.inlayHintsDisposable = monaco.languages.registerInlayHintsProvider( 
        
                 "python", 
        
                 this, 
        
               ); 
        
               this.completionDisposable = monaco.languages.registerCompletionItemProvider( 
        
                 "python", 
        
                 this, 
        
               ); 
        
               this.editorOpenerDisposable = monaco.editor.registerEditorOpener(this); 
        
               this.formatDisposable = 
        
                 monaco.languages.registerDocumentFormattingEditProvider("python", this); 
        
             }

MichaReiser · 2025-07-07T16:11:43Z

I'll leave it up to you if this PR should close astral-sh/ty#260

…d the multilineTokenSupport capability of the client.

github-actions · 2025-07-07T22:31:57Z

`ruff-ecosystem` results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

Formatter (stable)

✅ ecosystem check detected no format changes.

Formatter (preview)

✅ ecosystem check detected no format changes.

…c_tokens * 'main' of https://github.com/astral-sh/ruff: (27 commits) [ty] First cut at semantic token provider (astral-sh#19108) [`flake8-simplify`] Make example error out-of-the-box (`SIM116`) (astral-sh#19111) [`flake8-use-pathlib`] Make example error out-of-the-box (`PTH210`) (astral-sh#19189) [`flake8-use-pathlib`] Add autofixes for `PTH203`, `PTH204`, `PTH205` (astral-sh#18922) [`flake8-type-checking`] Fix syntax error introduced by fix (`TC008`) (astral-sh#19150) [`flake8-pyi`] Make example error out-of-the-box (`PYI007`, `PYI008`) (astral-sh#19103) Update Rust crate indicatif to 0.18.0 (astral-sh#19165) [ty] Add separate CI job for memory usage stats (astral-sh#19134) [ty] Add documentation for server traits (astral-sh#19137) Rename to `SessionSnapshot`, move unwind assertion closer (astral-sh#19177) [`flake8-type-checking`] Make example error out-of-the-box (`TC001`) (astral-sh#19151) [ty] Bare `ClassVar` annotations (astral-sh#15768) [ty] Re-enable multithreaded pydantic benchmark (astral-sh#19176) [ty] Implement equivalence for protocols with method members (astral-sh#18659) [ty] Use RHS inferred type for bare `Final` symbols (astral-sh#19142) [ty] Support declaration-only attributes (astral-sh#19048) [ty] Sync vendored typeshed stubs (astral-sh#19174) Update dependency pyodide to ^0.28.0 (astral-sh#19164) Update NPM Development dependencies (astral-sh#19170) Update taiki-e/install-action action to v2.56.7 (astral-sh#19169) ...

First cut at semantic token provider.

6e23135

UnboundVariable requested review from AlexWaygood, MichaReiser, carljm, dcreager and sharkdp as code owners July 2, 2025 22:45

Fixed formatting issues.

cfa7aaf

UnboundVariable changed the title ~~First cut at semantic token provider.~~ [ty] First cut at semantic token provider Jul 2, 2025

Fixed typos

aeb36f5

carljm reviewed Jul 3, 2025

View reviewed changes

crates/ty_server/src/server/api/requests/semantic_tokens.rs Outdated Show resolved Hide resolved

dhruvmanila reviewed Jul 3, 2025

View reviewed changes

AlexWaygood added server Related to the LSP server ty Multi-file analysis & type inference labels Jul 3, 2025

dhruvmanila requested changes Jul 3, 2025

View reviewed changes

AlexWaygood removed their request for review July 3, 2025 10:54

UnboundVariable added 13 commits July 4, 2025 10:48

Changed token modifiers from a vector to bit flags to avoid extra mem…

a4e1e0e

…ory allocation.

Changed semantic token function to accept Option<TextRange> where `…

b5ad248

…None` means "entire document".

Simplified semantic_tokens function to return SemanticTokens rath…

5541bcc

…er than `Option<SemanticTokens>`.

Changed SemanticTokens to implement Deref trait and avoid exposin…

4f4cef6

…g `tokens` as a public field.

Abstracted out visit_parameters and added handling of lambda parame…

b7e3414

…ters.

Eliminated the need to sort tokens.

816cf9e

Incorporated code review feedback.

720163c

Added support for concatenated string literals and bytes string liter…

98f356f

…als.

Removed unnecessary use of raw string literals.

7694654

Refactored tests to use a helper function.

7fb25e5

Improved tests.

108a201

Moved semantic token range request handler to its own file.

d58d1f7

Added Ranged trait to SemanticToken.

8065af4

UnboundVariable and others added 5 commits July 4, 2025 14:03

Fixed offset encoding logic.

83cb322

Formatting update

a728999

Moved visitor functions as per code review feedback.

41ed581

Merge branch 'main' into semantic_tokens

6fd0d9b

Fixed merge-related issues.

b50a4f1

dhruvmanila approved these changes Jul 7, 2025

View reviewed changes

MichaReiser approved these changes Jul 7, 2025

View reviewed changes

UnboundVariable added 10 commits July 7, 2025 11:19

More changes from code review feedback.

2301ea2

More code review feedback.

33bf244

Switched to bitflags crate for tracking modifiers.

77be4e2

More code review changes.

f735ebf

Incorporated more code review feedback.

540413f

More code review feedback.

473a57d

Fixed length and offset encoding logic to honor requested encoding an…

e8483e0

…d the multilineTokenSupport capability of the client.

Converted all semantic token tests to snapshots.

f0efb01

Improved classification of parameters.

4264579

Fixed formatting

3be32e1

UnboundVariable merged commit 278f930 into astral-sh:main Jul 7, 2025
37 checks passed

UnboundVariable deleted the semantic_tokens branch July 7, 2025 22:34

MichaReiser mentioned this pull request Jul 8, 2025

Add semantic syntax highlighting to playground astral-sh/ty#776

Closed

Comments

Conversation

UnboundVariable commented Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

mypy_primer results

Uh oh!

codspeed-hq bot commented Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CodSpeed WallTime Performance Report

Merging #19108 will not alter performance

Summary

Uh oh!

carljm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dhruvmanila left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dhruvmanila left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

UnboundVariable commented Jul 4, 2025

Uh oh!

dhruvmanila left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MichaReiser commented Jul 7, 2025

Uh oh!

MichaReiser commented Jul 7, 2025

Uh oh!

github-actions bot commented Jul 7, 2025

ruff-ecosystem results

Linter (stable)

Linter (preview)

Formatter (stable)

Formatter (preview)

Uh oh!

Uh oh!

UnboundVariable commented Jul 2, 2025 •

edited

Loading

github-actions bot commented Jul 2, 2025 •

edited

Loading

`mypy_primer` results

codspeed-hq bot commented Jul 2, 2025 •

edited

Loading

dhruvmanila left a comment •

edited

Loading

dhruvmanila left a comment •

edited

Loading

`ruff-ecosystem` results