Skip to content

Fix prompttypes measure#2799

Merged
isidorn merged 3 commits intomicrosoft:mainfrom
kevin-m-kent:kevin-m-kent/fix-prompttypes-measure
Jan 20, 2026
Merged

Fix prompttypes measure#2799
isidorn merged 3 commits intomicrosoft:mainfrom
kevin-m-kent:kevin-m-kent/fix-prompttypes-measure

Conversation

@kevin-m-kent
Copy link
Contributor

@kevin-m-kent kevin-m-kent commented Jan 12, 2026 •
edited
Loading

It appears that our Measures.prompttypes is capturing the array length rather than the total character length for each prompt object. It should capture characters. This PR intends to fix that.

@kevin-m-kent kevin-m-kent mentioned this pull request Jan 12, 2026
Closed
@isidorn
Copy link
Contributor

isidorn commented Jan 13, 2026

Thanks for the PR, assigning first to @lramos15 to get his thoughts

cc @roblourens

@roblourens
Copy link
Member

roblourens commented Jan 15, 2026

LGTM

@isidorn
Copy link
Contributor

isidorn commented Jan 19, 2026

@kevin-m-kent looks like tests are failing - can you take a look?

@kevin-m-kent
Copy link
Contributor Author

kevin-m-kent commented Jan 19, 2026 •
edited
Loading

@isidorn if I'm reading these test cases correctly, it looks like they might be setup with the wrong expectation? It's expecting 1:1 and it outputs 1:12 now.

The actual input is "hello world!", which is 12 characters, so I think the test case needs to be updated. I will push to the PR to do that.

FAIL  src/extension/prompt/node/test/defaultIntentRequestHandler.spec.ts > defaultIntentRequestHandler > makes a tool call turn
Error: Snapshot `defaultIntentRequestHandler > makes a tool call turn 2` mismatched

- Expected
+ Received

@@ -176,11 +176,11 @@
          "contextTypes": "none",
          "conversationId": "some-session-id",
          "isParticipantDetected": "false",
          "languageId": undefined,
          "model": "gpt-4.1-2025-04-14",
-         "promptTypes": "1:1",
+         "promptTypes": "1:12",
          "requestId": "turn-id-0",
          "responseId": "turn-id-0",
          "responseType": "success",
          "toolCounts": "{"my_func":1}",
        },
@@ -411,11 +411,11 @@
          "contextTypes": "none",
          "conversationId": "some-session-id",
          "isParticipantDetected": "false",
          "languageId": undefined,
          "model": "gpt-4.1-2025-04-14",
-         "promptTypes": "1:1",
+         "promptTypes": "1:12",
          "requestId": "turn-id-0",
          "responseId": "turn-id-0",
          "responseType": "success",
          "toolCounts": "{}",
        },

@kevin-m-kent
Copy link
Contributor Author

kevin-m-kent commented Jan 19, 2026 •
edited
Loading

I pushed the change above. Let's see if it fixes the unit tests now (this makes the expectation to be 1:12 instead of 1:1)

@kevin-m-kent kevin-m-kent force-pushed the kevin-m-kent/fix-prompttypes-measure branch from bc2aa46 to 0688ed8 Compare January 19, 2026 15:41
@kevin-m-kent kevin-m-kent force-pushed the kevin-m-kent/fix-prompttypes-measure branch from 0688ed8 to b0a7815 Compare January 19, 2026 15:48
@kevin-m-kent kevin-m-kent marked this pull request as ready for review January 19, 2026 16:07
Copilot AI review requested due to automatic review settings January 19, 2026 16:07
@kevin-m-kent
Copy link
Contributor Author

kevin-m-kent commented Jan 19, 2026 •
edited
Loading

@isidorn / @roblourens / @lramos15 I think we should be good now but please take a look.

Copilot AI reviewed Jan 19, 2026
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a telemetry bug where the promptTypes measurement was incorrectly capturing the array length of message content parts instead of the total character count of the text content.

Changes:

  • Fixed promptTypes telemetry to use getTextPart() helper for accurate character counting
  • Updated test snapshots to reflect the corrected telemetry values (from 1:1 to 1:12 for "hello world!")

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
src/extension/prompt/node/chatParticipantTelemetry.ts Added getTextPart import and updated both PanelChatTelemetry and InlineChatTelemetry to use getTextPart(msg.content).length for accurate character counting
src/extension/prompt/node/test/snapshots/defaultIntentRequestHandler.spec.ts.snap Updated test snapshots to reflect corrected promptTypes values showing character count (12) instead of array length (1)

@vs-code-engineering vs-code-engineering bot added this to the January 2026 milestone Jan 19, 2026
@isidorn isidorn added this pull request to the merge queue Jan 20, 2026
Merged via the queue into microsoft:main with commit 058f629 Jan 20, 2026
14 of 15 checks passed
@kevin-m-kent
Copy link
Contributor Author

kevin-m-kent commented Jan 24, 2026

Validated, the data looks good. Example:

query

	"prompttypes": "0:33747,1:2566,1:21664,1:13492,2:1028,3:127,2:56,3:1936,2:172,3:1822,2:1570,1:10831,2:0,3:208,2:36,3:208,2:1546,1:11758,2:0,3:119,2:687,1:2682,2:134,3:120,2:378,1:11412,2:0,3:39284,2:65,3:76253,2:1238,3:2077,2:0,3:2996,2:87,3:5174,2:480,3:120,2:952,1:2289,2:87,3:6913,2:51,3:129,2:70,3:547,2:120,3:1542,2:94,3:1806,2:0,3:123,2:40,3:3832,2:106,3:123,2:0,3:143,2:49,3:123,2:31,3:4033,2:72,3:410,2:42,3:4020,2:0,3:123"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@roblourens roblourens roblourens approved these changes

@aeschli aeschli aeschli approved these changes

Assignees

@lramos15 lramos15

Labels

None yet

Projects

None yet

Milestone

January 2026

Development

Successfully merging this pull request may close these issues.

6 participants

@kevin-m-kent @isidorn @roblourens @aeschli @lramos15