Skip to content

Commit fd1399f

Browse files
roomoteroomote
committed
fix(telemetry): stop code-index error retry storm and hash PostHog distinct IDs (#830)
- Add a per-workspace circuit breaker to the code-index file watcher: after 3 consecutive failed batches it drops further file-system events until the watcher is recreated (embedder/vector-store config change), stopping the retry loop behind the 254k-event Code Index Error spike. - Remove the two per-call captureToolUsage sites re-added by #1073 so tool usage reports only as the aggregated toolsUsed breakdown on Task Completed; drop the dead captureToolUsage/captureConversationMessage methods from TelemetryService. - Salt-hash the PostHog distinct ID (SHA-256 + fixed salt) in PostHogTelemetryClient and before posting machineId to the webview, so the raw VS Code machine ID never leaves the extension host. - Update PRIVACY.md: hashed machine-ID wording and a clarification that the Cloud sync pipeline is separate from PostHog telemetry (issue #830).
1 parent 4d1a543 commit fd1399f

17 files changed

Lines changed: 334 additions & 90 deletions

‎PRIVACY.md‎

Lines changed: 11 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -33,13 +33,17 @@ go—and, importantly, where they don't.
3333
model), it is stored locally on your device and never sent to us or any third
3434
party, except the provider you have chosen.
3535
- **Telemetry (Usage Data)**: We collect feature usage and error data to help us
36-
improve Zoo Code. This telemetry is powered by PostHog and includes your VS
37-
Code machine ID, feature usage patterns, and exception reports. The VS Code
38-
machine ID is a persistent identifier and may be considered personal data in
39-
some jurisdictions; we use it only for product analytics and error grouping.
40-
PostHog event data is retained for 12 months under our current project
41-
configuration. This PostHog-based telemetry does **not** collect your code or AI
42-
prompts. Telemetry is on by default. To turn it off, choose "Disabled" in
36+
improve Zoo Code. This telemetry is powered by PostHog and includes a salted,
37+
hashed (pseudonymized) form of your VS Code machine ID, feature usage
38+
patterns, and exception reports. The raw VS Code machine ID never leaves your
39+
device; we use the hashed identifier only for product analytics and error
40+
grouping. PostHog event data is retained for 12 months under our current
41+
project configuration. This PostHog-based telemetry does **not** collect your
42+
code or AI prompts. (Separately, if you sign in to Zoo Code Cloud and enable
43+
Cloud sync, the Cloud sync pipeline — which is distinct from this PostHog
44+
telemetry — does transmit task messages so they can be shared or synced
45+
through your Cloud account; that feature is controlled by its own Cloud sync
46+
settings.) Telemetry is on by default. To turn it off, choose "Disabled" in
4347
the settings or in the notice shown on first use. The notice remains until
4448
you explicitly Accept or Decline. Telemetry also follows VS Code's own
4549
global telemetry setting: if you turn that off, Zoo Code stops sending

‎packages/telemetry/src/PostHogTelemetryClient.ts‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ import {
1515
} from "@roo-code/types"
1616

1717
import { BaseTelemetryClient } from "./BaseTelemetryClient"
18+
import { hashDistinctId } from "./distinct-id"
1819

1920
/**
2021
* PostHogTelemetryClient handles telemetry event tracking for the Roo Code extension.
@@ -23,7 +24,9 @@ import { BaseTelemetryClient } from "./BaseTelemetryClient"
2324
*/
2425
export class PostHogTelemetryClient extends BaseTelemetryClient {
2526
private client: PostHog
26-
private distinctId: string = vscode.env.machineId
27+
// Pseudonymous distinct ID: salted SHA-256 of the machine ID, never the raw
28+
// value (issue #830).
29+
private distinctId: string = hashDistinctId(vscode.env.machineId)
2730
// Git repository properties that should be filtered out
2831
private readonly gitPropertyNames = ["repositoryUrl", "repositoryName", "defaultBranch"]
2932

‎packages/telemetry/src/TelemetryService.ts‎

Lines changed: 0 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -211,10 +211,6 @@ export class TelemetryService {
211211
})
212212
}
213213

214-
public captureConversationMessage(taskId: string, source: "user" | "assistant"): void {
215-
this.captureEvent(TelemetryEventName.TASK_CONVERSATION_MESSAGE, { taskId, source })
216-
}
217-
218214
public captureLlmCompletion(
219215
taskId: string,
220216
properties: {
@@ -232,10 +228,6 @@ export class TelemetryService {
232228
this.captureEvent(TelemetryEventName.MODE_SWITCH, { taskId, newMode })
233229
}
234230

235-
public captureToolUsage(taskId: string, tool: string): void {
236-
this.captureEvent(TelemetryEventName.TOOL_USED, { taskId, tool })
237-
}
238-
239231
public captureCheckpointCreated(taskId: string): void {
240232
this.captureEvent(TelemetryEventName.CHECKPOINT_CREATED, { taskId })
241233
}

‎packages/telemetry/src/__tests__/PostHogTelemetryClient.test.ts‎

Lines changed: 33 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,11 @@ import { PostHog } from "posthog-node"
88
import { type TelemetryPropertiesProvider, TelemetryEventName, ApiProviderError } from "@roo-code/types"
99

1010
import { PostHogTelemetryClient } from "../PostHogTelemetryClient"
11+
import { hashDistinctId } from "../distinct-id"
12+
13+
// The client must use the salted hash as its PostHog distinct ID, never the raw
14+
// VS Code machine ID (issue #830).
15+
const hashedMachineId = hashDistinctId("test-machine-id")
1116

1217
vi.mock("posthog-node")
1318

@@ -277,7 +282,7 @@ describe("PostHogTelemetryClient", () => {
277282
})
278283

279284
expect(mockPostHogClient.capture).toHaveBeenCalledWith({
280-
distinctId: "test-machine-id",
285+
distinctId: hashedMachineId,
281286
event: TelemetryEventName.TASK_CREATED,
282287
properties: expect.objectContaining({
283288
appVersion: "1.0.0",
@@ -286,6 +291,22 @@ describe("PostHogTelemetryClient", () => {
286291
})
287292
})
288293

294+
it("never uses the raw VS Code machine ID as the PostHog distinct ID", async () => {
295+
const client = new PostHogTelemetryClient()
296+
client.updateTelemetryState(true)
297+
298+
await client.capture({
299+
event: TelemetryEventName.TASK_CREATED,
300+
properties: { test: "value" },
301+
})
302+
303+
expect(mockPostHogClient.capture).toHaveBeenCalledTimes(1)
304+
const captureCall = mockPostHogClient.capture.mock.calls[0][0]
305+
expect(captureCall.distinctId).toBe(hashedMachineId)
306+
expect(captureCall.distinctId).not.toBe("test-machine-id")
307+
expect(captureCall.distinctId).not.toContain("test-machine-id")
308+
})
309+
289310
it("should filter out git repository properties when capturing events", async () => {
290311
const client = new PostHogTelemetryClient()
291312
client.updateTelemetryState(true)
@@ -313,7 +334,7 @@ describe("PostHogTelemetryClient", () => {
313334
})
314335

315336
expect(mockPostHogClient.capture).toHaveBeenCalledWith({
316-
distinctId: "test-machine-id",
337+
distinctId: hashedMachineId,
317338
event: TelemetryEventName.TASK_CREATED,
318339
properties: expect.objectContaining({
319340
appVersion: "1.0.0",
@@ -402,7 +423,7 @@ describe("PostHogTelemetryClient", () => {
402423

403424
expect(mockPostHogClient.captureException).toHaveBeenCalledWith(
404425
error,
405-
"test-machine-id",
426+
hashedMachineId,
406427
expect.objectContaining({
407428
customProp: "value",
408429
$app_version: "1.0.0",
@@ -417,7 +438,7 @@ describe("PostHogTelemetryClient", () => {
417438
const error = new Error("Test error")
418439
await client.captureException(error)
419440

420-
expect(mockPostHogClient.captureException).toHaveBeenCalledWith(error, "test-machine-id", {
441+
expect(mockPostHogClient.captureException).toHaveBeenCalledWith(error, hashedMachineId, {
421442
$app_version: undefined,
422443
})
423444
})
@@ -485,7 +506,7 @@ describe("PostHogTelemetryClient", () => {
485506
const error = new Error("Internal server error")
486507
client.captureException(error)
487508

488-
expect(mockPostHogClient.captureException).toHaveBeenCalledWith(error, "test-machine-id", {
509+
expect(mockPostHogClient.captureException).toHaveBeenCalledWith(error, hashedMachineId, {
489510
$app_version: undefined,
490511
})
491512
})
@@ -497,7 +518,7 @@ describe("PostHogTelemetryClient", () => {
497518
const error = Object.assign(new Error("Internal server error"), { status: 500 })
498519
client.captureException(error)
499520

500-
expect(mockPostHogClient.captureException).toHaveBeenCalledWith(error, "test-machine-id", {
521+
expect(mockPostHogClient.captureException).toHaveBeenCalledWith(error, hashedMachineId, {
501522
$app_version: undefined,
502523
})
503524
})
@@ -537,7 +558,7 @@ describe("PostHogTelemetryClient", () => {
537558

538559
// The implementation overrides error.message with the extracted message from getErrorMessage
539560
expect(error.message).toBe("Upstream provider error: model overloaded")
540-
expect(mockPostHogClient.captureException).toHaveBeenCalledWith(error, "test-machine-id", {
561+
expect(mockPostHogClient.captureException).toHaveBeenCalledWith(error, hashedMachineId, {
541562
$app_version: undefined,
542563
})
543564
})
@@ -558,7 +579,7 @@ describe("PostHogTelemetryClient", () => {
558579

559580
// The implementation overrides error.message with the extracted message from getErrorMessage
560581
expect(error.message).toBe("Upstream provider: connection timeout")
561-
expect(mockPostHogClient.captureException).toHaveBeenCalledWith(error, "test-machine-id", {
582+
expect(mockPostHogClient.captureException).toHaveBeenCalledWith(error, hashedMachineId, {
562583
$app_version: undefined,
563584
})
564585
})
@@ -576,7 +597,7 @@ describe("PostHogTelemetryClient", () => {
576597

577598
// Verify error message remains the primary message
578599
expect(error.message).toBe("Primary error message")
579-
expect(mockPostHogClient.captureException).toHaveBeenCalledWith(error, "test-machine-id", {
600+
expect(mockPostHogClient.captureException).toHaveBeenCalledWith(error, hashedMachineId, {
580601
$app_version: undefined,
581602
})
582603
})
@@ -589,7 +610,7 @@ describe("PostHogTelemetryClient", () => {
589610
client.captureException(error)
590611

591612
// The implementation auto-extracts properties from ApiProviderError
592-
expect(mockPostHogClient.captureException).toHaveBeenCalledWith(error, "test-machine-id", {
613+
expect(mockPostHogClient.captureException).toHaveBeenCalledWith(error, hashedMachineId, {
593614
provider: "OpenRouter",
594615
modelId: "gpt-4",
595616
operation: "createMessage",
@@ -606,7 +627,7 @@ describe("PostHogTelemetryClient", () => {
606627
client.captureException(error, { customProperty: "value" })
607628

608629
// additionalProperties take precedence over auto-extracted properties
609-
expect(mockPostHogClient.captureException).toHaveBeenCalledWith(error, "test-machine-id", {
630+
expect(mockPostHogClient.captureException).toHaveBeenCalledWith(error, hashedMachineId, {
610631
provider: "OpenRouter",
611632
modelId: "gpt-4",
612633
operation: "createMessage",
@@ -622,7 +643,7 @@ describe("PostHogTelemetryClient", () => {
622643
const error = new Error("Regular error")
623644
client.captureException(error, { customProperty: "value" })
624645

625-
expect(mockPostHogClient.captureException).toHaveBeenCalledWith(error, "test-machine-id", {
646+
expect(mockPostHogClient.captureException).toHaveBeenCalledWith(error, hashedMachineId, {
626647
customProperty: "value",
627648
$app_version: undefined,
628649
})

‎packages/telemetry/src/distinct-id.ts‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
import { createHash } from "crypto"
2+
3+
/**
4+
* Fixed, non-secret application salt. The raw VS Code machine ID is a stable,
5+
* device-scoped identifier that may be considered personal data; combining it
6+
* with this salt under SHA-256 yields a pseudonymous distinct ID that is still
7+
* stable per device, so PostHog analytics and error grouping keep working while
8+
* the raw identifier never leaves the extension host.
9+
*/
10+
const DISTINCT_ID_SALT = "zoo-code-telemetry-distinct-id-v1"
11+
12+
export function hashDistinctId(machineId: string): string {
13+
return createHash("sha256").update(`${DISTINCT_ID_SALT}:${machineId}`).digest("hex")
14+
}

‎packages/telemetry/src/index.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
11
export * from "./BaseTelemetryClient"
2+
export * from "./distinct-id"
23
export * from "./PostHogTelemetryClient"
34
export * from "./TelemetryService"

‎packages/types/src/vscode-extension-host.ts‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -389,6 +389,10 @@ export type ExtensionState = Pick<
389389
cwd?: string // Current working directory
390390
telemetrySetting: TelemetrySetting
391391
telemetryKey?: string
392+
// Salted SHA-256 hash of vscode.env.machineId (see hashDistinctId in
393+
// @roo-code/telemetry), not the raw machine ID: the webview uses this value as
394+
// its PostHog distinct ID, so the raw identifier never crosses the webview
395+
// boundary (issue #830).
392396
machineId?: string
393397
// Live vscode.env.isTelemetryEnabled, so the webview's own PostHog client can respect
394398
// the VS Code global telemetry toggle the same way the extension-side gate does --

‎src/core/assistant-message/__tests__/presentAssistantMessage-tool-usage-attribution.spec.ts‎

Lines changed: 0 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,6 @@ vi.mock("@roo-code/core", () => ({
4141
vi.mock("@roo-code/telemetry", () => ({
4242
TelemetryService: {
4343
instance: {
44-
captureToolUsage: vi.fn(),
4544
captureConsecutiveMistakeError: vi.fn(),
4645
captureEvent: vi.fn(),
4746
},
@@ -167,8 +166,6 @@ describe("presentAssistantMessage - tool usage attribution", () => {
167166

168167
expect(mockTask.recordToolUsage).toHaveBeenCalledTimes(1)
169168
expect(mockTask.recordToolUsage).toHaveBeenCalledWith("read_file")
170-
expect(TelemetryService.instance.captureToolUsage).toHaveBeenCalledTimes(1)
171-
expect(TelemetryService.instance.captureToolUsage).toHaveBeenCalledWith(mockTask.taskId, "read_file")
172169
})
173170

174171
it("records a valid dynamic mcp_ tool name as use_mcp_tool", async () => {
@@ -186,7 +183,6 @@ describe("presentAssistantMessage - tool usage attribution", () => {
186183
await presentAssistantMessage(mockTask as unknown as Task, baseSnapshot)
187184

188185
expect(mockTask.recordToolUsage).toHaveBeenCalledWith("use_mcp_tool")
189-
expect(TelemetryService.instance.captureToolUsage).toHaveBeenCalledWith(mockTask.taskId, "use_mcp_tool")
190186
})
191187

192188
it("records a malformed mcp_ tool name as use_mcp_tool, not the raw name", async () => {
@@ -358,8 +354,6 @@ describe("presentAssistantMessage - tool usage attribution", () => {
358354

359355
expect(mockTask.recordToolUsage).toHaveBeenCalledTimes(1)
360356
expect(mockTask.recordToolUsage).toHaveBeenCalledWith("use_mcp_tool")
361-
expect(TelemetryService.instance.captureToolUsage).toHaveBeenCalledTimes(1)
362-
expect(TelemetryService.instance.captureToolUsage).toHaveBeenCalledWith(mockTask.taskId, "use_mcp_tool")
363357
})
364358

365359
it("records no attempt when the MCP server is not on the mode's allow-list", async () => {
@@ -401,7 +395,6 @@ describe("presentAssistantMessage - tool usage attribution", () => {
401395
// The server is disallowed, so the call never reaches onValidated:
402396
// no success attempt is recorded for a call that was never permitted to execute.
403397
expect(mockTask.recordToolUsage).not.toHaveBeenCalled()
404-
expect(TelemetryService.instance.captureToolUsage).not.toHaveBeenCalled()
405398
})
406399

407400
it("rejects at dispatch when use_mcp_tool is in disabledTools, without invoking the server method", async () => {
@@ -466,7 +459,6 @@ describe("presentAssistantMessage - tool usage attribution", () => {
466459

467460
// No success attribution on rejection; a safe static failure key instead.
468461
expect(mockTask.recordToolUsage).not.toHaveBeenCalled()
469-
expect(TelemetryService.instance.captureToolUsage).not.toHaveBeenCalled()
470462
expect(mockTask.recordToolError).toHaveBeenCalledWith("use_mcp_tool", expect.any(String))
471463
expect(mockTask.consecutiveMistakeCount).toBe(1)
472464
// The stream must not be interrupted by a validation rejection.
@@ -532,7 +524,6 @@ describe("presentAssistantMessage - tool usage attribution", () => {
532524
expect(mockHub.callTool).not.toHaveBeenCalled()
533525

534526
expect(mockTask.recordToolUsage).not.toHaveBeenCalled()
535-
expect(TelemetryService.instance.captureToolUsage).not.toHaveBeenCalled()
536527
expect(mockTask.recordToolError).toHaveBeenCalledWith("use_mcp_tool", expect.any(String))
537528
expect(mockTask.consecutiveMistakeCount).toBe(1)
538529
expect(mockTask.didAlreadyUseTool).toBe(false)
@@ -620,7 +611,6 @@ describe("presentAssistantMessage - tool usage attribution", () => {
620611
)
621612

622613
expect(mockTask.recordToolUsage).not.toHaveBeenCalled()
623-
expect(TelemetryService.instance.captureToolUsage).not.toHaveBeenCalled()
624614
expect(mockTask.recordToolError).toHaveBeenCalledWith("use_mcp_tool", expect.any(String))
625615
expect(mockTask.consecutiveMistakeCount).toBe(1)
626616
expect(mockTask.didAlreadyUseTool).toBe(false)

‎src/core/assistant-message/presentAssistantMessage.ts‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -419,7 +419,6 @@ async function presentAssistantMessageBlock(cline: Task, policySnapshot: Request
419419
? undefined
420420
: () => {
421421
cline.recordToolUsage("use_mcp_tool")
422-
TelemetryService.instance.captureToolUsage(cline.taskId, "use_mcp_tool")
423422
},
424423
})
425424
break
@@ -802,10 +801,11 @@ async function presentAssistantMessageBlock(cline: Task, policySnapshot: Request
802801

803802
// Validation passed: record exactly one attempt at this single
804803
// central point. Individual tool handlers must not also record
805-
// usage, or the attempt would be double-counted.
804+
// usage, or the attempt would be double-counted. Usage reaches
805+
// telemetry as a per-task toolsUsed breakdown on Task Completed
806+
// (flushTelemetryInstallment), never as one event per tool call.
806807
const recordName = toTelemetryToolName(block.name, isCustomTool, stateExperiments)
807808
cline.recordToolUsage(recordName)
808-
TelemetryService.instance.captureToolUsage(cline.taskId, recordName)
809809

810810
// Track legacy format usage for read_file tool (for migration monitoring).
811811
// Model ids on telemetry events are non-policy labels: a live

‎src/core/webview/ClineProvider.ts‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -84,7 +84,7 @@ import {
8484
type TaskExecutionContext,
8585
} from "../task/providerHandoff"
8686
import { aggregateTaskCostsRecursive, type AggregatedCosts } from "./aggregateTaskCosts"
87-
import { TelemetryService } from "@roo-code/telemetry"
87+
import { TelemetryService, hashDistinctId } from "@roo-code/telemetry"
8888
import { CloudService, getRooCodeApiUrl } from "@roo-code/cloud"
8989

9090
import { Package } from "../../shared/package"
@@ -3225,7 +3225,9 @@ export class ClineProvider
32253225
}
32263226

32273227
const telemetryKey = process.env.POSTHOG_API_KEY
3228-
const machineId = vscode.env.machineId
3228+
// The webview uses this value as its PostHog distinct ID, so it must be the
3229+
// salted hash: the raw machine ID never crosses the webview boundary (issue #830).
3230+
const machineId = hashDistinctId(vscode.env.machineId)
32293231
const vscodeTelemetryEnabled = vscode.env.isTelemetryEnabled
32303232
const mergedAllowedCommands = this.mergeAllowedCommands(allowedCommands)
32313233
const mergedDeniedCommands = this.mergeDeniedCommands(deniedCommands)

0 commit comments

Comments
 (0)