feat: support Cursor CSV token imports by Miss-you · Pull Request #14 · Miss-you/codetok

Miss-you · 2026-03-12T17:25:21Z

Summary

add a cursor provider that parses imported Cursor dashboard CSV exports from disk
wire Cursor into daily and session with --cursor-dir plus e2e coverage
document that Cursor Tab token usage is unsupported because the export does not provide a defensible Tab token split

Validation

make fmt
make vet
make test
make build
Built-binary smoke checks for daily and session against e2e/testdata/cursor
make lint skipped because golangci-lint is not installed in this environment

Notes

Cursor support is local-file based only in this change. No remote Cursor API calls were added.

Copilot

Pull request overview

Adds first-class Cursor support to codetok by importing and parsing Cursor dashboard usage export CSVs as session-like records, wiring the provider into the CLI, and extending unit/e2e coverage and docs accordingly.

Changes:

Introduce a new provider/cursor implementation that scans a directory for .csv exports and maps Cursor token columns into provider.TokenUsage.
Wire Cursor into daily/session via blank imports and a new --cursor-dir override flag, plus e2e fixtures/tests.
Update English/Chinese READMEs and add an implementation plan doc for Cursor token support.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
README.md	Documents Cursor as a supported provider, adds `--cursor-dir` flag and Cursor parsing details.
README_zh.md	Chinese documentation updates mirroring README.md (Cursor support + flags + limitations).
provider/cursor/parser.go	New Cursor provider: scans for CSVs and converts rows into `SessionInfo` records.
provider/cursor/parser_test.go	Unit tests covering CSV parsing, malformed row skipping, and multi-file collection.
provider/cursor/testdata/usage-export.csv	Unit test fixture CSV for Cursor exports.
cmd/session.go	Adds Cursor provider import and `--cursor-dir` flag for `session`.
cmd/daily.go	Adds Cursor provider import and `--cursor-dir` flag for `daily`.
cmd/daily_test.go	Updates test command scaffolding to include the new `cursor-dir` flag.
e2e/e2e_test.go	Adds e2e coverage validating Cursor shows up in JSON and dashboard outputs.
e2e/testdata/cursor/usage-export.csv	E2E fixture CSV for Cursor exports.
docs/plans/2026-03-13-cursor-token-implementation.md	Implementation plan for adding Cursor token support.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-12T17:30:41Z

 - **Kimi CLI** — parses `~/.kimi/sessions/**/wire.jsonl`
 - **Claude Code** — parses `~/.claude/projects/**/*.jsonl` (with streaming deduplication)
 - **Codex CLI** — parses `~/.codex/sessions/**/*.jsonl`
+- **Cursor** — parses imported Cursor usage export CSVs from `~/.codetok/cursor/**/*.csv`


The supported-provider path uses ~/.codetok/cursor/**/*.csv, but later the README documents Cursor as ~/.codetok/cursor/*.csv. Since the Cursor provider walks the directory recursively (WalkDir), it would be clearer to keep the documented glob consistent (and reflect recursion) in all sections.

Made the Cursor README paths consistent around ~/.codetok/cursor/**/*.csv so the docs reflect recursive discovery in all sections.

Copilot · 2026-03-12T17:30:41Z

 - Parses `event_msg` events with `payload.type="token_count"`
 - Takes the last (cumulative) token count per session

+**Cursor** — `~/.codetok/cursor/*.csv`


This section documents Cursor CSVs as ~/.codetok/cursor/*.csv, but the Cursor provider scans recursively for .csv files under the directory. Consider updating the path/glob here to match the supported-provider section and the actual recursive behavior.

Suggested change

**Cursor** — `~/.codetok/cursor/*.csv`

**Cursor** — `~/.codetok/cursor/**/*.csv`

Updated this section to ~/.codetok/cursor/**/*.csv so it matches the recursive file scan.

Copilot · 2026-03-12T17:30:41Z

 ## How It Works

-codetok reads local session data that AI coding CLIs store on disk. Each provider has its own parser that understands the tool's data format. All session files are parsed in parallel using bounded goroutines (default: `min(NumCPU, 8)`, configurable via `CODETOK_WORKERS` env var).
+codetok reads local session data and imported usage exports stored on disk. Each provider has its own parser that understands the tool's data format. All session files are parsed in parallel using bounded goroutines (default: `min(NumCPU, 8)`, configurable via `CODETOK_WORKERS` env var).


The README claims “All session files are parsed in parallel”, but the new Cursor provider currently parses CSV files sequentially (it does not use provider.ParseParallel). Either adjust this sentence to exclude Cursor imports, or update the Cursor provider to parse files in parallel to keep the statement accurate.

Suggested change

codetok reads local session data and imported usage exports stored on disk. Each provider has its own parser that understands the tool's data format. All session files are parsed in parallel using bounded goroutines (default: `min(NumCPU, 8)`, configurable via `CODETOK_WORKERS` env var).

codetok reads local session data and imported usage exports stored on disk. Each provider has its own parser that understands the tool's data format. All supported session log files are parsed in parallel using bounded goroutines (default: `min(NumCPU, 8)`, configurable via `CODETOK_WORKERS` env var); Cursor CSV imports are processed sequentially.

Adjusted the README wording to distinguish parallel JSONL parsing from sequential Cursor CSV parsing, which matches the current implementation.

Copilot · 2026-03-12T17:30:42Z

 - **Kimi CLI** — 解析 `~/.kimi/sessions/**/wire.jsonl`
 - **Claude Code** — 解析 `~/.claude/projects/**/*.jsonl`（含流式去重）
 - **Codex CLI** — 解析 `~/.codex/sessions/**/*.jsonl`
+- **Cursor** — 解析 `~/.codetok/cursor/**/*.csv` 下导入的 Cursor 用量导出文件


文档上方使用了 ~/.codetok/cursor/**/*.csv，但后面 Cursor 部分写的是 ~/.codetok/cursor/*.csv。由于 Cursor provider 实际上会递归扫描目录（WalkDir），建议统一这里的路径/通配符描述，并明确支持子目录。

Suggested change

- **Cursor** — 解析 `~/.codetok/cursor/**/*.csv` 下导入的 Cursor 用量导出文件

- **Cursor** — 解析 `~/.codetok/cursor` 目录下（含子目录）的所有 `.csv` Cursor 用量导出文件

已统一 README 里的 Cursor 路径描述，保持为 ~/.codetok/cursor/**/*.csv，与递归扫描行为一致。

Copilot · 2026-03-12T17:30:42Z

 - 解析 `event_msg` 事件中 `payload.type="token_count"` 的记录
 - 取最后一条累计 token 计数

+**Cursor** — `~/.codetok/cursor/*.csv`


这里写的是 ~/.codetok/cursor/*.csv，但 Cursor provider 实际会递归扫描目录下所有 .csv 文件。建议把这里的路径/通配符改成与上方一致，并反映递归行为。

Suggested change

**Cursor** — `~/.codetok/cursor/*.csv`

**Cursor** — `~/.codetok/cursor/**/*.csv` 下的所有 CSV 文件

这里已改成 ~/.codetok/cursor/**/*.csv，现在和上方说明及实际扫描行为一致。

Copilot · 2026-03-12T17:30:42Z

 ## 工作原理

-codetok 读取 AI 编程 CLI 工具存储在本地磁盘的会话数据。每个 Provider 有独立的解析器，理解对应工具的数据格式。所有会话文件通过有界 goroutine 并行解析（默认：`min(NumCPU, 8)`，可通过 `CODETOK_WORKERS` 环境变量配置）。
+codetok 读取本地磁盘上的会话数据和手动导入的用量导出文件。每个 Provider 有独立的解析器，理解对应工具的数据格式。所有会话文件通过有界 goroutine 并行解析（默认：`min(NumCPU, 8)`，可通过 `CODETOK_WORKERS` 环境变量配置）。


这里写“所有会话文件通过有界 goroutine 并行解析”，但 Cursor provider 目前是顺序解析 CSV（未使用 provider.ParseParallel）。建议要么修正文案（说明仅部分 provider 并行），要么让 Cursor 解析也并行以保持描述准确。

Suggested change

codetok 读取本地磁盘上的会话数据和手动导入的用量导出文件。每个 Provider 有独立的解析器，理解对应工具的数据格式。所有会话文件通过有界 goroutine 并行解析（默认：`min(NumCPU, 8)`，可通过 `CODETOK_WORKERS` 环境变量配置）。

codetok 读取本地磁盘上的会话数据和手动导入的用量导出文件。每个 Provider 有独立的解析器，理解对应工具的数据格式。Kimi / Claude Code / Codex 等 JSONL 会话文件通过有界 goroutine 并行解析（默认：`min(NumCPU, 8)`，可通过 `CODETOK_WORKERS` 环境变量配置），Cursor 的 CSV 用量导出文件目前按文件顺序解析。

文案已区分 JSONL 的并行解析和 Cursor CSV 的顺序解析，现在与实现一致。

Copilot · 2026-03-12T17:30:43Z

+	reader := csv.NewReader(f)
+	records, err := reader.ReadAll()
+	if err != nil {
+		return nil, err
+	}


csv.Reader.ReadAll() loads the entire CSV into memory and will fail the whole file on the first CSV parsing error (e.g., an inconsistent field count), which undermines the intent to “skip malformed rows” and can be costly for large exports. Consider streaming row-by-row (Read() loop) and handling/continuing on per-row errors; if you want to tolerate inconsistent columns, configure the reader accordingly (e.g., allow variable fields) so one bad row doesn’t drop the entire file.

feat: support Cursor CSV token imports

01e118e

Copilot AI review requested due to automatic review settings March 12, 2026 17:25

Copilot started reviewing on behalf of Miss-you March 12, 2026 17:26 View session

Copilot AI reviewed Mar 12, 2026

View reviewed changes

fix: harden cursor csv parsing

36ff045

Miss-you merged commit 6c2ee88 into main Mar 13, 2026
7 checks passed

	Cursor — `~/.codetok/cursor/*.csv`
	Cursor — `~/.codetok/cursor/*/.csv`

	codetok reads local session data and imported usage exports stored on disk. Each provider has its own parser that understands the tool's data format. All session files are parsed in parallel using bounded goroutines (default: `min(NumCPU, 8)`, configurable via `CODETOK_WORKERS` env var).
	codetok reads local session data and imported usage exports stored on disk. Each provider has its own parser that understands the tool's data format. All supported session log files are parsed in parallel using bounded goroutines (default: `min(NumCPU, 8)`, configurable via `CODETOK_WORKERS` env var); Cursor CSV imports are processed sequentially.

	- Cursor — 解析 `~/.codetok/cursor/*/.csv` 下导入的 Cursor 用量导出文件
	- Cursor — 解析 `~/.codetok/cursor` 目录下（含子目录）的所有 `.csv` Cursor 用量导出文件

	codetok 读取本地磁盘上的会话数据和手动导入的用量导出文件。每个 Provider 有独立的解析器，理解对应工具的数据格式。所有会话文件通过有界 goroutine 并行解析（默认：`min(NumCPU, 8)`，可通过 `CODETOK_WORKERS` 环境变量配置）。
	codetok 读取本地磁盘上的会话数据和手动导入的用量导出文件。每个 Provider 有独立的解析器，理解对应工具的数据格式。Kimi / Claude Code / Codex 等 JSONL 会话文件通过有界 goroutine 并行解析（默认：`min(NumCPU, 8)`，可通过 `CODETOK_WORKERS` 环境变量配置），Cursor 的 CSV 用量导出文件目前按文件顺序解析。

Conversation

Miss-you commented Mar 12, 2026

Summary

Validation

Notes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Miss-you Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Miss-you Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Miss-you Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Miss-you Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Miss-you Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Miss-you Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants