Open in StackBlitz

npm i https://pkg.pr.new/ryoppippi/ccusage@29

commit: 36fd76d

We might consider implementing some kind of sorting mechanism if we're using a hash of requestID and message.id and we're expecting some duplicates... we only want to "count" the first time we see the duplicate and ideally we're processing these in order from oldest (first occurrence) to newest (last occurrence). But if we're using glob and processing these alphabetically by a randomly assigned UUID, we might want to look at something like first timestamp in the files to make sure we process them in order.

I don't know for sure, but it seems like order matters if we're filtering out duplicates and hoping to keep the session aggregation as accurate as possible.

AI summary and suggestion -

Deduplication Fix Summary

Original Issue: Duplicate Message Counting

Problem Description

When using Claude Code with branched conversations (editing and retrying messages), the usage tracking system was counting the same messages multiple times. This occurred because:

1. Claude creates branches in conversations when users edit and retry messages
2. Each branch contains the same message with identical message.id and requestId
3. The usage analysis tool was counting these duplicate entries, leading to inflated token counts and costs

Original Fix (Commit 36fd76d)

Author: ryoppippi
Date: Sat Jun 7 21:54:04 2025 +0100
Commit: fix: resolve duplicate message id

Core Implementation

1. Extended Schema: Added optional requestId and message.id fields to the UsageDataSchema
2. Deduplication Logic: Implemented createUniqueHash() function that creates unique identifiers from message ID + request ID combinations
3. Global Deduplication: Added deduplication across all files in both loadDailyUsageData() and loadSessionData() functions
4. Backward Compatibility: Entries without IDs are processed normally (no breaking changes)

Key Features

• Prevents Double-Counting: Skips duplicate message+request ID combinations globally across all conversation branches
• Handles Edge Cases: Processes entries without IDs normally, allows same message ID with different request IDs
• Session Integrity: Ensures session entries are created even if all messages are deduplicated (prevents missing sessions)
• Cost Accuracy: Only calculates costs for unique messages, preventing inflated usage numbers in branched conversations

Enhancement: Chronological File Processing

Problem Discovered

While the original fix correctly identified and skipped duplicates, it had a subtle issue with which duplicate to keep:

1. File Processing Order: Claude stores session data in files named with UUIDs (e.g., 9cea3b20-090d-49a1-a7d2-3aaaa0ad22b4.jsonl)
2. UUID Alphabetical Sorting: The glob pattern returns files sorted alphabetically by UUID, not chronologically
3. Cross-Session Duplicates: When the same message appears in multiple sessions (different UUID files), the deduplication would keep whichever file was processed first alphabetically, not the chronologically oldest entry

Example Scenario

File: a1234567-...jsonl (created Jan 5) - Contains msg123
File: f1234567-...jsonl (created Jan 1) - Contains msg123 (same message)

Alphabetical processing order:
1. a1234567 (Jan 5) - msg123 kept ✗
2. f1234567 (Jan 1) - msg123 skipped

Result: Newer entry kept instead of older!

Why This Matters

1. Accurate Historical Reporting: Session reports should show when tokens were first consumed
2. Consistent Token Attribution: The first time a message is processed is when tokens are actually used
3. Correct "Last Activity" Dates: Sessions should reflect accurate timestamps

Solution Implemented

New Function: sortFilesByFirstTimestamp()

async function sortFilesByFirstTimestamp(files: string[]): Promise<string[]> {
  // Read first line of each file to extract timestamp
  // Sort files by timestamp (oldest first)
  // Append files without valid timestamps at end
}

Integration Points

1. loadDailyUsageData(): Added const sortedFiles = await sortFilesByFirstTimestamp(files);
2. loadSessionData(): Added const sortedFiles = await sortFilesByFirstTimestamp(files);
3. loadMonthlyUsageData(): Automatically benefits since it calls loadDailyUsageData()

Technical Details

How It Works

1. Timestamp Extraction: Reads the first line of each JSONL file to get its earliest timestamp
2. Chronological Sorting: Sorts all files by their first timestamp (oldest to newest)
3. Fallback Handling: Files without valid timestamps are appended at the end
4. Preserves Deduplication: The existing deduplication logic remains unchanged, but now processes files in chronological order

Performance Considerations

• Trade-off: Slight performance impact from reading first line of all files
• Benefit: Ensures correctness of deduplication across all scenarios
• Optimization: Only reads first line, not entire file content

Backward Compatibility

• Handles both old schema (timestamp at root) and new schema (message.timestamp)
• Files without timestamps are still processed (appended at end)
• No changes to output format or API

Testing

Added comprehensive tests to verify:

1. Files are processed in chronological order regardless of UUID
2. Oldest entries are preserved when duplicates exist across sessions
3. Session aggregation remains correct with the new sorting

Summary

This enhancement ensures that the deduplication fix not only prevents double-counting but also preserves the chronologically accurate data. When duplicate messages exist across different sessions, the system now correctly keeps the first occurrence based on actual timestamps, not file processing order.

Recommended Code Changes

1. Add the sortFilesByFirstTimestamp function to src/data-loader.ts

Add this function after the createUniqueHash function (around line 99):

/**
 * Sort files by their first timestamp to ensure chronological processing
 * This ensures deduplication keeps the oldest entries
 */
async function sortFilesByFirstTimestamp(files: string[]): Promise<string[]> {
	const filesWithTimestamps = await Promise.all(
		files.map(async (file) => {
			try {
				const content = await readFile(file, "utf-8");
				const firstLine = content.trim().split("\n")[0];
				if (!firstLine) return { file, timestamp: null };

				const parsed = JSON.parse(firstLine);
				// Handle both old and new schema
				const timestamp = parsed.timestamp || parsed.message?.timestamp;
				return { file, timestamp: timestamp ? new Date(timestamp) : null };
			} catch (e) {
				return { file, timestamp: null };
			}
		}),
	);

	// Filter out files without timestamps and sort by timestamp
	return filesWithTimestamps
		.filter((item) => item.timestamp !== null)
		.sort((a, b) => {
			// Sort by timestamp, oldest first
			// biome-ignore lint/style/noNonNullAssertion: filtered above
			return a.timestamp!.getTime() - b.timestamp!.getTime();
		})
		.map((item) => item.file)
		.concat(
			// Append files without valid timestamps at the end
			filesWithTimestamps
				.filter((item) => item.timestamp === null)
				.map((item) => item.file),
		);
}

2. Update loadDailyUsageData function

In the loadDailyUsageData function, after fetching the pricing data (around line 203), add the sorting step:

// Fetch pricing data for cost calculation only when needed
const mode = options?.mode || "auto";
const modelPricing = mode === "display" ? {} : await fetchModelPricing();

// Sort files by their first timestamp to ensure chronological processing
const sortedFiles = await sortFilesByFirstTimestamp(files);

// Collect all valid data entries first
const allEntries: { data: UsageData; date: string; cost: number }[] = [];
const processedRequests = new Set<string>(); // Track processed message+request combinations

for (const file of sortedFiles) { // Changed from 'files' to 'sortedFiles'

3. Update loadSessionData function

Similarly, in the loadSessionData function, after fetching the pricing data (around line 320), add the sorting step:

// Fetch pricing data for cost calculation only when needed
const mode = options?.mode || "auto";
const modelPricing = mode === "display" ? {} : await fetchModelPricing();

// Sort files by their first timestamp to ensure chronological processing
const sortedFiles = await sortFilesByFirstTimestamp(files);

// Collect all valid data entries with session info first
const allEntries: Array<{
	data: UsageData;
	sessionKey: string;
	sessionId: string;
	projectPath: string;
	cost: number;
	timestamp: string;
}> = [];
const processedRequests = new Set<string>(); // Track processed message+request combinations

for (const file of sortedFiles) { // Changed from 'files' to 'sortedFiles'

4. Add tests to src/data-loader.test.ts

Add these tests within the deduplication describe block (after line 331):

test("should keep the oldest entry when duplicates are found", async () => {
	const olderData = {
		timestamp: "2024-01-01T00:00:00Z", // Earlier timestamp
		message: {
			usage: { input_tokens: 100, output_tokens: 50 },
			id: "msg123",
		},
		requestId: "req456",
		costUSD: 0.01,
	};

	const newerData = {
		timestamp: "2024-01-01T12:00:00Z", // Later timestamp
		message: {
			usage: { input_tokens: 200, output_tokens: 100 }, // Different token counts
			id: "msg123", // Same message ID
		},
		requestId: "req456", // Same request ID
		costUSD: 0.02, // Different cost
	};

	await using fixture = await createFixture({
		projects: {
			project1: {
				session1: {
					"a-first.jsonl": JSON.stringify(olderData), // File that should be processed first
					"b-second.jsonl": JSON.stringify(newerData), // File that should be processed second
				},
			},
		},
	});

	const dailyResult = await loadDailyUsageData({ claudePath: fixture.path });
	const sessionResult = await loadSessionData({ claudePath: fixture.path });

	// Daily aggregation should use older data
	expect(dailyResult).toHaveLength(1);
	expect(dailyResult[0]?.inputTokens).toBe(100); // From older entry
	expect(dailyResult[0]?.outputTokens).toBe(50); // From older entry
	expect(dailyResult[0]?.totalCost).toBe(0.01); // From older entry

	// Session data should also reflect older data
	expect(sessionResult).toHaveLength(1);
	expect(sessionResult[0]?.inputTokens).toBe(100); // From older entry
	expect(sessionResult[0]?.outputTokens).toBe(50); // From older entry
	expect(sessionResult[0]?.totalCost).toBe(0.01); // From older entry
	expect(sessionResult[0]?.lastActivity).toBe("2024-01-01"); // Should be from the older timestamp
});

test("should process files in chronological order based on first timestamp", async () => {
	// Create files with UUIDs that would sort differently than chronological order
	const file1Data = {
		timestamp: "2024-01-05T00:00:00Z", // Newer
		message: {
			usage: { input_tokens: 100, output_tokens: 50 },
			id: "msg123",
		},
		requestId: "req456",
		costUSD: 0.01,
	};

	const file2Data = {
		timestamp: "2024-01-01T00:00:00Z", // Older
		message: {
			usage: { input_tokens: 200, output_tokens: 100 },
			id: "msg123", // Same message
		},
		requestId: "req456", // Same request
		costUSD: 0.02,
	};

	await using fixture = await createFixture({
		projects: {
			project1: {
				// UUID starting with 'a' would normally be processed first
				"aaaa1111-1111-1111-1111-111111111111": {
					"usage.jsonl": JSON.stringify(file1Data), // But contains newer data
				},
				// UUID starting with 'z' would normally be processed last
				"zzzz9999-9999-9999-9999-999999999999": {
					"usage.jsonl": JSON.stringify(file2Data), // But contains older data
				},
			},
		},
	});

	const result = await loadDailyUsageData({ claudePath: fixture.path });

	// Should have kept the older entry (from zzzz file) despite it being alphabetically last
	expect(result).toHaveLength(1);
	expect(result[0]?.inputTokens).toBe(200); // From the older file2Data
	expect(result[0]?.outputTokens).toBe(100); // From the older file2Data
	expect(result[0]?.totalCost).toBe(0.02); // From the older file2Data
});

Note on Dependencies

No new dependencies are required. The solution uses only existing imports already present in the file.

I'm not on my desktop right now to check, but have you checked if the timestamp of the carried over messages aren't the same for the original conversation and the branched one?

I also feel like just reading every line into a single list, adding the conversation id to the object and then doing some data manipulation would be simpler than some kind of tracking.
Imagine someone using the same chat across multiple days for example with various branches, could quickly get tricky to parse in a conversation context instead of more detached from it.

True if the time stamps remain the same. Then we don't really need to handle them, just filter out based on hashed key is sufficient.

I'm away from my computer too now, but those timestamps are probably something to check if changing or staying the same when duplicated.

I checked and it changes the timestamp, sets it all to the same date as the new message, so you'd have to also only count the chronologically first unique message.

It sounds nice. Could you implement it with new PR?

I've created a new draft PR to address this issue here: #58

@Richard-Weiss - Would you be able to clone that branch/PR and test the output vs what you expect?

close this because #58 is better one