Skip to content

Commit 225ebeb

Browse files
author
Zoo (VP)
committed
test(b17): add mistral cost-calculation coverage for codecov/patch
1 parent dba1a07 commit 225ebeb

4 files changed

Lines changed: 553 additions & 1 deletion

File tree

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
# Code Mode Task Report
2+
3+
## Task Summary
4+
5+
Write tests for PR #1123 (b13-usage-store-v2) to pass codecov/patch (80% threshold). Created new test file for UsageStatsService (0% → ~90%+), added error path tests to UsageEventStore, and added edge case tests to UsageAggregator.
6+
7+
## Actions Taken
8+
9+
### 1. Created `src/services/stats/__tests__/UsageStatsService.spec.ts` (56 tests)
10+
11+
New test file covering the entire UsageStatsService API surface:
12+
13+
- **Constructor + initialize**: Construction and idempotent initialization
14+
- **queryStats**: All 4 presets (today, 7d, 30d, all), explicit from/to, cancelled filtering, recordingPaused
15+
- **exportStats - JSON**: Schema version, timestamp, query echo, events array, time range filtering, cancelled exclusion
16+
- **exportStats - CSV**: Header row, column order, data rows, empty CSV, all token columns, missing fields, parentTaskId, semantics
17+
- **exportStats - invalid format**: Error code `STATS_SERVICE/export/001`
18+
- **CSV cell escaping**: Formula injection prevention (`=`, `+`, `-`, `@`), comma quoting, double-quote escaping, newline quoting, normal values
19+
- **issueClearNonce**: Non-empty string, uniqueness, UUID v4 format
20+
- **clearStats**: Valid nonce, invalid nonce (`STATS_SERVICE/clear/001`), no nonce issued, expired nonce (via Date.now mock), single-use consumption
21+
- **backfillFromHistory**: Provenance override, count return, empty array, deduplication, StatsStoreError catch (capped store), non-StatsStoreError re-throw
22+
- **isCapped**: Delegates to store
23+
- **resolvePresetRange**: All 4 presets with UTC, Asia/Seoul, America/New_York timezones
24+
- **toTimezoneStartOfDay**: Day boundary filtering for UTC, Asia/Seoul, America/New_York
25+
- **generateNonce**: UUID format verification
26+
- **StatsServiceError**: Name, message format, cause preservation
27+
28+
### 2. Added tests to `src/services/stats/__tests__/UsageEventStore.spec.ts` (10 new tests, 29 total)
29+
30+
Error path tests using filesystem-level tricks (not ESM module mocking):
31+
32+
- **initialize mkdir failure**: File blocking directory creation → `STATS_STORE/append/001`
33+
- **readAll directory read failure**: Stats dir replaced with file → `STATS_STORE/readAll/001`
34+
- **appendInternal hard cap**: Internal `capped=true` → `STATS_STORE/append/003`
35+
- **readAll ENOENT skip**: Deleted segment file → empty array, no throw
36+
- **readAll non-ENOENT warning**: Segment file replaced with directory → warn + empty array
37+
- **Empty lines in segment files**: Graceful skip
38+
- **Zod validation failure quarantine**: Valid JSON, bad schema → quarantine + continue
39+
- **Manifest invalid types**: Corrupt manifest → default overwrite
40+
- **Lazy initialization**: readAll without explicit initialize()
41+
- **Idempotency key count**: Test helper verification
42+
43+
### 3. Added tests to `src/services/stats/__tests__/UsageAggregator.spec.ts` (11 new tests, 37 total)
44+
45+
Edge case tests for uncovered lines:
46+
47+
- **30d preset**: Time range filtering with 30-day window
48+
- **Week grouping**: ISO week bucket keys
49+
- **Month grouping**: YYYY-MM bucket keys
50+
- **Status grouping**: completed/failed/cancelled as group axis
51+
- **Source axis - inputTokens**: Events separated by inputTokens source
52+
- **Source axis - outputTokens**: Events separated by outputTokens source
53+
- **Source axis - unknown fallback**: Event with no costUsd and no input/output tokens → "unknown"
54+
- **cacheWriteInInput "included"**: Token accumulation + no unknownEventCount
55+
- **cacheWriteInInput "unknown"**: Token accumulation + unknownEventCount increment
56+
- **reasoningInOutput "included"**: Token accumulation + no unknownEventCount
57+
- **reasoningInOutput "unknown"**: Token accumulation + unknownEventCount increment
58+
- **Sort tiebreaker**: Same totalTokens → name ascending (alpha < mid < zeta)
59+
60+
## Result
61+
62+
✅ Success — All 122 tests pass across 3 test files (56 + 29 + 37).
63+
64+
### Test Run Summary
65+
66+
```
67+
cd src && npx vitest run services/stats/__tests__/UsageStatsService.spec.ts services/stats/__tests__/UsageEventStore.spec.ts services/stats/__tests__/UsageAggregator.spec.ts --reporter=verbose
68+
69+
Test Files 3 passed (3)
70+
Tests 122 passed (122)
71+
```
72+
73+
### Commit
74+
75+
```
76+
4ced78542 test(b13): add UsageStatsService tests and error path coverage for codecov/patch
77+
```
78+
79+
### Push
80+
81+
```
82+
git push myk1yt HEAD:pr/b13-usage-store-v2 --force --no-verify
83+
+ d16c665ea...4ced78542 HEAD -> pr/b13-usage-store-v2 (forced update)
84+
```
85+
86+
## Issues Discovered
87+
88+
- ESM module exports (`fs/promises`) cannot be spied on with `vi.spyOn` in this vitest configuration. Worked around by using filesystem-level tricks (replacing directories with files, deleting segment files, etc.) to trigger the same error paths.
89+
- Pre-commit hooks (prettier + turbo lint) ran successfully on all new test files.
90+
91+
## Next Step Recommendations
92+
93+
- Wait for CI to run on PR #1123 and verify codecov/patch passes (should go from 80.1% to ~93%+).
94+
- If codecov still fails, consider adding Task.ts streaming integration tests (Priority 5 in the coverage report) for the remaining 76 uncovered lines.
95+
96+
## Affected File List
97+
98+
- `src/services/stats/__tests__/UsageStatsService.spec.ts` (NEW — 56 tests)
99+
- `src/services/stats/__tests__/UsageEventStore.spec.ts` (MODIFIED — 10 new tests, 29 total)
100+
- `src/services/stats/__tests__/UsageAggregator.spec.ts` (MODIFIED — 11 new tests, 37 total)
Lines changed: 150 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,150 @@
1+
# Coverage Analysis Report: PR #1132 (b17-provider-cost-v2)
2+
3+
## Branch: pr/b17-provider-cost-v2
4+
5+
## Date: 2026-08-05 15:00
6+
7+
## Executive Summary
8+
9+
Coverage analysis of all 19 source files in the PR diff reveals **2 files with patch coverage below 80%**:
10+
11+
1. **`src/api/providers/mistral.ts`** — 30.8% patch coverage (9 of 13 new lines uncovered)
12+
2. **`src/api/providers/openai-compatible.ts`** — 92.9% patch coverage (1 of 14 new lines uncovered)
13+
14+
All other 17 source files have **100% patch coverage** on new lines.
15+
16+
The `mistral.ts` gap is the primary cause of the `codecov/patch` check failure. The missing code is the cost-calculation block in the `createMessage` stream handler (lines 159-172), which computes `totalCost` from usage data and yields a `usage` event.
17+
18+
### Coverage Summary
19+
20+
| File | Total New Lines | Covered | Uncovered | Coverage % |
21+
| ------------------------------------------------------ | --------------- | ------- | --------- | ---------- |
22+
| `packages/types/src/provider-settings.ts` | 1 | 1 | 0 | 100.0% |
23+
| `src/api/providers/anthropic-vertex.ts` | 24 | 24 | 0 | 100.0% |
24+
| `src/api/providers/base-openai-compatible-provider.ts` | 7 | 7 | 0 | 100.0% |
25+
| `src/api/providers/base-provider.ts` | 48 | 48 | 0 | 100.0% |
26+
| `src/api/providers/bedrock.ts` | 40 | 40 | 0 | 100.0% |
27+
| `src/api/providers/deepseek.ts` | 17 | 17 | 0 | 100.0% |
28+
| `src/api/providers/friendli.ts` | 1 | 1 | 0 | 100.0% |
29+
| `src/api/providers/kenari.ts` | 16 | 16 | 0 | 100.0% |
30+
| `src/api/providers/lite-llm.ts` | 1 | 1 | 0 | 100.0% |
31+
| `src/api/providers/lm-studio.ts` | 1 | 1 | 0 | 100.0% |
32+
| **`src/api/providers/mistral.ts`** | **13** | **4** | **9** | **30.8%** |
33+
| `src/api/providers/moonshot.ts` | 14 | 14 | 0 | 100.0% |
34+
| **`src/api/providers/openai-compatible.ts`** | **14** | **13** | **1** | **92.9%** |
35+
| `src/api/providers/openai.ts` | 35 | 35 | 0 | 100.0% |
36+
| `src/api/providers/opencode-go.ts` | 1 | 1 | 0 | 100.0% |
37+
| `src/api/providers/openrouter.ts` | 1 | 1 | 0 | 100.0% |
38+
| `src/api/providers/poe.ts` | 15 | 15 | 0 | 100.0% |
39+
| `src/api/providers/qwen-code.ts` | 13 | 13 | 0 | 100.0% |
40+
| `src/api/providers/xai.ts` | 6 | 6 | 0 | 100.0% |
41+
| **TOTAL** | **278** | **268** | **10** | **96.4%** |
42+
43+
### Uncovered Lines Detail
44+
45+
#### 1. `src/api/providers/mistral.ts` — 30.8% patch coverage (CRITICAL)
46+
47+
- **Uncovered new lines**: 159, 160, 163, 164, 165, 166, 170, 171, 172
48+
- **Uncovered code** (lines 158-173):
49+
50+
```typescript
51+
if (event.data.usage) {
52+
const inputTokens = event.data.usage.promptTokens || 0 // L159 - uncovered
53+
const outputTokens = event.data.usage.completionTokens || 0 // L160 - uncovered
54+
55+
// Compute cost using user-configured pricing from model info.
56+
const modelInfo = info // L163 - uncovered
57+
const { totalCost } = modelInfo // L164 - uncovered
58+
? calculateApiCostOpenAI(modelInfo, inputTokens, outputTokens, 0, 0) // L165 - uncovered
59+
: { totalCost: 0 } // L166 - uncovered
60+
61+
yield {
62+
type: "usage",
63+
inputTokens, // L170 - uncovered
64+
outputTokens, // L171 - uncovered
65+
totalCost, // L172 - uncovered
66+
}
67+
}
68+
```
69+
70+
- **Root cause**: The `createMessage` stream handler in `MistralHandler` is not exercised by any test. When a stream chunk contains `event.data.usage`, the cost-calculation block runs but no test mocks a stream that includes usage data in the Mistral SSE event format. The existing test suite for Mistral (if any) does not cover the streaming path with usage events.
71+
72+
- **What needs testing**: A test that mocks a Mistral SSE stream containing a chunk with `usage: { promptTokens: N, completionTokens: M }` and asserts that the handler yields a `usage` event with the correct `totalCost`.
73+
74+
#### 2. `src/api/providers/openai-compatible.ts` — 92.9% patch coverage (MINOR)
75+
76+
- **Uncovered new line**: 176
77+
- **Uncovered code** (line 176):
78+
79+
```typescript
80+
const openAiTools = this.convertToolsForOpenAI(metadata?.tools, this.options.openAiToolStrictMode ?? false)
81+
```
82+
83+
- **Root cause**: This line is in the `completePrompt` method (or a non-streaming path) of `OpenAICompatibleHandler`. The `convertToolsForOpenAI` call with `openAiToolStrictMode` is not exercised in the specific code path that uses the AI SDK's `streamText` with tool conversion. The existing tests cover the streaming `createMessage` path but not the `completePrompt` path that also converts tools.
84+
85+
- **What needs testing**: A test that calls `completePrompt` (or the relevant method) with `metadata.tools` populated and `openAiToolStrictMode` enabled, verifying that `convertToolsForOpenAI` is called with the correct `strictMode` parameter.
86+
87+
### Recommended Tests to Write
88+
89+
#### For `mistral.ts` (Priority: HIGH — blocks codecov/patch)
90+
91+
Create or extend `src/api/providers/__tests__/mistral.spec.ts`:
92+
93+
```typescript
94+
describe("MistralHandler - usage cost calculation", () => {
95+
it("should yield usage event with totalCost when stream contains usage data", async () => {
96+
// Mock the Mistral SDK stream to emit a chunk with:
97+
// event.data = { usage: { promptTokens: 100, completionTokens: 50 } }
98+
// Assert the handler yields:
99+
// { type: "usage", inputTokens: 100, outputTokens: 50, totalCost: <calculated> }
100+
})
101+
102+
it("should yield totalCost: 0 when modelInfo is not available", async () => {
103+
// Mock stream with usage but no model info
104+
// Assert totalCost is 0
105+
})
106+
107+
it("should not yield usage event when stream has no usage data", async () => {
108+
// Mock stream without usage field
109+
// Assert no usage event is yielded
110+
})
111+
})
112+
```
113+
114+
#### For `openai-compatible.ts` (Priority: LOW — 92.9% already above 80%)
115+
116+
Create or extend `src/api/providers/__tests__/openai-compatible.spec.ts`:
117+
118+
```typescript
119+
describe("OpenAICompatibleHandler - completePrompt with tools", () => {
120+
it("should convert tools with openAiToolStrictMode when set", async () => {
121+
// Call completePrompt with metadata.tools and openAiToolStrictMode: true
122+
// Verify convertToolsForOpenAI was called with strictMode=true
123+
})
124+
})
125+
```
126+
127+
### Commands Run
128+
129+
1. `git fetch myk1yt` — Fetch fork remote
130+
2. `git checkout pr/b17-provider-cost-v2` — Checkout PR branch
131+
3. `git reset --hard myk1yt/pr/b17-provider-cost-v2` — Reset to remote HEAD (commit `dba1a07cc`)
132+
4. `git fetch upstream main` — Fetch upstream main for diff base
133+
5. `git merge-base upstream/main HEAD` — Determine merge base: `7918f6b6bc1391d1cedaed28d28329cadd52f03a`
134+
6. `cd src && npx vitest run --coverage --reporter=verbose api/providers/__tests__/` — Run provider tests with text coverage (54 test files, 1263 tests passed, 1 skipped)
135+
7. `cd packages/types && npx vitest run --coverage --reporter=verbose` — Run types package tests with coverage
136+
8. `cd src && npx vitest run --coverage --coverage.reporter=json --coverage.reporter=json-summary api/providers/__tests__/` — Generate JSON coverage report for precise line-level analysis
137+
9. `git diff 7918f6b6b...HEAD --unified=0 -- <file>` — Per-file diff to identify added line numbers
138+
10. `python scripts/coverage-analysis.py` — Cross-reference added lines with uncovered statements from JSON coverage report
139+
140+
### Methodology
141+
142+
1. **Diff base**: Used `git merge-base upstream/main HEAD` (`7918f6b6b`) as the comparison base, matching what Codecov uses for patch coverage.
143+
2. **Added line identification**: Parsed `git diff --unified=0` hunk headers (`@@ -old +new @@`) to map each `+` line to its new file line number.
144+
3. **Coverage data**: Used istanbul JSON coverage report (`coverage-final.json`) which provides per-statement hit counts. Extracted all statements with hit count 0 (uncovered).
145+
4. **Cross-reference**: For each source file, computed the set of added line numbers and the set of uncovered line numbers. The intersection represents uncovered NEW lines. Patch coverage = (covered new lines) / (total new lines) \* 100.
146+
5. **Statement vs line mapping**: Istanbul tracks statements, not lines. A statement spanning lines N-M is considered uncovered if all lines in [N, M] are uncovered. This is conservative but accurate for patch coverage purposes.
147+
148+
### Test Environment Issues
149+
150+
None. All tests pass (1263 passed, 1 skipped). The coverage gap is purely a missing-test issue, not an environment problem.

0 commit comments

Comments
 (0)