|
| 1 | +# Coverage Analysis Report: PR #1132 (b17-provider-cost-v2) |
| 2 | + |
| 3 | +## Branch: pr/b17-provider-cost-v2 |
| 4 | + |
| 5 | +## Date: 2026-08-05 15:00 |
| 6 | + |
| 7 | +## Executive Summary |
| 8 | + |
| 9 | +Coverage analysis of all 19 source files in the PR diff reveals **2 files with patch coverage below 80%**: |
| 10 | + |
| 11 | +1. **`src/api/providers/mistral.ts`** — 30.8% patch coverage (9 of 13 new lines uncovered) |
| 12 | +2. **`src/api/providers/openai-compatible.ts`** — 92.9% patch coverage (1 of 14 new lines uncovered) |
| 13 | + |
| 14 | +All other 17 source files have **100% patch coverage** on new lines. |
| 15 | + |
| 16 | +The `mistral.ts` gap is the primary cause of the `codecov/patch` check failure. The missing code is the cost-calculation block in the `createMessage` stream handler (lines 159-172), which computes `totalCost` from usage data and yields a `usage` event. |
| 17 | + |
| 18 | +### Coverage Summary |
| 19 | + |
| 20 | +| File | Total New Lines | Covered | Uncovered | Coverage % | |
| 21 | +| ------------------------------------------------------ | --------------- | ------- | --------- | ---------- | |
| 22 | +| `packages/types/src/provider-settings.ts` | 1 | 1 | 0 | 100.0% | |
| 23 | +| `src/api/providers/anthropic-vertex.ts` | 24 | 24 | 0 | 100.0% | |
| 24 | +| `src/api/providers/base-openai-compatible-provider.ts` | 7 | 7 | 0 | 100.0% | |
| 25 | +| `src/api/providers/base-provider.ts` | 48 | 48 | 0 | 100.0% | |
| 26 | +| `src/api/providers/bedrock.ts` | 40 | 40 | 0 | 100.0% | |
| 27 | +| `src/api/providers/deepseek.ts` | 17 | 17 | 0 | 100.0% | |
| 28 | +| `src/api/providers/friendli.ts` | 1 | 1 | 0 | 100.0% | |
| 29 | +| `src/api/providers/kenari.ts` | 16 | 16 | 0 | 100.0% | |
| 30 | +| `src/api/providers/lite-llm.ts` | 1 | 1 | 0 | 100.0% | |
| 31 | +| `src/api/providers/lm-studio.ts` | 1 | 1 | 0 | 100.0% | |
| 32 | +| **`src/api/providers/mistral.ts`** | **13** | **4** | **9** | **30.8%** | |
| 33 | +| `src/api/providers/moonshot.ts` | 14 | 14 | 0 | 100.0% | |
| 34 | +| **`src/api/providers/openai-compatible.ts`** | **14** | **13** | **1** | **92.9%** | |
| 35 | +| `src/api/providers/openai.ts` | 35 | 35 | 0 | 100.0% | |
| 36 | +| `src/api/providers/opencode-go.ts` | 1 | 1 | 0 | 100.0% | |
| 37 | +| `src/api/providers/openrouter.ts` | 1 | 1 | 0 | 100.0% | |
| 38 | +| `src/api/providers/poe.ts` | 15 | 15 | 0 | 100.0% | |
| 39 | +| `src/api/providers/qwen-code.ts` | 13 | 13 | 0 | 100.0% | |
| 40 | +| `src/api/providers/xai.ts` | 6 | 6 | 0 | 100.0% | |
| 41 | +| **TOTAL** | **278** | **268** | **10** | **96.4%** | |
| 42 | + |
| 43 | +### Uncovered Lines Detail |
| 44 | + |
| 45 | +#### 1. `src/api/providers/mistral.ts` — 30.8% patch coverage (CRITICAL) |
| 46 | + |
| 47 | +- **Uncovered new lines**: 159, 160, 163, 164, 165, 166, 170, 171, 172 |
| 48 | +- **Uncovered code** (lines 158-173): |
| 49 | + |
| 50 | +```typescript |
| 51 | +if (event.data.usage) { |
| 52 | + const inputTokens = event.data.usage.promptTokens || 0 // L159 - uncovered |
| 53 | + const outputTokens = event.data.usage.completionTokens || 0 // L160 - uncovered |
| 54 | + |
| 55 | + // Compute cost using user-configured pricing from model info. |
| 56 | + const modelInfo = info // L163 - uncovered |
| 57 | + const { totalCost } = modelInfo // L164 - uncovered |
| 58 | + ? calculateApiCostOpenAI(modelInfo, inputTokens, outputTokens, 0, 0) // L165 - uncovered |
| 59 | + : { totalCost: 0 } // L166 - uncovered |
| 60 | + |
| 61 | + yield { |
| 62 | + type: "usage", |
| 63 | + inputTokens, // L170 - uncovered |
| 64 | + outputTokens, // L171 - uncovered |
| 65 | + totalCost, // L172 - uncovered |
| 66 | + } |
| 67 | +} |
| 68 | +``` |
| 69 | + |
| 70 | +- **Root cause**: The `createMessage` stream handler in `MistralHandler` is not exercised by any test. When a stream chunk contains `event.data.usage`, the cost-calculation block runs but no test mocks a stream that includes usage data in the Mistral SSE event format. The existing test suite for Mistral (if any) does not cover the streaming path with usage events. |
| 71 | + |
| 72 | +- **What needs testing**: A test that mocks a Mistral SSE stream containing a chunk with `usage: { promptTokens: N, completionTokens: M }` and asserts that the handler yields a `usage` event with the correct `totalCost`. |
| 73 | + |
| 74 | +#### 2. `src/api/providers/openai-compatible.ts` — 92.9% patch coverage (MINOR) |
| 75 | + |
| 76 | +- **Uncovered new line**: 176 |
| 77 | +- **Uncovered code** (line 176): |
| 78 | + |
| 79 | +```typescript |
| 80 | +const openAiTools = this.convertToolsForOpenAI(metadata?.tools, this.options.openAiToolStrictMode ?? false) |
| 81 | +``` |
| 82 | + |
| 83 | +- **Root cause**: This line is in the `completePrompt` method (or a non-streaming path) of `OpenAICompatibleHandler`. The `convertToolsForOpenAI` call with `openAiToolStrictMode` is not exercised in the specific code path that uses the AI SDK's `streamText` with tool conversion. The existing tests cover the streaming `createMessage` path but not the `completePrompt` path that also converts tools. |
| 84 | + |
| 85 | +- **What needs testing**: A test that calls `completePrompt` (or the relevant method) with `metadata.tools` populated and `openAiToolStrictMode` enabled, verifying that `convertToolsForOpenAI` is called with the correct `strictMode` parameter. |
| 86 | + |
| 87 | +### Recommended Tests to Write |
| 88 | + |
| 89 | +#### For `mistral.ts` (Priority: HIGH — blocks codecov/patch) |
| 90 | + |
| 91 | +Create or extend `src/api/providers/__tests__/mistral.spec.ts`: |
| 92 | + |
| 93 | +```typescript |
| 94 | +describe("MistralHandler - usage cost calculation", () => { |
| 95 | + it("should yield usage event with totalCost when stream contains usage data", async () => { |
| 96 | + // Mock the Mistral SDK stream to emit a chunk with: |
| 97 | + // event.data = { usage: { promptTokens: 100, completionTokens: 50 } } |
| 98 | + // Assert the handler yields: |
| 99 | + // { type: "usage", inputTokens: 100, outputTokens: 50, totalCost: <calculated> } |
| 100 | + }) |
| 101 | + |
| 102 | + it("should yield totalCost: 0 when modelInfo is not available", async () => { |
| 103 | + // Mock stream with usage but no model info |
| 104 | + // Assert totalCost is 0 |
| 105 | + }) |
| 106 | + |
| 107 | + it("should not yield usage event when stream has no usage data", async () => { |
| 108 | + // Mock stream without usage field |
| 109 | + // Assert no usage event is yielded |
| 110 | + }) |
| 111 | +}) |
| 112 | +``` |
| 113 | + |
| 114 | +#### For `openai-compatible.ts` (Priority: LOW — 92.9% already above 80%) |
| 115 | + |
| 116 | +Create or extend `src/api/providers/__tests__/openai-compatible.spec.ts`: |
| 117 | + |
| 118 | +```typescript |
| 119 | +describe("OpenAICompatibleHandler - completePrompt with tools", () => { |
| 120 | + it("should convert tools with openAiToolStrictMode when set", async () => { |
| 121 | + // Call completePrompt with metadata.tools and openAiToolStrictMode: true |
| 122 | + // Verify convertToolsForOpenAI was called with strictMode=true |
| 123 | + }) |
| 124 | +}) |
| 125 | +``` |
| 126 | + |
| 127 | +### Commands Run |
| 128 | + |
| 129 | +1. `git fetch myk1yt` — Fetch fork remote |
| 130 | +2. `git checkout pr/b17-provider-cost-v2` — Checkout PR branch |
| 131 | +3. `git reset --hard myk1yt/pr/b17-provider-cost-v2` — Reset to remote HEAD (commit `dba1a07cc`) |
| 132 | +4. `git fetch upstream main` — Fetch upstream main for diff base |
| 133 | +5. `git merge-base upstream/main HEAD` — Determine merge base: `7918f6b6bc1391d1cedaed28d28329cadd52f03a` |
| 134 | +6. `cd src && npx vitest run --coverage --reporter=verbose api/providers/__tests__/` — Run provider tests with text coverage (54 test files, 1263 tests passed, 1 skipped) |
| 135 | +7. `cd packages/types && npx vitest run --coverage --reporter=verbose` — Run types package tests with coverage |
| 136 | +8. `cd src && npx vitest run --coverage --coverage.reporter=json --coverage.reporter=json-summary api/providers/__tests__/` — Generate JSON coverage report for precise line-level analysis |
| 137 | +9. `git diff 7918f6b6b...HEAD --unified=0 -- <file>` — Per-file diff to identify added line numbers |
| 138 | +10. `python scripts/coverage-analysis.py` — Cross-reference added lines with uncovered statements from JSON coverage report |
| 139 | + |
| 140 | +### Methodology |
| 141 | + |
| 142 | +1. **Diff base**: Used `git merge-base upstream/main HEAD` (`7918f6b6b`) as the comparison base, matching what Codecov uses for patch coverage. |
| 143 | +2. **Added line identification**: Parsed `git diff --unified=0` hunk headers (`@@ -old +new @@`) to map each `+` line to its new file line number. |
| 144 | +3. **Coverage data**: Used istanbul JSON coverage report (`coverage-final.json`) which provides per-statement hit counts. Extracted all statements with hit count 0 (uncovered). |
| 145 | +4. **Cross-reference**: For each source file, computed the set of added line numbers and the set of uncovered line numbers. The intersection represents uncovered NEW lines. Patch coverage = (covered new lines) / (total new lines) \* 100. |
| 146 | +5. **Statement vs line mapping**: Istanbul tracks statements, not lines. A statement spanning lines N-M is considered uncovered if all lines in [N, M] are uncovered. This is conservative but accurate for patch coverage purposes. |
| 147 | + |
| 148 | +### Test Environment Issues |
| 149 | + |
| 150 | +None. All tests pass (1263 passed, 1 skipped). The coverage gap is purely a missing-test issue, not an environment problem. |
0 commit comments