Skip to content

Commit 21be412

Browse files
hi-ogawacodexclaude
authored
feat: expose per-retry/repeat attempt result (#11232)
Co-authored-by: Hiroshi Ogawa <[email protected]> Co-authored-by: Codex (GPT-6) <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 08bd93b commit 21be412

9 files changed

Lines changed: 409 additions & 3 deletions

File tree

‎docs/api/advanced/runner.md‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -268,6 +268,37 @@ export interface TaskResult {
268268
* The zero-based index of the current repeat.
269269
*/
270270
repeatCount?: number
271+
/**
272+
* Individual results for every retry and repeat attempt.
273+
*/
274+
attempts?: TaskResultAttempt[]
275+
}
276+
277+
export interface TaskResultAttempt {
278+
/**
279+
* The state after applying the test's expected failure option.
280+
*/
281+
state: 'pass' | 'fail' | 'skip'
282+
/**
283+
* Errors produced by this attempt.
284+
*/
285+
errors?: TestError[]
286+
/**
287+
* How long in milliseconds the attempt took to run.
288+
*/
289+
duration: number
290+
/**
291+
* Time in milliseconds when the attempt started running.
292+
*/
293+
startTime: number
294+
/**
295+
* The zero-based retry index within the repeat.
296+
*/
297+
retryIndex: number
298+
/**
299+
* The zero-based repeat index.
300+
*/
301+
repeatIndex: number
271302
}
272303
```
273304

‎docs/api/advanced/test-case.md‎

Lines changed: 53 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -228,7 +228,7 @@ interface TestResultPassed {
228228
```
229229

230230
::: warning
231-
Note that the test with `passed` state can still have errors attached - this can happen if `retry` was triggered at least once.
231+
Note that the test with `passed` state can still have errors attached - this can happen if `retry` was triggered at least once. Use [`attempts()`](#attempts) to see which run produced each error.
232232
:::
233233

234234
## diagnostic
@@ -278,6 +278,58 @@ interface TestDiagnostic {
278278
`diagnostic()` will return `undefined` if the test was not scheduled to run yet.
279279
:::
280280

281+
## attempts <Version>5.1.0</Version> {#attempts}
282+
283+
```ts
284+
function attempts(): ReadonlyArray<TestAttempt>
285+
```
286+
287+
Results of every run of the test, in execution order. A test runs more than once when [`retry`](/config/retry) or [`repeats`](/api/test#repeats) is configured, and each run produces one entry. Unlike `result()`, which only reports the final outcome and merges errors from all runs, each attempt keeps its own state, errors and timing.
288+
289+
```ts
290+
interface TestAttempt {
291+
/**
292+
* The state of this attempt. `fails` tests are already inverted,
293+
* so an expected failure is reported as `passed`.
294+
*/
295+
readonly state: 'passed' | 'failed' | 'skipped'
296+
/**
297+
* Errors thrown during this attempt only.
298+
*/
299+
readonly errors: ReadonlyArray<TestError> | undefined
300+
/**
301+
* How long in milliseconds the attempt took to run.
302+
*/
303+
readonly duration: number
304+
/**
305+
* Time in milliseconds when the attempt started.
306+
*/
307+
readonly startTime: number
308+
/**
309+
* The zero-based retry index within the current repeat.
310+
*/
311+
readonly retryIndex: number
312+
/**
313+
* The zero-based repeat index.
314+
*/
315+
readonly repeatIndex: number
316+
}
317+
```
318+
319+
For example, a test with `retry: 2` that passes on its third run returns:
320+
321+
```ts
322+
[
323+
{ state: 'failed', retryIndex: 0, repeatIndex: 0, errors: [/* ... */] },
324+
{ state: 'failed', retryIndex: 1, repeatIndex: 0, errors: [/* ... */] },
325+
{ state: 'passed', retryIndex: 2, repeatIndex: 0, errors: undefined },
326+
]
327+
```
328+
329+
::: info
330+
`attempts()` returns an empty array if the test has not run yet.
331+
:::
332+
281333
## annotations
282334

283335
```ts

‎packages/vitest/src/node/reporters/reported-tasks.ts‎

Lines changed: 34 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -194,6 +194,20 @@ export class TestCase extends ReportedTaskImplementation {
194194
} satisfies TestResultFailed
195195
}
196196

197+
/**
198+
* Individual results for every retry and repeat attempt.
199+
*/
200+
public attempts(): ReadonlyArray<TestAttempt> {
201+
return (this.task.result?.attempts || []).map((attempt) => ({
202+
state: attempt.state === 'pass' ? 'passed' : attempt.state === 'fail' ? 'failed' : 'skipped',
203+
errors: attempt.errors as TestError[] | undefined,
204+
duration: attempt.duration,
205+
startTime: attempt.startTime,
206+
retryIndex: attempt.retryIndex,
207+
repeatIndex: attempt.repeatIndex,
208+
}))
209+
}
210+
197211
/**
198212
* Test annotations added via the `task.annotate` API during the test execution.
199213
*/
@@ -609,10 +623,29 @@ function buildOptions(task: RunnerTestCase | RunnerTestSuite): TaskOptions {
609623
}
610624
}
611625

612-
export type TestSuiteState = 'skipped' | 'pending' | 'failed' | 'passed'
626+
export type TestAttemptState = 'skipped' | 'failed' | 'passed'
627+
export type TestSuiteState = TestAttemptState | 'pending'
613628
export type TestModuleState = TestSuiteState | 'queued'
614629
export type TestState = TestResult['state']
615630

631+
/**
632+
* The result of an individual test attempt.
633+
*/
634+
export interface TestAttempt {
635+
/** The state after applying the test's expected failure option. */
636+
readonly state: TestAttemptState
637+
/** Errors produced by this attempt. */
638+
readonly errors: ReadonlyArray<TestError> | undefined
639+
/** How long in milliseconds the attempt took to run. */
640+
readonly duration: number
641+
/** Time in milliseconds when the attempt started running. */
642+
readonly startTime: number
643+
/** The zero-based retry index within the repeat. */
644+
readonly retryIndex: number
645+
/** The zero-based repeat index. */
646+
readonly repeatIndex: number
647+
}
648+
616649
export type TestResult = TestResultPassed | TestResultFailed | TestResultSkipped | TestResultPending
617650

618651
export interface TestResultPending {

‎packages/vitest/src/public/index.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,7 @@ export type {
8787
TaskBase as RunnerTaskBase,
8888
TaskEventPack as RunnerTaskEventPack,
8989
TaskResult as RunnerTaskResult,
90+
TaskResultAttempt as RunnerTaskResultAttempt,
9091
TaskResultPack as RunnerTaskResultPack,
9192
Test as RunnerTestCase,
9293
File as RunnerTestFile,

‎packages/vitest/src/public/node.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,8 @@ export type { Report } from '../node/reporters/report'
8989
export type {
9090
ModuleDiagnostic,
9191
TaskOptions,
92+
TestAttempt,
93+
TestAttemptState,
9294
TestCase,
9395
TestCollection,
9496
TestDiagnostic,

‎packages/vitest/src/runtime/runner/run.ts‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ import type {
1212
Task,
1313
TaskMeta,
1414
TaskResult,
15+
TaskResultAttempt,
1516
TaskResultPack,
1617
TaskState,
1718
TaskUpdateEvent,
@@ -599,6 +600,7 @@ async function runTest(test: Test, runner: VitestRunner): Promise<void> {
599600
state: 'run',
600601
startTime: unixNow(),
601602
retryCount: 0,
603+
attempts: [],
602604
}
603605
updateTask('test-prepare', test, runner)
604606

@@ -615,7 +617,20 @@ async function runTest(test: Test, runner: VitestRunner): Promise<void> {
615617
test.result.state = 'run' as TaskState
616618
const retry = getRetryCount(test.retry)
617619
for (let retryCount = 0; retryCount <= retry; retryCount++) {
620+
const attemptStart = now()
621+
const attemptStartTime = unixNow()
618622
const attemptErrorsStart = test.result.errors?.length ?? 0
623+
const recordAttempt = (state: TaskResultAttempt['state']) => {
624+
const errors = test.result!.errors?.slice(attemptErrorsStart)
625+
test.result!.attempts!.push({
626+
state,
627+
errors: errors?.length ? errors : undefined,
628+
duration: now() - attemptStart,
629+
startTime: attemptStartTime,
630+
retryIndex: retryCount,
631+
repeatIndex: repeatCount,
632+
})
633+
}
619634
const beforeEachCleanups: unknown[] = []
620635
// fixtureCheckpoint is passed by callAroundEachHooks - it represents the count
621636
// of fixture cleanup functions AFTER all aroundEach fixtures have been resolved
@@ -714,12 +729,14 @@ async function runTest(test: Test, runner: VitestRunner): Promise<void> {
714729

715730
// skipped with new PendingError
716731
if (test.result?.pending || test.result?.state === 'skip') {
732+
recordAttempt('skip')
717733
test.mode = 'skip'
718734
test.result = {
719735
state: 'skip',
720736
note: test.result?.note,
721737
pending: true,
722738
duration: now() - start,
739+
attempts: test.result.attempts,
723740
}
724741
updateTask('test-finished', test, runner)
725742
setCurrentTest(undefined)
@@ -744,6 +761,7 @@ async function runTest(test: Test, runner: VitestRunner): Promise<void> {
744761
}
745762
}
746763
}
764+
recordAttempt(test.result.state === 'pass' ? 'pass' : 'fail')
747765
if (test.result.state === 'pass') {
748766
break
749767
}

‎packages/vitest/src/runtime/runner/types.ts‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -158,6 +158,24 @@ export interface TaskMeta {
158158
__vitest_label__?: string
159159
}
160160

161+
/**
162+
* The result of an individual test attempt.
163+
*/
164+
export interface TaskResultAttempt {
165+
/** The state after applying the test's expected failure option. */
166+
state: 'pass' | 'fail' | 'skip'
167+
/** Errors produced by this attempt. */
168+
errors?: TestError[]
169+
/** How long in milliseconds the attempt took to run. */
170+
duration: number
171+
/** Time in milliseconds when the attempt started running. */
172+
startTime: number
173+
/** The zero-based retry index within the repeat. */
174+
retryIndex: number
175+
/** The zero-based repeat index. */
176+
repeatIndex: number
177+
}
178+
161179
/**
162180
* The result of calling a task.
163181
*/
@@ -174,6 +192,10 @@ export interface TaskResult {
174192
* if `expect.soft()` failed multiple times or `retry` was triggered.
175193
*/
176194
errors?: TestError[]
195+
/**
196+
* Individual results for every retry and repeat attempt.
197+
*/
198+
attempts?: TaskResultAttempt[]
177199
/**
178200
* How long in milliseconds the task took to run.
179201
*/

‎test/e2e/test/reported-tasks.test.ts‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -276,6 +276,39 @@ it('correctly reports flaky tests', ({ testModule }) => {
276276
const result = testFlaky.result()!
277277
expect(result.state).toBe('passed')
278278
expect(result.errors).toHaveLength(2)
279+
expect(
280+
testFlaky.attempts().map((attempt) => ({
281+
state: attempt.state,
282+
errors: attempt.errors?.map((error) => error.message),
283+
retryIndex: attempt.retryIndex,
284+
repeatIndex: attempt.repeatIndex,
285+
})),
286+
).toMatchInlineSnapshot(`
287+
[
288+
{
289+
"errors": [
290+
"expected +0 to be 2 // Object.is equality",
291+
],
292+
"repeatIndex": 0,
293+
"retryIndex": 0,
294+
"state": "failed",
295+
},
296+
{
297+
"errors": [
298+
"expected 1 to be 2 // Object.is equality",
299+
],
300+
"repeatIndex": 0,
301+
"retryIndex": 1,
302+
"state": "failed",
303+
},
304+
{
305+
"errors": undefined,
306+
"repeatIndex": 0,
307+
"retryIndex": 2,
308+
"state": "passed",
309+
},
310+
]
311+
`)
279312
})
280313

281314
it('correctly reports repeated tests', ({ testModule }) => {
@@ -287,6 +320,65 @@ it('correctly reports repeated tests', ({ testModule }) => {
287320
const result = testRepeated.result()!
288321
expect(result.state).toBe('failed')
289322
expect(result.errors).toHaveLength(6)
323+
expect(
324+
testRepeated.attempts().map((attempt) => ({
325+
state: attempt.state,
326+
errors: attempt.errors?.map((error) => error.message),
327+
retryIndex: attempt.retryIndex,
328+
repeatIndex: attempt.repeatIndex,
329+
})),
330+
).toMatchInlineSnapshot(`
331+
[
332+
{
333+
"errors": [
334+
"expected 1 to be 2 // Object.is equality",
335+
],
336+
"repeatIndex": 0,
337+
"retryIndex": 0,
338+
"state": "failed",
339+
},
340+
{
341+
"errors": [
342+
"expected 1 to be 2 // Object.is equality",
343+
],
344+
"repeatIndex": 1,
345+
"retryIndex": 0,
346+
"state": "failed",
347+
},
348+
{
349+
"errors": [
350+
"expected 1 to be 2 // Object.is equality",
351+
],
352+
"repeatIndex": 2,
353+
"retryIndex": 0,
354+
"state": "failed",
355+
},
356+
{
357+
"errors": [
358+
"expected 1 to be 2 // Object.is equality",
359+
],
360+
"repeatIndex": 3,
361+
"retryIndex": 0,
362+
"state": "failed",
363+
},
364+
{
365+
"errors": [
366+
"expected 1 to be 2 // Object.is equality",
367+
],
368+
"repeatIndex": 4,
369+
"retryIndex": 0,
370+
"state": "failed",
371+
},
372+
{
373+
"errors": [
374+
"expected 1 to be 2 // Object.is equality",
375+
],
376+
"repeatIndex": 5,
377+
"retryIndex": 0,
378+
"state": "failed",
379+
},
380+
]
381+
`)
290382
})
291383

292384
it('correctly passed down metadata', ({ testModule }) => {

0 commit comments

Comments
 (0)