Skip to content

Add support for AI agent environment variables to quiet test output#21135

Merged
Jarred-Sumner merged 9 commits into
mainfrom
claude/add-claudecode-flag
Jul 18, 2025
Merged

Add support for AI agent environment variables to quiet test output#21135
Jarred-Sumner merged 9 commits into
mainfrom
claude/add-claudecode-flag

Conversation

@robobun

@robobun robobun commented Jul 17, 2025 •
edited by Jarred-Sumner
Loading

Copy link
Copy Markdown
Collaborator

Summary

This PR adds support for AI agent environment variables to enable quiet test output in Bun's test runner. When an AI agent is detected through environment variables like CLAUDECODE=1, REPL_ID=1, or IS_CODE_AGENT=1, the test runner reduces verbosity by only showing failures while still maintaining accurate test summary counts.

Changes

  • Uses existing Output.isAIAgent() function in src/output.zig which detects:
    • CLAUDECODE=1 - Claude Code
    • REPL_ID=1 - Replit
    • IS_CODE_AGENT=1 - Generic AI agent flag
    • (Future support for Gemini, Codex, and Cursor background agents)
  • Modified test output functions in src/cli/test_command.zig:
    • writeTestStatusLine() - Skip non-failure status indicators in quiet mode
    • printTestLine() - Skip non-failure test line output (with full JUnit support)
  • Added comprehensive test suite with snapshots in test/cli/test/claudecode-flag.test.ts

Behavior

When AI agent is detected:

  • Only test failures are displayed with their full output
  • Passing, skipped, and todo test indicators are hidden
  • Summary statistics are still shown at the end
  • JUnit reporter output remains unaffected
  • Error messages are more concise (e.g., "0 test files matching..." instead of friendly hints)

When no AI agent is detected:

  • All test output is shown normally (existing behavior)

Example Output

Normal mode (no AI agent):

test.js:
(pass) passing test [1.00ms]
(fail) failing test [2.00ms]
(skip) skipped test
(todo) todo test

 1 pass
 1 skip
 1 todo
 1 fail

Quiet mode (AI agent detected):

test.js:
(fail) failing test [2.00ms]

 1 pass
 1 skip
 1 todo
 1 fail

Test plan

  • All existing tests pass
  • New comprehensive test suite with snapshots
  • Comparison tests between AI agent and normal modes
  • Debug build compilation successful
  • JUnit reporter functionality preserved
  • Summary statistics accuracy maintained

This feature is useful for AI coding assistants where reduced output verbosity improves readability and context efficiency while still providing essential failure information.

🤖 Generated with Claude Code

Claude Bot @claude
Add CLAUDECODE=1 environment variable to quiet test output
dfb63f8 
This adds support for the CLAUDECODE environment variable to enable quiet test output,
which reduces verbosity by only showing failures while still maintaining accurate
test summary counts.

When CLAUDECODE=1 is set:
- Only test failures are displayed with their full output
- Passing, skipped, and todo test indicators are hidden
- Summary statistics are still shown at the end
- JUnit reporter output remains unaffected

When CLAUDECODE=0 or unset:
- All test output is shown normally (existing behavior)

This feature is useful for automated testing environments where reduced output
verbosity improves readability while still providing essential failure information.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
@robobun

robobun commented Jul 17, 2025 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author
Updated 8:31 PM PT - Jul 17th, 2025

@Jarred-Sumner, your commit 25d749a is building: #20874

@RiskyMH

RiskyMH commented Jul 17, 2025

Copy link
Copy Markdown
Contributor

this probably should also force disable ansi (if it somehow gets enabled, as its a waste of its tokens)

RiskyMH
RiskyMH reviewed Jul 17, 2025
View reviewed changes
Comment thread src/cli/test_command.zig Outdated
Comment on lines 652 to 656

@RiskyMH RiskyMH Jul 17, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it should just always run this code here then? not too good to duplicate the code

Claude Bot @claude
Enhance CLAUDECODE=1 to suppress filenames until first failure
45af2b5 
- Add filename suppression logic that only shows file paths when there's a failure
- Remove duplicate JUnit reporting code for cleaner implementation
- Track current file info in CommandLineReporter for deferred filename printing
- Add printFilenameIfNeeded function to print filename only when needed
- Maintain full JUnit reporting functionality in quiet mode

This makes the quiet mode even more focused - showing only essential failure
information while completely hiding successful test output.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
@robobun
robobun force-pushed the claude/add-claudecode-flag branch from 5c480c2 to 45af2b5 Compare July 17, 2025 15:42
@mizulu

mizulu commented Jul 17, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor
  1. This feature seem vaguely familiar
    https://github.com/oven-sh/bun/compare/claude/implement-agent-flag

there they use --agent
which I think is slightly better, as it is more generic(for the day you switch to another agent)

  1. some test runners implements flags to control that even more generically
    which is what we should have for bun test runner

Jest : https://jestjs.io/docs/cli#--onlyfailures

--onlyFailures

Vitest

--silent   // to remove all outputs 
--silent passed-only  //  let errors passthrough

--no-color
https://vitest.dev/guide/cli.html#color

I think we need those explicit flags so we have more control future wise
as this feature might not be just agent specific, and can be useful
for users as well, especially the --onlyFailures

after we have proper options for the test runner,
then feel free to add some AGENT env with some convenience defaults

@mizulu mizulu mentioned this pull request Jul 17, 2025
Closed
@190n

This comment was marked as resolved.

Sign in to view
@Jarred-Sumner Jarred-Sumner changed the title Add CLAUDECODE=1 environment variable to quiet test output Add support for AI agent environment variables to quiet test output Jul 18, 2025
@Jarred-Sumner
Jarred-Sumner merged commit 851fa7d into main Jul 18, 2025
52 of 57 checks passed
@Jarred-Sumner
Jarred-Sumner deleted the claude/add-claudecode-flag branch July 18, 2025 04:20
robobun pushed a commit to robobun/bun that referenced this pull request Jul 18, 2025
Claude Bot @claude
Add documentation for AI agent environment variables in test runner
ea83418 
Documents the AI agent integration feature that enables quieter test output
for AI coding assistants. This feature was added in PR oven-sh#21135 and supports
environment variables like CLAUDECODE=1, REPL_ID=1, and IS_CODE_AGENT=1
to minimize test output verbosity while preserving failure information.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
@robobun robobun mentioned this pull request Jul 18, 2025
Merged
3 tasks
@robobun robobun added the claude label Aug 14, 2025
@hi-ogawa hi-ogawa mentioned this pull request Aug 20, 2025
Closed
4 tasks
This was referenced Mar 2, 2026
Open
Open
@sleitor sleitor mentioned this pull request Mar 2, 2026
Open
@sw-square sw-square mentioned this pull request Mar 3, 2026
Closed
6 tasks
@blva blva mentioned this pull request Jul 6, 2026
Draft
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@RiskyMH RiskyMH RiskyMH left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

claude

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@robobun @RiskyMH @mizulu @190n @Jarred-Sumner