Logging & Debugging

Logging & Debugging#

MassGen provides comprehensive logging to help you understand agent coordination, debug issues, and review decision-making processes.

Logging Directory Structure#

All logs are stored in the .massgen/massgen_logs/ directory with timestamped subdirectories:

.massgen/
└── massgen_logs/
    └── log_YYYYMMDD_HHMMSS/           # Timestamped log directory
        ├── agent_a/                    # Agent-specific coordination logs
        │   └── YYYYMMDD_HHMMSS_NNNNNN/ # Timestamped coordination steps
        │       ├── answer.txt          # Agent's answer at this step
        │       ├── context.txt         # Context available to agent
        │       ├── execution_trace.md  # Full tool calls, results, and reasoning
        │       └── workspace/          # Agent workspace (if filesystem tools used)
        ├── agent_b/                    # Second agent's logs
        │   └── ...
        ├── agent_outputs/              # Consolidated output files
        │   ├── agent_a.txt             # Complete output from agent_a
        │   ├── agent_b.txt             # Complete output from agent_b
        │   ├── final_presentation_agent_X.txt  # Winning agent's final answer
        │   ├── final_presentation_agent_X_latest.txt  # Symlink to latest
        │   └── system_status.txt       # System status and metadata
        ├── final/                      # Final presentation phase
        │   └── agent_X/                # Winning agent's final work
        │       ├── answer.txt          # Final answer
        │       └── context.txt         # Final context
        ├── coordination_events.json    # Structured coordination events
        ├── coordination_table.txt      # Human-readable coordination table
        ├── vote.json                   # Final vote tallies and consensus data
        ├── massgen.log                 # Complete debug log (or massgen_debug.log in debug mode)
        ├── snapshot_mappings.json      # Workspace snapshot metadata
        └── execution_metadata.yaml     # Query, config, and execution details

Note

When agents use filesystem tools, each coordination step will also contain a workspace/ directory showing the files the agent created or modified during that step.

Per-Attempt Logging (Orchestration Restart)#

When orchestration restart is enabled, each restart attempt gets its own isolated directory:

.massgen/massgen_logs/log_YYYYMMDD_HHMMSS/
├── attempt_1/          # First attempt (complete log structure)
├── attempt_2/          # Second attempt after restart
├── attempt_3/          # Third attempt if needed
└── final/              # Copy of accepted result

For multi-turn: turn_1/attempt_1/, turn_1/attempt_2/, turn_1/final/

Log Files Explained#

Agent Coordination Logs#

Location: agent_<id>/YYYYMMDD_HHMMSS_NNNNNN/

Each coordination step gets a timestamped directory containing:

answer.txt - The agent’s answer/proposal at this step
context.txt - What answers/context the agent could see (recent answers from other agents)
execution_trace.md - Complete execution history with full tool calls, results, and reasoning

Use cases:

Review what each agent proposed during coordination
Understand how agents’ thinking evolved as they saw other agents’ work
Debug why specific decisions were made

Execution Traces#

Location: agent_<id>/YYYYMMDD_HHMMSS_NNNNNN/execution_trace.md

Execution traces are the most detailed debug artifacts available. Each trace captures the complete execution history for that answer/vote iteration:

Contents:

Tool calls - Complete tool names and arguments (not truncated)
Tool results - Full output from each tool (not truncated)
Reasoning blocks - Model’s internal thinking/chain-of-thought (if available)
Round markers - Which coordination round the activity occurred in
Timestamps - When each action occurred

Example execution trace:

# Execution Trace: agent_a
**Model**: gemini-2.5-flash | **Started**: 2025-01-10 13:56:31

## Round 1 (Answer 1.1)

### Tool Call: mcp__filesystem__read_file
**Args**:
```json
{"path": "/workspace/main.py"}
```

### Tool Result: mcp__filesystem__read_file
```
def main():
    print("Hello world")
    # ... full file content
```

### Reasoning
I need to understand the existing code structure before making changes.
The main.py file shows a simple entry point...

### Answer Submitted (1.1)
Created the requested feature with proper error handling...

Use cases:

Deep debugging - See exactly what an agent did and why
Compression recovery - Agents can read their own trace to recover lost context
Cross-agent analysis - Understand how other agents approached the problem
Tool failure analysis - Full arguments and error messages for failed tools

Accessing traces:

# View an agent's execution trace for a specific step
cat .massgen/massgen_logs/log_20251010_135631/agent_a/20251010_135655_287787/execution_trace.md

# Search for specific tool calls across all traces
grep -r "Tool Call:" .massgen/massgen_logs/log_*/agent_*/*/execution_trace.md

# Find traces with errors
grep -l "Tool Error:" .massgen/massgen_logs/log_*/agent_*/*/execution_trace.md

Consolidated Agent Outputs#

Location: agent_outputs/

Contains merged outputs from all coordination rounds:

agent_<id>.txt - Complete output history for each agent
final_presentation_agent_<id>.txt - Winning agent’s final presentation
final_presentation_agent_<id>_latest.txt - Symlink to latest (for automation)
system_status.txt - System metadata and status

Final Presentation#

Location: final/agent_<id>/

The winning agent’s final answer after coordination:

answer.txt - Complete final answer
context.txt - Final context used for presentation

Coordination Events#

Location: coordination_events.json

Structured JSON log of all coordination events:

{
  "event_id": "E42",
  "timestamp": "2025-10-08T01:40:29",
  "agent_id": "agent_a",
  "event_type": "vote",
  "data": {
    "vote_for": "agent_b.2",
    "reason": "More comprehensive approach..."
  }
}

Event types:

started_streaming - Agent begins thinking
new_answer - Agent provides labeled answer
vote - Agent votes for an answer
restart - Agent requests restart
restart_completed - Agent finishes restart
final_answer - Winner provides final response

Vote Summary#

Location: vote.json

Final vote tallies and consensus information:

{
  "votes": {
    "agent_a": {
      "voted_for": "agent_b",
      "reason": "More comprehensive analysis"
    },
    "agent_b": {
      "voted_for": "agent_b",
      "reason": "Best captures key insights"
    }
  },
  "winner": "agent_b",
  "consensus_reached": true
}

Use cases:

Understand final consensus decision
Review voting patterns across agents
Analyze decision-making rationale

Main Debug Log#

Location: massgen.log

Complete debug log with all system operations:

Backend API calls and responses
Tool usage and results
Coordination state transitions
Error messages and stack traces

Enable with --debug flag for verbose logging.

Execution Metadata#

Location: execution_metadata.yaml

This file captures the complete execution context for reproducibility:

query: "Your original question"
timestamp: "2025-10-13T14:30:22"
config_path: "/path/to/config.yaml"
config:
  agents:
    - id: "agent1"
      backend:
        type: "gemini"
        model: "gemini-2.5-flash"
    # ... full config
cli_args:
  config: "/path/to/config.yaml"
  question: "Your original question"
  debug: false
  # ... all CLI arguments
git:
  commit: "a1b2c3d4e5f6..."
  branch: "main"
python_version: "3.13.0"
massgen_version: "0.0.33"
working_directory: "/path/to/project"

Contents:

query - The user’s original query/prompt
timestamp - When the execution started (ISO 8601 format)
config_path - Path or description of config used
config - Complete configuration (full YAML/JSON content)
cli_args - All command-line arguments passed to massgen
git - Git repository info (commit hash, branch) if in a git repo
python_version - Python interpreter version
massgen_version - MassGen package version
working_directory - Current working directory

Use cases:

Reproduce the exact same run - All information needed to recreate execution
Debug configuration issues - Full config and CLI args captured
Share execution details - Send metadata file to team members
Create test cases - Convert real runs into regression tests
Track experiments - Git commit ensures you know which code version was used
Environment debugging - Python version and working directory help diagnose environment issues

Multi-turn sessions:

For interactive multi-turn mode, each turn gets its own execution_metadata.yaml with additional fields:

# ... standard fields above ...
cli_args:
  mode: "interactive"
  turn: 3
  session_id: "session_20251013_143022"

Coordination Table#

The coordination table (coordination_table.txt) is a human-readable visualization of the entire multi-agent coordination process.

Structure#

+-------------------------------------------------------------------+
|   Event  |           Agent 1           |           Agent 2           |
|----------+-----------------------------+-----------------------------+
|   USER   | Original user question                                     |
|==========+=============================+=============================+
|     E1   |     📋 Context: []          |      ⏳ (waiting)            |
|          |  💭 Started streaming       |                             |
|----------+-----------------------------+-----------------------------+
|     E2   |     🔄 (streaming)          |   ✨ NEW ANSWER: agent2.1   |
|          |                             |👁️  Preview: Summary...      |
|----------+-----------------------------+-----------------------------+

Key sections:

Header - Event symbols, status symbols, and terminology
Event log - Chronological coordination events
Summary - Final statistics per agent
Totals - Overall coordination metrics

Event Symbols#

Actions:

💭 Started streaming - Agent begins thinking/processing
✨ NEW ANSWER - Agent provides a labeled answer
🗳️ VOTE - Agent votes for an answer
💭 Reason - Reasoning behind the vote
👁️ Preview - Content of the answer
🔁 RESTART TRIGGERED - Agent requests to restart
✅ RESTART COMPLETED - Agent finishes restart
🎯 FINAL ANSWER - Winner provides final response
🏆 Winner selected - System announces winner

Status:

💭 (streaming) - Currently thinking/processing
⏳ (waiting) - Idle, waiting for turn
✅ (answered) - Has provided an answer
✅ (voted) - Has cast a vote
✅ (completed) - Task completed
🎯 (final answer given) - Winner completed final answer

Answer Labels#

Each answer gets a unique identifier:

Format: agent{N}.{attempt}

N = Agent number (1, 2, 3…)
attempt = New answer number (1, 2, 3…)

Examples:

agent1.1 = Agent 1’s first answer
agent2.1 = Agent 2’s first answer
agent1.2 = Agent 1’s second answer (after restart)
agent1.final = Agent 1’s final answer (if winner)

Coordination Flow#

The table shows how agents coordinate:

Agents see recent answers - Each agent can view the most recent answers from other agents
Decide next action - Each agent chooses to either:
- Provide a new/refined answer
- Vote for an existing answer they think is best
All agents vote - Coordination continues until all agents have voted
Final presentation - The agent with the most votes delivers the final answer

Example interpretation:

E7: Agent 1 provides answer agent1.1
E13: Agent 1 votes for agent1.1 (self-vote)
E19: Agent 2 votes for agent1.1 (consensus!)
E39: Agent 1 selected as winner
E39: Agent 1 provides final answer

What agents see:

During coordination, agents see snapshots of each other’s work through workspace snapshots and answer context. This allows agents to build on insights, catch errors, and converge on the best solution.

Summary Statistics#

At the bottom of the coordination table:

Metric	Description
Answers	Number of distinct answers provided
Votes	Number of votes cast
Restarts	Number of times agent restarted (cleared memory)
Status	Final completion status

Accessing Logs#

Log Analysis Commands#

MassGen provides the massgen logs command for quick log analysis without manual file navigation.

Summary of most recent run:

massgen logs

# Example output:
# ╭──────────────────────────── MassGen Run Summary ─────────────────────────────╮
# │ Create a website about Bob Dylan                                             │
# │                                                                               │
# │ Winner: agent_a | Agents: 1 | Duration: 7.2m | Cost: $0.54                   │
# ╰───────────────────────────────────────────────────────────────────────────────╯
#
# Tokens: Input: 6,035,629 | Output: 21,279 | Reasoning: 7,104
#
# Rounds (5): answer: 1 | vote: 1 | presentation: 2 | post_evaluation: 1
#   Errors: 0 | Timeouts: 0
#
# ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━┳━━━━━━┓
# ┃ Tool                                      ┃ Calls ┃  Time ┃  Avg ┃ Fail ┃
# ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━╇━━━━━━┩
# │ mcp__command_line__execute_command        │    47 │  4.4s │ 94ms │      │
# │ mcp__planning__update_task_status         │    13 │ 228ms │ 18ms │      │
# └───────────────────────────────────────────┴───────┴───────┴──────┴──────┘

Available subcommands:

Command	Description
`massgen logs` or `massgen logs summary`	Display run summary with tokens, rounds, and top tools
`massgen logs tools`	Full tool breakdown table sorted by execution time
`massgen logs tools --sort calls`	Sort tools by call count instead of time
`massgen logs list`	List recent runs with timestamps, costs, and questions
`massgen logs list --limit 20`	Show more runs (default: 10)
`massgen logs open`	Open log directory in system file manager (Finder/Explorer)

Filtering by analysis status:

# Show which logs have been analyzed (have ANALYSIS_REPORT.md)
massgen logs list                    # Shows "Analyzed" column with ✓ for analyzed logs
massgen logs list --analyzed         # Only logs with ANALYSIS_REPORT.md
massgen logs list --unanalyzed       # Only logs without analysis

Common options:

# Analyze a specific log directory
massgen logs --log-dir .massgen/massgen_logs/log_20251218_134125_867383/turn_1/attempt_1

# Output raw JSON for scripting
massgen logs summary --json

Tool breakdown example:

massgen logs tools

# ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━┳━━━━━━┓
# ┃ Tool                                      ┃ Calls ┃  Time ┃  Avg ┃ Fail ┃
# ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━╇━━━━━━┩
# │ mcp__command_line__execute_command        │    47 │  4.4s │ 94ms │      │
# │ mcp__planning__update_task_status         │    13 │ 228ms │ 18ms │      │
# │ mcp__filesystem__write_file               │     7 │ 181ms │ 26ms │      │
# │ mcp__planning__create_task_plan           │     2 │  36ms │ 18ms │      │
# ├───────────────────────────────────────────┼───────┼───────┼──────┼──────┤
# │ TOTAL                                     │    69 │  4.8s │      │      │
# └───────────────────────────────────────────┴───────┴───────┴──────┴──────┘

List recent runs:

massgen logs list --limit 5

# ┏━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
# ┃ # ┃ Timestamp        ┃ Duration ┃  Cost ┃ Analyzed ┃ Question                    ┃
# ┡━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
# │ 1 │ 2025-12-18 13:41 │     7.2m │ $0.54 │    ✓     │ Create a website about...   │
# │ 2 │ 2025-12-17 23:01 │    16.2m │ $1.23 │    -     │ Build a REST API...         │
# │ 3 │ 2025-12-17 22:30 │     3.1m │ $0.12 │    ✓     │ Explain quantum computing...│
# └───┴──────────────────┴──────────┴───────┴──────────┴─────────────────────────────┘

Analyzing Logs#

The massgen logs analyze command helps you generate analysis reports for log sessions.

Generate analysis prompt (for coding CLIs):

# Generate a prompt to use in Claude Code, Cursor, etc.
massgen logs analyze                 # Analyze latest log
massgen logs analyze --log-dir PATH  # Analyze specific log

This outputs a prompt that references the massgen-log-analyzer skill, which you can paste into your coding CLI.

Run multi-agent self-analysis:

# Run MassGen with 3 agents to analyze the log from different perspectives
massgen logs analyze --mode self

# Choose UI mode (default: rich_terminal)
massgen logs analyze --mode self --ui automation   # Headless mode
massgen logs analyze --mode self --ui webui        # Web UI mode

# Use custom analysis config
massgen logs analyze --mode self --config my_analysis.yaml

Self-analysis mode:

Runs a 2-agent team using Gemini Flash with Docker execution
Agents analyze from different perspectives (correctness, efficiency, behavior)
Produces an ANALYSIS_REPORT.md in the log directory
Log directory is mounted read-only to protect existing files

Note

Self-analysis mode currently requires a Gemini API key (GEMINI_API_KEY); to use other models, see massgen/configs/analysis/log_analysis.yaml then adjust it or create a new one and pass it to the analyze command using –config For Logfire integration, also set LOGFIRE_READ_TOKEN in your .env file. Without it, agents will use local log files only.

During Execution#

Press ‘r’ key during execution to view real-time coordination table in your terminal.

After Execution#

Find latest log directory:

# Using massgen logs open (recommended)
massgen logs open

# Or manually
ls -t .massgen/massgen_logs/ | head -1

View coordination table:

cat .massgen/massgen_logs/log_20251008_013641/coordination_table.txt

View specific agent output:

cat .massgen/massgen_logs/log_20251008_013641/agent_outputs/agent_a.txt

View final answer:

cat .massgen/massgen_logs/log_20251008_013641/agent_outputs/final_presentation_*_latest.txt

Debug Mode#

Enable detailed logging with the --debug flag:

uv run python -m massgen.cli \
  --debug \
  --config your_config.yaml \
  "Your question"

What debug mode logs:

✅ Full API request/response bodies
✅ Tool call arguments and results
✅ Coordination state transitions
✅ File operation details
✅ MCP server communication
✅ Error stack traces

Debug log location: .massgen/massgen_logs/log_YYYYMMDD_HHMMSS/massgen_debug.log

Common Debugging Scenarios#

Agent Not Converging#

Check: coordination_table.txt

Look for:

Agents changing votes frequently
New answers in every round
No clear vote majority

Solution: Review agent answers to understand disagreement points.

Agent Errors#

Check: massgen.log for error messages

Search for:

grep -i "error" .massgen/massgen_logs/log_*/massgen.log
grep -i "exception" .massgen/massgen_logs/log_*/massgen.log

Tool Failures#

Check: agent_outputs/agent_<id>.txt

Look for tool call failures and error messages.

Also check: massgen.log for detailed tool execution logs

Understanding Agent Decisions#

Review coordination rounds:

Open coordination_table.txt
Find the round where decision changed
Check agent_<id>/YYYYMMDD_HHMMSS_NNNNNN/context.txt to see what the agent could see
Check agent_<id>/YYYYMMDD_HHMMSS_NNNNNN/answer.txt for the agent’s reasoning
Check agent_<id>/YYYYMMDD_HHMMSS_NNNNNN/execution_trace.md for complete tool usage and thinking

Performance Analysis#

Check summary statistics in coordination_table.txt:

High restart count = Agents changing approach frequently
Low vote count = Quick consensus
Many answers = Iterative refinement

Log Retention#

Logs are stored indefinitely by default.

Clean old logs manually:

# Remove logs older than 7 days
find .massgen/massgen_logs/ -type d -name "log_*" -mtime +7 -exec rm -rf {} +

Disk space check:

du -sh .massgen/massgen_logs/

Best Practices#

Review coordination table first - Best overview of what happened
Use debug mode for troubleshooting - Full details when needed
Archive important logs - Move successful runs to separate directory
Check final presentation - Verify winning agent’s work quality
Monitor log size - Clean old logs periodically

Integration with CI/CD#

Automated log parsing:

import json

# Parse coordination events
with open(".massgen/massgen_logs/log_latest/coordination_events.json") as f:
    events = json.load(f)

# Extract final answer
with open(".massgen/massgen_logs/log_latest/agent_outputs/final_presentation_*_latest.txt") as f:
    final_answer = f.read()

Exit status:

MassGen exits with status 0 on success, non-zero on failure.

uv run python -m massgen.cli --config config.yaml "Question" && echo "Success"

Logfire Observability#

MassGen supports Logfire for advanced structured tracing and observability.

Note

Logfire is an optional dependency. Install it with:

pip install "massgen[observability]"

# Or with uv
uv pip install "massgen[observability]"

When enabled, Logfire provides:

Automatic LLM instrumentation - Traces all OpenAI and Anthropic API calls with request/response details
Tool execution tracing - Spans for MCP tool calls with timing and success/failure metrics
Coordination events - Structured logs for agent coordination, voting, and winner selection
Token usage metrics - Detailed tracking of input/output/reasoning/cached tokens
Integrated with loguru - All existing log messages flow through Logfire when enabled

Enabling Logfire#

Via CLI flag (recommended):

massgen --logfire --config your_config.yaml "Your question"

Via environment variable:

export MASSGEN_LOGFIRE_ENABLED=true
massgen --config your_config.yaml "Your question"

Setting Up Logfire#

Install MassGen with observability support:

pip install "massgen[observability]"

# Or with uv
uv pip install "massgen[observability]"

Create a Logfire account at https://logfire.pydantic.dev/

Authenticate with Logfire:

# Authenticate (this creates ~/.logfire/credentials.json)
uv run logfire auth

Alternatively, set the token directly:
```
export LOGFIRE_TOKEN=your_token_here
```

Run MassGen with Logfire enabled:

massgen --logfire --config your_config.yaml "Your question"

What Gets Traced#

When Logfire is enabled, MassGen automatically traces:

LLM API Calls:

All requests to OpenAI-compatible APIs (GPT-4, etc.)
All requests to Anthropic Claude API
All requests to Google GenAI (Gemini) API
Request parameters, response content, and timing
Token usage breakdown

Note

For Gemini tracing, set OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true to capture full prompts and completions. Without this, content appears as <elided>.

Tool Executions:

MCP server tool calls with full input/output
Custom tools (like read_media, write_file, etc.)
Agent attribution via massgen.agent_id span attribute
Execution time in milliseconds
Success/failure status
Error messages when tools fail

Coordination Events:

coordination_started - When agent coordination begins
winner_selected - When voting completes and a winner is chosen
Vote counts and participating agents

Example Logfire Dashboard View:

┌─ coordination.session (45.2s) ──────────────────────────────┐
│  task: "Build a REST API for user management"              │
│  num_agents: 3                                              │
│  agent_ids: agent_a, agent_b, agent_c                      │
│                                                             │
│  ├─ llm.call [claude-3-5-sonnet] (3.1s)                   │
│  │   input_tokens: 1,234                                   │
│  │   output_tokens: 567                                    │
│  │                                                         │
│  ├─ mcp.filesystem.write_file (0.8s)                      │
│  │   input_chars: 245                                      │
│  │   output_chars: 12                                      │
│  │   success: true                                         │
│  │                                                         │
│  ├─ [info] Agent answer: agent1.1                         │
│  │   agent_id: agent_a, iteration: 1, round: 1            │
│  │                                                         │
│  ├─ llm.call [gpt-4] (3.5s)                               │
│  │   input_tokens: 2,456                                   │
│  │   output_tokens: 823                                    │
│  │                                                         │
│  ├─ [info] Agent answer: agent2.1                         │
│  │   agent_id: agent_b, iteration: 1, round: 1            │
│  │                                                         │
│  ├─ [info] Agent vote: agent_a -> agent2.1                │
│  │   reason: "More comprehensive solution"                │
│  │                                                         │
│  ├─ [info] Agent vote: agent_b -> agent2.1                │
│  │                                                         │
│  └─ [info] Winner selected: agent2.1                      │
│      vote_counts: {agent2.1: 2}                           │
└────────────────────────────────────────────────────────────┘

What Gets Logged (Meaningful Events Only):

To reduce noise, MassGen only logs meaningful coordination events:

Session span (coordination.session) - Top-level span for the entire coordination
LLM API calls - Automatic instrumentation of OpenAI, Anthropic, and Gemini calls
Tool executions - MCP tool calls with input/output sizes and timing
Agent answers - When an agent provides a new answer (with label like agent1.1)
Agent votes - When an agent casts a vote (with reason)
Winner selection - When voting completes and winner is determined
Final answer - When the winning agent presents the final response

Note: Individual coordination iterations are tracked internally but not logged to Logfire to avoid cluttering the trace with less useful information.

Tool Execution Attributes#

All tool execution events include rich attributes for filtering, grouping, and debugging:

Attribute	Description
`agent_id`	The ID of the agent executing the tool (e.g., `agent_a`, `agent_b`)
`tool_name`	The full tool name (e.g., `mcp__filesystem__write_file`)
`tool_type`	Tool category: `mcp` for MCP tools, `custom` for built-in tools
`success`	Boolean indicating whether the tool call succeeded
`execution_time_ms`	Execution time in milliseconds
`input_chars`	Number of characters in the tool input/arguments
`output_chars`	Number of characters in the tool output/result
`error_message`	Error message if the tool call failed (null on success)
`server_name`	MCP server name for MCP tools (e.g., `filesystem`, `command_line`)
`arguments_preview`	First 200 characters of tool arguments (for pattern analysis)
`output_preview`	First 200 characters of tool output (for debugging)
`round_number`	Which coordination round the tool was called in (0, 1, 2, …)
`round_type`	Type of round: `initial_answer`, `voting`, `presentation`

LLM API Call Attributes#

All LLM API call spans include these attributes for agent attribution:

Attribute	Description
`massgen.agent_id`	The ID of the agent making the call
`llm.provider`	Provider name (`anthropic`, `openai`, `gemini`, etc.)
`llm.model`	Model being called (`claude-3-opus`, `gpt-4o`, etc.)
`llm.operation`	API operation type (typically `stream`)
`gen_ai.system`	OpenTelemetry semantic convention for provider
`gen_ai.request.model`	OpenTelemetry semantic convention for model

Example Logfire Queries#

These attributes enable powerful filtering and analysis in the Logfire dashboard:

Find slowest tool calls:

SELECT
  attributes->>'tool.name' as tool_name,
  (attributes->>'tool.execution_time_ms')::float as execution_time_ms,
  attributes->>'massgen.agent_id' as agent_id
FROM records
WHERE attributes->>'tool.type' = 'mcp'
ORDER BY (attributes->>'tool.execution_time_ms')::float DESC

Find failed tools with their arguments:

SELECT
  attributes->>'tool.name' as tool_name,
  attributes->>'tool.arguments_preview' as arguments_preview,
  attributes->>'tool.error_message' as error_message,
  attributes->>'massgen.agent_id' as agent_id
FROM records
WHERE attributes->>'tool.success' = 'false'

Tools with large outputs (potential cost drivers):

SELECT
  attributes->>'mcp.server' as server_name,
  attributes->>'tool.name' as tool_name,
  (attributes->>'tool.output_chars')::int as output_chars,
  attributes->>'massgen.agent_id' as agent_id
FROM records
WHERE (attributes->>'tool.output_chars')::int > 10000
ORDER BY (attributes->>'tool.output_chars')::int DESC

Pattern analysis - which arguments lead to failures:

SELECT
  attributes->>'tool.arguments_preview' as arguments_preview,
  COUNT(*) as fail_count
FROM records
WHERE attributes->>'tool.success' = 'false'
GROUP BY attributes->>'tool.arguments_preview'
ORDER BY fail_count DESC

Tool usage by MCP server:

SELECT
  attributes->>'mcp.server' as server_name,
  COUNT(*) as calls,
  AVG((attributes->>'tool.execution_time_ms')::float) as avg_time_ms
FROM records
WHERE attributes->>'tool.type' = 'mcp'
GROUP BY attributes->>'mcp.server'

LLM calls by agent:

SELECT
  attributes->>'massgen.agent_id' as agent_id,
  attributes->>'llm.model' as model,
  COUNT(*) as calls
FROM records
WHERE span_name LIKE 'llm.%'
GROUP BY attributes->>'massgen.agent_id', attributes->>'llm.model'

All activity for a specific agent:

SELECT span_name, start_timestamp, duration
FROM records
WHERE attributes->>'massgen.agent_id' = 'agent_a'
ORDER BY start_timestamp

Environment Variables#

Variable	Description
`MASSGEN_LOGFIRE_ENABLED`	Set to `true` to enable Logfire (alternative to `--logfire` flag)
`LOGFIRE_TOKEN`	Your Logfire API token (if not using `logfire auth login`)
`LOGFIRE_SERVICE_NAME`	Override the service name (default: `massgen`). Read by Logfire library.
`LOGFIRE_ENVIRONMENT`	Set environment tag (e.g., `production`, `development`). Read by Logfire library.
`OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT`	Set to `true` to capture Gemini prompts/completions (otherwise shows `<elided>`)

Programmatic Usage#

When using MassGen as a library, you can configure Logfire programmatically:

from massgen.structured_logging import configure_observability

# Enable observability with custom settings
configure_observability(
    enabled=True,
    service_name="my-app",
    environment="production",
)

# Now run your orchestrator
from massgen.orchestrator import Orchestrator
orchestrator = Orchestrator(config)
result = await orchestrator.run("Your question")

Graceful Degradation#

Logfire integration is designed to be non-intrusive:

Logfire not installed? - You’ll see a helpful message: ⚠️ Logfire not installed. Install with: pip install massgen[observability]
Not authenticated? - You’ll see: Logfire requires authentication. Run 'logfire auth' to authenticate
Logfire disabled? - All logging falls back to standard loguru
Network issues? - Logfire handles connectivity gracefully

This means you can always enable the --logfire flag without worrying about breaking your workflow - it will show helpful guidance if Logfire needs to be set up.

Logging & Debugging

Contents

Logging & Debugging#

Logging Directory Structure#

Per-Attempt Logging (Orchestration Restart)#

Log Files Explained#

Agent Coordination Logs#

Execution Traces#

Consolidated Agent Outputs#

Final Presentation#

Coordination Events#

Vote Summary#

Main Debug Log#

Execution Metadata#

Coordination Table#

Structure#

Event Symbols#

Answer Labels#

Coordination Flow#

Summary Statistics#

Accessing Logs#

Log Analysis Commands#

Analyzing Logs#

During Execution#

After Execution#

Debug Mode#

Common Debugging Scenarios#

Agent Not Converging#

Agent Errors#

Tool Failures#

Understanding Agent Decisions#

Performance Analysis#

Log Retention#

Best Practices#

Integration with CI/CD#

Sharing Sessions#

Prerequisites#

Sharing a Session#

Managing Shared Sessions#

Authentication Errors#

Logfire Observability#

Enabling Logfire#

Setting Up Logfire#

What Gets Traced#

Tool Execution Attributes#

LLM API Call Attributes#

Example Logfire Queries#

Environment Variables#

Programmatic Usage#

Graceful Degradation#

See Also#