fix(crewai): close LLM span on LLMCallFailedEvent - #3368

Summary

When a CrewAI LLM call fails, CrewAI emits LLMCallFailedEvent instead of LLMCallCompletedEvent and then re-raises (crewai/llms/base_llm.py, _emit_call_failed_event). The CrewAI event listener had no handler for the failed event, so the Observer entered in on_llm_started was never exited.

Why

• on_llm_started (deepeval/integrations/crewai/handler.py:182) stores an Observer in span_observers, keyed by llm_{id(source)}_{event.call_id}, and calls observer.__enter__(). Only on_llm_completed (deepeval/integrations/crewai/handler.py:242) popped and exited that observer, so on a failed call:
  • the LLM span stayed open in trace_manager.active_spans forever (no end_time, no status);
  • Observer.__exit__ only calls trace_manager.end_trace when no other span of the trace is still active (deepeval/tracing/tracing.py:1333-1341), so once the root span closed, the whole trace stayed in active_traces and was never finalized/posted;
  • the span_observers entry was never removed and accumulated on every failed call.
• The same asymmetry existed for tools: on_tool_started (deepeval/integrations/crewai/handler.py:327) appends metadata to tool_observers_stack, but CrewAI emits ToolUsageErrorEvent instead of ToolUsageFinishedEvent when a tool raises, so the entry was never popped.

Verified on the unpatched tree by emitting LLMCallStartedEvent + LLMCallFailedEvent for one call and exiting the root Observer: the LlmSpan remains with end_time=None, the trace remains in active_traces, and span_observers still holds the entry.

Changes

• deepeval/integrations/crewai/handler.py: add an on_llm_failed listener that pops the observer, marks the span ERRORED with the event's error through observer.__exit__, and restores the span context the same way on_llm_completed does; add an on_tool_error listener that pops the stale metadata entry from tool_observers_stack.
• tests/test_integrations/test_crewai/test_crewai_error_events.py: regression tests that emit the started/failed and started/error event pairs through crewai_event_bus directly (no API key required) and assert the span is closed as ERRORED with the error recorded, the trace finalizes once the root span exits, and no observer state leaks.

Testing

With crewai==1.15.22 (the handler's call_id-keyed events match crewai 1.x):

• New tests, before this fix: pytest tests/test_integrations/test_crewai/test_crewai_error_events.py → 2 failed:
  • test_llm_call_failed_event_closes_llm_span_and_finalizes_trace — AssertionError: assert 1 == 0 (LlmSpan still in trace_manager.active_spans after the root span exited)
  • test_tool_usage_error_event_clears_pending_tool_metadata — AssertionError: assert not True (metadata still in tool_observers_stack)
• New tests, with this fix: 2 passed in 0.02s
• Existing crewai tests: pytest tests/test_integrations/test_crewai/ → 16 failed, 2 passed before and after the fix with an identical failure list; all 16 fail with ValueError: OPENAI_API_KEY is required because they run real crew.kickoff() calls (the suite notes an OpenAI key is required, e.g. tests/test_integrations/test_crewai/test_sync.py:104), so they cannot run in a keyless environment.
• black --check on both changed files: 2 files would be left unchanged.
• ruff check on both changed files: only the 3 pre-existing F401 unused imports in handler.py, identical to main.