Skip to content

Please revisit decision to emit an event for every message #1621

@stephentoub

Description

@stephentoub

The latest (1.28) spec requires emitting an event for every individual message. This is problematic in a variety of ways:

  • Messages that are part of the same request are not individual entities: they are only meaningful as part of the overall request being submitted to the LLM, which functionally turns the totality of those messages into a single prompt. The whole set of messages / the whole request should be logged, not each tiny piece of it individually.
  • A request might contain dozens or hundreds of messages. Logging each individually spams the listener, which then needs to do work to reconstitute the original request.
  • Any sampling being performed also then needs to do such work, as it otherwise might drop portions of a request.
  • The same message ends up getting logged on its own many times, each time a new request with the same history is resubmitted (augmented with additional messages). That's easy to understand when the messages are grouped, but when they're logged individually, it's much harder to reason about the duplication.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions