Skip to content

Triplet view collapses requests without valid prompt token IDs #572

Closed

Description

@rioyu123

What happens

The format=triplet event view currently groups every model_request by prompt_token_ids and keeps the last event for each key. When those IDs are absent or empty, every request gets the same empty key, so unrelated calls in one rollout are silently collapsed.

Malformed values can be worse: a nested list reaches tuple(prompt_token_ids) and raises TypeError, turning the triplet query into a 500 response.

This is reachable through the event POST endpoint, whose event data is intentionally open-ended, and through OpenAI-compatible responses that do not echo prompt token IDs. _trim_model_request also defaults missing IDs to an empty list.

Minimal reproduction

  1. Append two model_request events to one rollout with different non-empty response token IDs.
  2. Give both events prompt_token_ids: [] (or omit the field).
  3. Query /api/rollouts/{id}/events?format=triplet.

Only the second request is returned. Using prompt_token_ids: [[1]] instead raises TypeError: unhashable type: 'list'.

Expected behavior

Prompt-based last-wins deduplication should remain unchanged when IDs are a valid, non-empty list of integers. When no valid key exists, the events should be retained in insertion order because the value cannot establish that they are duplicates.

I have a focused fix with server and VERL multimodal-alignment regression coverage ready and will link the PR here.

Activity

  1. hzy46 commented on Aug 28, 2026

    @hzy46
    Contributor

    Thanks for report this issue.

    For the PR, can the code be simplifed to just add the following sentences?

    if not(prompt_token_ids):
        # ignore
        continue
    
  2. rioyu123 commented on Aug 28, 2026

    @rioyu123
    ContributorAuthor

    Thanks — I've simplified #573 along those lines, keeping the original last-index map and removing the two helpers and superseded-index tracking.

    One detail: the guard needs to retain the request's index before continue, otherwise the final inclusion filter drops requests with missing/empty IDs. I also kept the integer-list check: [[1]] is non-empty but still raises when used as a key. The VERL image-alignment path follows the same rule.

    The full CPU test suite passes (92 tests), along with Ruff, Pyright, pre-commit and the package build. The 16 invalid-ID regressions also fail on unchanged upstream, so they exercise the original failure modes.

  3. hzy46 commented on Aug 28, 2026

    @hzy46
    Contributor

    Merged. Thank you for this contribution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions