Repository navigation
epp: parse multipart video requests for routing - #2937
Conversation
The OpenAI parser claimed no video path, so a video request fell through to passthrough-parser: no model was extracted for routing and the body stayed opaque, while the pinned vLLM-Omni backend serves both endpoints from one multipart form handler. Claims both video suffixes and parses the shared form, extracting the model and the scheduler-relevant cost fields (steps, size, dimensions, frames, fps, count). Media parts and the JSON-encoded reference fields are left in the raw payload so the forwarded body is byte-identical to the request, and file parts are drained to io.Discard rather than read into a buffer. Field bounds mirror the backend's own, so overflow and malformed numbers fail at the router instead of being clamped or zeroed into the cost fields. See ws/A2/backend_contract.md for the source-derived wire contract. Not included: rewriting the model field inside a multipart body. The router forwards the multipart bytes verbatim, so RewriteModelName has no marshalable payload to edit on this path; routing by the extracted model works, rewriting does not. Signed-off-by: 0z5a <[email protected]>
The video parser forwarded the body as raw bytes, which is not marshalable, so the director skipped the model rewrite on /v1/videos and /v1/videos/sync and the model the backend validated could differ from the one admission used. The payload is now a MultipartPayload that keeps the received body and splices a replacement into the last model form field, which is the field the backend reads. Every other part is copied verbatim, so media bytes, part headers, and the boundary lines are unchanged. The integer form fields accept the surrounding whitespace and integral float spellings the backend accepts, so the router no longer rejects a request the backend serves. Signed-off-by: 0z5a <[email protected]>
|
🚨 Unsigned commits detected! Please sign your commits. For instructions on how to set up GPG/SSH signing and verify your commits, please see GitHub Documentation. |
f8e5cef to
698a32a
Compare
golangci-lint's goconst flags the multipart form field names that the images-edits parser, the video parser and RewriteModelName all spell out. Declare model and num_inference_steps once and use them at every site, and do the same for the test fixture's default media type. No behaviour change: the literals are identical, gofmt realigned the const block, and the focused parser tests and go build still pass. Signed-off-by: 0z5a <[email protected]>
ad26d3d to
032ea03
Compare
|
Update: the last |
zetxqx
left a comment
There was a problem hiding this comment.
Thanks for adding this, we don't need to support rewrite for video API so the new multiPartPaylaod is not needed here. Please convert it directly to rawPaylaod.
| // request. Only scheduler-relevant scalar fields are read; the media parts and the | ||
| // JSON-encoded reference fields stay in Payload, which keeps the received bytes so a | ||
| // model rewrite can replace the model part alone and still forward the media verbatim. | ||
| func parseVideosRequest(body []byte, headers map[string]string) (*fwkrh.ParseResult, error) { |
There was a problem hiding this comment.
There are a lot of common parts of multipart parsing with image edits multipart
can you extract the common part instead of repeating?
| // so the parts the router does not interpret, media uploads in particular, keep the | ||
| // bytes, the headers, and the boundary lines the client sent. A payload with nothing | ||
| // to rewrite marshals to the received body itself. | ||
| type MultipartPayload struct { |
There was a problem hiding this comment.
I see you want to add a new MultipartPayload to support modelRewrite. However in vllmomni doc: https://docs.vllm.ai/projects/vllm-omni/en/latest/serving/videos_api/#start-the-server the model in the video request is not needed.
So we can follow what we already have similar to imageEdits parsing the paylaod to rawPayload
There was a problem hiding this comment.
the test name is confusing. We should add this in openai_test.go
There was a problem hiding this comment.
Test file name is confusing
There was a problem hiding this comment.
test file name is confusing
Reuse the multipart field reader with image edits and preserve raw video payloads for backend forwarding. Signed-off-by: 0z5a <[email protected]>
f2b60b9 to
4c4acd5
Compare
|
Addressed in 4c4acd5: video multipart requests now use RawPayload without model rewriting; parsing is shared with image edits, and the tests are in the existing OpenAI, handler, and director suites. Targeted Go tests, build/vet, and live sync/async E2E passed. The speed comparison is in the PR description. Thanks for the review. |
|
@0z5a is this ready for review? I saw it is in draft still. |
|
@0z5a This PR has merge conflicts with gh pr checkout 2937
git fetch origin main
git rebase origin/main
# resolve conflicts
git push --force-with-lease |
Signed-off-by: 0z5a <[email protected]>
Signed-off-by: 0z5a <[email protected]>
ahg-g
left a comment
There was a problem hiding this comment.
zetxqx's earlier comments are addressed, and the parsing and forwarding look correct. Two inline: stale bound comments in types.go, and a note on async video polling with more than one replica.
| // VideoGenerationRequest represents the OpenAI-compatible vLLM-Omni | ||
| // /v1/videos and /v1/videos/sync request body. Both endpoints are served by the | ||
| // same handler and accept multipart/form-data, so they share this shape. | ||
| // Field bounds mirror the pinned backend contract; omitted fields stay nil |
There was a problem hiding this comment.
Drop the backend bounds from the VideoGenerationRequest comments: "Field bounds mirror the pinned backend contract" (L636), "at least 1" (L647), "1..200" (L657) and "1..10".
Why: e98ff34 removed the router-side bounds checks, as zetxqx asked on 10-08, so these comments now describe validation the router doesn't do → readers will assume out-of-range values are rejected before reaching vLLM-Omni.
There was a problem hiding this comment.
Fixed in 29f416e. Removed the backend bound claims from the VideoGenerationRequest comments (Width/Height, NumInferenceSteps, NumOutputsPerPrompt). The nil-handling note stays.
| imagesEditsAPI = "images/edits" | ||
| // videosAPI is the OpenAI-compatible vLLM-Omni asynchronous video generation | ||
| // endpoint. Requests are multipart/form-data. | ||
| videosAPI = "videos" |
There was a problem hiding this comment.
Document that async /v1/videos needs a single-replica pool, or session affinity, for status and content requests.
Why: the create POST is routed by load, but GET /v1/videos/{id} and /content have no body, so the router sends them to a random pod (handlers/request.go:44) → with more than one vLLM-Omni replica, a poll lands on a pod that doesn't hold the job → status and content calls return 404. The E2E in the description used one backend, so it doesn't cover this. /v1/videos/sync isn't affected.
There was a problem hiding this comment.
Fixed in 29f416e. The videosAPI comment now says that status and content GETs have no body and are routed to a random endpoint, so async /v1/videos needs a single-replica pool or session affinity. /v1/videos/sync is unaffected.
|
@0z5a This PR has merge conflicts with gh pr checkout 2937
git fetch origin main
git rebase origin/main
# resolve conflicts
git push --force-with-lease |
The router no longer checks VideoGenerationRequest field bounds after e98ff34, so the comments describing them were misleading. Document that async status and content requests carry no body, so they are routed to a random endpoint and need a single-replica pool or session affinity. Signed-off-by: 0z5a <[email protected]>
Signed-off-by: 0z5a <[email protected]>
29f416e to
e0d4d14
Compare
|
@0z5a This PR has merge conflicts with gh pr checkout 2937
git fetch origin main
git rebase origin/main
# resolve conflicts
git push --force-with-lease |
| switch name { | ||
| case reqcommon.FieldModel: | ||
| extractedBody.Model = string(value) | ||
| case "prompt": |
There was a problem hiding this comment.
When rebasing onto main, carry #3721's case "negative_prompt": images.NegativePrompt = string(value) into this image-edits field switch, and keep its ParseRequest_ImagesEdits test row.
Why: main's parseImagesEditsRequest now parses negative_prompt for the image-edit token estimate (#3721). This PR replaces that function's body with the shared parseMultipartFields reader, and the merge conflict lands on exactly these cases, so resolving toward this branch silently drops the field and image-edit requests go back to being estimated without their negative prompt, with no compile error.
| // videosAPI is the OpenAI-compatible vLLM-Omni asynchronous video generation | ||
| // endpoint. Requests are multipart/form-data. Status and content requests | ||
| // (GET /v1/videos/{id} and /v1/videos/{id}/content) have no body, so they are | ||
| // routed to a random endpoint. With more than one replica, a poll can reach a | ||
| // pod that does not hold the job, so the pool needs a single replica or | ||
| // session affinity. | ||
| videosAPI = "videos" |
There was a problem hiding this comment.
Move the async-polling caveat into parsers/README.md as a Drawback bullet under openai-parser, next to the Decisions one.
* **Drawback**: For async video generation, `POST /v1/videos`, the status and content requests (`GET /v1/videos/{id}`, `GET /v1/videos/{id}/content`) carry no body and are routed to a random endpoint, so a pool with more than one vLLM-Omni replica needs session affinity. `POST /v1/videos/sync` is unaffected.Why: The caveat currently lives only in a Go constant's comment, and the parser README, where operators look up supported APIs, does not mention video at all. An operator who scales a vLLM-Omni pool past one replica gets intermittent 404s on job polls with nothing in the docs explaining why.
There was a problem hiding this comment.
@0z5a Reminder: please add the async-polling caveat to parsers/README.md as a Drawback bullet under openai-parser (suggested text above). This is the last blocking item; the RewriteModelName revert is an optional nit. Everything else looks good, and the PR still merges cleanly and passes tests against current main.
There was a problem hiding this comment.
Done in 59c9f86. Added the Drawback bullet with your text under openai-parser in parsers/README.md, right after the Decisions one.
| func (p *OpenAIParser) RewriteModelName(payload fwkrh.MarshalablePayload, model string) (fwkrh.MarshalablePayload, error) { | ||
| m, ok := payload.(fwkrh.PayloadMap) | ||
| if !ok { | ||
| switch m := payload.(type) { | ||
| case fwkrh.PayloadMap: | ||
| m[reqcommon.FieldModel] = model | ||
| return m, nil | ||
| default: | ||
| return payload, nil | ||
| } |
There was a problem hiding this comment.
Nit: revert RewriteModelName to the original type assertion; this single-case type switch is left over from the removed MultipartPayload case.
m, ok := payload.(fwkrh.PayloadMap)
if !ok {
return payload, nil
}
m[reqcommon.FieldModel] = model
return m, nilWhy: Video bodies are RawPayload now, so this function's behavior is unchanged from main and the diff here is churn in a hot-path rewriter that reviewers have to re-verify.
There was a problem hiding this comment.
Done in 59c9f86. RewriteModelName is back to the original type assertion (with reqcommon.FieldModel).
Signed-off-by: 0z5a <[email protected]>
ef072ab to
bb2b6be
Compare
| if videos.Prompt == "" { | ||
| return nil, errors.New("invalid videos request: must have prompt field") | ||
| } | ||
| return &fwkrh.ParseResult{Body: extractedBody, SkipResponseProcessing: false}, nil |
There was a problem hiding this comment.
Handle video/ responses alongside audio/ in OpenAIParser.ParseResponse, and add a video/mp4 case to TestOpenAIParser_ParseResponse.
if strings.HasPrefix(mediaType, "audio/") || strings.HasPrefix(mediaType, "video/") || mediaType == octetStreamType {Why: parseVideosRequest sets SkipResponseProcessing: false, and POST /v1/videos/sync returns raw video/mp4 bytes -> ParseResponse misses the binary media-type branch and passes the MP4 body to extractUsage (json.Unmarshal) -> every synchronous video response logs a response-parsing error in HandleResponseBody and drops any x-vllm-omni-*-tokens usage headers.
There was a problem hiding this comment.
Done. ParseResponse treats video/ like audio/ and reads usage from the response headers only. Added two video/mp4 cases to TestOpenAIParser_ParseResponse, with and without the x-vllm-omni-*-tokens headers. vLLM-Omni's /v1/videos/sync currently sends no token headers, so the case without headers is the one that matches today's server: it returns no usage and no error.
| case "fps": | ||
| fps, err := strconv.ParseFloat(string(value), 64) | ||
| if err != nil { | ||
| return fmt.Errorf("invalid videos fps field: %w", err) | ||
| } | ||
| videos.FPS = &fps |
There was a problem hiding this comment.
Reject NaN and Inf in parseVideosRequest's fps branch, and add "NaN" and "Inf" cases to TestOpenAIParser_ParseRequestVideosInvalidForms.
case "fps":
fps, err := strconv.ParseFloat(string(value), 64)
if err != nil || math.IsNaN(fps) || math.IsInf(fps, 0) {
return fmt.Errorf("invalid videos fps field: %q", value)
}
videos.FPS = &fpsWhy: strconv.ParseFloat accepts "NaN", "Inf", and "-Inf" without error -> non-finite floats enter VideoGenerationRequest.FPS -> downstream frame/cost arithmetic produces NaN/Inf and json.Marshal on VideoGenerationRequest fails with an unsupported-value error.
There was a problem hiding this comment.
Done. The fps branch rejects NaN and Inf after ParseFloat. Added NaN, Inf and -Inf cases to TestOpenAIParser_ParseRequestVideosInvalidForms.
| func derefInt(v *int64) string { | ||
| if v == nil { | ||
| return "<nil>" | ||
| } | ||
| return strconv.FormatInt(*v, 10) | ||
| } | ||
|
|
||
| // derefFloat formats an optional float pointer for logging without | ||
| // dereferencing a nil, so String stays safe on a partially populated request. | ||
| func derefFloat(v *float64) string { | ||
| if v == nil { | ||
| return "<nil>" | ||
| } | ||
| return strconv.FormatFloat(*v, 'g', -1, 64) | ||
| } |
There was a problem hiding this comment.
Return nilStr instead of "<nil>" in derefInt and derefFloat, and add TestVideoGenerationRequest_String in types_test.go.
Why: types.go already defines const nilStr = "<nil>" on line 36 (used on line 679 in VideoGenerationRequest.String()), so repeating raw "<nil>" literals duplicates the package constant, and VideoGenerationRequest.String() has no unit test alongside TestTextToSpeechRequest_String and TestAudioGenerateRequest_String.
There was a problem hiding this comment.
Done. derefInt and derefFloat return nilStr, and TestVideoGenerationRequest_String covers a partially populated request and a nil receiver.
POST /v1/videos/sync returns raw video/mp4 bytes, which ParseResponse passed to the JSON usage extractor. Treat video/ like audio/ and read usage from the response headers. Reject NaN and Inf for the fps form field, since strconv.ParseFloat accepts them. Use nilStr in derefInt and derefFloat and add a VideoGenerationRequest.String test. Signed-off-by: 0z5a <[email protected]>
Add the async /v1/videos polling caveat to the openai-parser section of parsers/README.md, where operators look up supported APIs. Also restore RewriteModelName's original type assertion; the type switch was left over from the removed MultipartPayload case. Signed-off-by: 0z5a <[email protected]>
/kind feature
What this PR does / why we need it:
The EPP OpenAI parser claims
POST /v1/videosandPOST /v1/videos/sync. It reads the model and scheduling fields from multipart forms so video generation requests can be routed. Video and image edits share one multipart field reader. The original multipart bytes are forwarded asRawPayload.This revision addresses review #5287195121: it removes
MultipartPayloadand video model rewriting, and places the video, registry, and forwarding tests in the existing package test files.Which issue(s) this PR fixes:
Related to #1935 (multipart and video parser stretch goal).
Validation:
go test -count=1 ./pkg/epp/framework/plugins/requesthandling/parsers/openai ./pkg/epp/handlers ./pkg/epp/requestcontrol.go build ./...andgo vet ./pkg/epp/framework/plugins/requesthandling/parsers/openai.2dd31c443f054b0e058a240c3e65c6b5141b65467f489b6afb8ee1dfc9767d02). Each decoded as H.264, 832x480, 9 frames.go test -count=1 ./pkg/epp/...has one failure inTestGenerateRequest_UnmarshalJSON/invalid_placeholder_preserves_error_field. The same failure reproduces on the pristine previous PR head (032ea03e) with Go 1.27.1: the test expects an older JSON error string. All other EPP packages passed.CPU benchmark (1 MiB multipart media, Intel Xeon Gold 6530, CPU 80,
GOMAXPROCS=1, five 200-operation samples per round; medians shown):The forwarding benchmark uses the same multipart request on both revisions and applies the director's payload decision for a different target model. The previous branch rewrites and marshals the video body; this revision forwards
RawPayload. It measures router CPU work, not video generation time. The shared parser adds some parse-only time, as shown in the last two rows.Release note: