Repository navigation
Commit 6fd6fad
committed
fix(openai-native): send body prompt_cache_key for cache affinity
What: the OpenAI-native handler now sends a task-scoped `prompt_cache_key`
in the Responses request body on both body builders (streaming and
completePrompt), and no longer sends the `session_id` request header.
The key is `metadata.taskId`, falling back to the handler session id,
clamped to OpenAI's 64-character limit, and omitted for models whose
catalog entry has `supportsPromptCache: false`.
Why: OpenAI keys prompt-cache routing off the body `prompt_cache_key`
field. The `session_id` header we were sending is not part of the
prompt-caching API and is ignored on the public Responses API, so every
OpenAI-native task was leaving cache hit-rate on the table. The header
also carries an underscore in its name, which hardened gateways
(Envoy/nginx `underscores_in_headers`) reject outright, so it was a
latent 400 for anyone behind such a proxy with a custom base URL.
`prompt_cache_retention: "24h"` is untouched: it is the supported
lifetime control on the pre-5.6 models that set it in the catalog, and
the existing retention specs pass unmodified.
Impact: cache affinity now follows the documented body field, stable
across turns of one task and distinct across tasks. No usage
accounting, cost, or other request field changes. openai-codex.ts and
xai.ts are untouched: codex already sends a body key, and xai never had
the header.1 parent 159c42e commit 6fd6fad
3 files changed
Lines changed: 197 additions & 9 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
440 | 440 | | |
441 | 441 | | |
442 | 442 | | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
| 451 | + | |
| 452 | + | |
| 453 | + | |
| 454 | + | |
| 455 | + | |
| 456 | + | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
443 | 507 | | |
444 | 508 | | |
445 | 509 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
534 | 534 | | |
535 | 535 | | |
536 | 536 | | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
537 | 557 | | |
538 | 558 | | |
539 | 559 | | |
| |||
1446 | 1466 | | |
1447 | 1467 | | |
1448 | 1468 | | |
| 1469 | + | |
| 1470 | + | |
| 1471 | + | |
| 1472 | + | |
| 1473 | + | |
| 1474 | + | |
| 1475 | + | |
| 1476 | + | |
| 1477 | + | |
| 1478 | + | |
| 1479 | + | |
| 1480 | + | |
| 1481 | + | |
| 1482 | + | |
| 1483 | + | |
| 1484 | + | |
| 1485 | + | |
| 1486 | + | |
| 1487 | + | |
| 1488 | + | |
| 1489 | + | |
| 1490 | + | |
| 1491 | + | |
| 1492 | + | |
| 1493 | + | |
| 1494 | + | |
| 1495 | + | |
| 1496 | + | |
| 1497 | + | |
| 1498 | + | |
| 1499 | + | |
| 1500 | + | |
| 1501 | + | |
| 1502 | + | |
| 1503 | + | |
| 1504 | + | |
| 1505 | + | |
| 1506 | + | |
| 1507 | + | |
| 1508 | + | |
| 1509 | + | |
| 1510 | + | |
| 1511 | + | |
| 1512 | + | |
| 1513 | + | |
| 1514 | + | |
| 1515 | + | |
| 1516 | + | |
| 1517 | + | |
| 1518 | + | |
| 1519 | + | |
| 1520 | + | |
| 1521 | + | |
| 1522 | + | |
| 1523 | + | |
| 1524 | + | |
| 1525 | + | |
| 1526 | + | |
1449 | 1527 | | |
1450 | 1528 | | |
1451 | 1529 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
34 | 34 | | |
35 | 35 | | |
36 | 36 | | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
37 | 43 | | |
38 | 44 | | |
39 | 45 | | |
| |||
96 | 102 | | |
97 | 103 | | |
98 | 104 | | |
99 | | - | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
100 | 109 | | |
101 | 110 | | |
102 | 111 | | |
103 | 112 | | |
104 | 113 | | |
105 | 114 | | |
106 | | - | |
107 | 115 | | |
108 | 116 | | |
109 | 117 | | |
| |||
330 | 338 | | |
331 | 339 | | |
332 | 340 | | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
333 | 346 | | |
334 | 347 | | |
335 | 348 | | |
| |||
347 | 360 | | |
348 | 361 | | |
349 | 362 | | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
350 | 367 | | |
351 | 368 | | |
352 | 369 | | |
| |||
379 | 396 | | |
380 | 397 | | |
381 | 398 | | |
| 399 | + | |
| 400 | + | |
382 | 401 | | |
383 | 402 | | |
384 | 403 | | |
| |||
418 | 437 | | |
419 | 438 | | |
420 | 439 | | |
421 | | - | |
422 | | - | |
| 440 | + | |
423 | 441 | | |
424 | 442 | | |
425 | 443 | | |
426 | | - | |
427 | 444 | | |
428 | 445 | | |
429 | 446 | | |
| |||
557 | 574 | | |
558 | 575 | | |
559 | 576 | | |
560 | | - | |
| 577 | + | |
561 | 578 | | |
562 | 579 | | |
563 | 580 | | |
| |||
568 | 585 | | |
569 | 586 | | |
570 | 587 | | |
571 | | - | |
572 | | - | |
| 588 | + | |
573 | 589 | | |
574 | 590 | | |
575 | 591 | | |
| |||
579 | 595 | | |
580 | 596 | | |
581 | 597 | | |
582 | | - | |
583 | 598 | | |
584 | 599 | | |
585 | 600 | | |
| |||
1420 | 1435 | | |
1421 | 1436 | | |
1422 | 1437 | | |
| 1438 | + | |
| 1439 | + | |
| 1440 | + | |
| 1441 | + | |
| 1442 | + | |
| 1443 | + | |
| 1444 | + | |
| 1445 | + | |
| 1446 | + | |
| 1447 | + | |
| 1448 | + | |
| 1449 | + | |
| 1450 | + | |
| 1451 | + | |
| 1452 | + | |
| 1453 | + | |
| 1454 | + | |
| 1455 | + | |
| 1456 | + | |
| 1457 | + | |
| 1458 | + | |
| 1459 | + | |
| 1460 | + | |
| 1461 | + | |
1423 | 1462 | | |
1424 | 1463 | | |
1425 | 1464 | | |
| |||
1531 | 1570 | | |
1532 | 1571 | | |
1533 | 1572 | | |
| 1573 | + | |
| 1574 | + | |
| 1575 | + | |
| 1576 | + | |
| 1577 | + | |
| 1578 | + | |
| 1579 | + | |
1534 | 1580 | | |
1535 | 1581 | | |
1536 | 1582 | | |
| |||
0 commit comments