Repository navigation
Commit 5d9b0f0
authored
fix(inference): prepend publisher prefix for Vertex non-Anthropic models (NVIDIA#2735)
* fix(inference): prepend publisher prefix for Vertex non-Anthropic models
Vertex AI's OpenAI-compatible endpoint requires the request body's model
field to carry a publisher prefix (e.g. google/gemini-2.5-flash), but
validate_vertex_model_id rejects slash as a path-traversal guard. This
created a deadlock: bare model IDs pass validation but are rejected by
Vertex with HTTP 400 "Malformed publisher model"; prefixed IDs are
rejected at configuration time.
Fix: in resolve_vertex_ai_route, compute body_model_id for non-Anthropic
routes by prepending the publisher from infer_vertex_publisher() or the
explicit VERTEX_AI_PUBLISHER config value. The bare model_id still goes
through the path-traversal validator unchanged. Anthropic rawPredict routes
encode the model in the URL path, not the body, and are unaffected. Both
the project/region path and the base-URL-override path apply the prefix.
For unrecognised models with no explicit publisher the bare ID is forwarded
unchanged; Vertex's 400 is the correct observable signal in that case.
Add an integration test in openshell-router that spins up a mock Vertex
endpoint accepting only the publisher-prefixed form and rejecting the bare
model name, verifying the body rewrite produces the required format.
Closes NVIDIA#2351
Signed-off-by: politerealism <[email protected]>
* fix(inference): propagate publisher-prefixed model_id through inference bundle
resolve_route_by_name_with_credentials built the ResolvedRoute with
config.model_id (the bare stored value) rather than resolved.route.model
(the publisher-prefixed value computed by resolve_vertex_ai_route). As a
result, the bundle delivered to sandboxes carried e.g. "gemini-2.5-flash"
instead of "google/gemini-2.5-flash", so live sandbox requests still hit
Vertex AI with the bare model name and received HTTP 400 "Malformed
publisher model".
Fix: use resolved.route.model in the bundle construction so the
publisher prefix survives the bundle boundary and the router sends the
correct body to Vertex AI.
Update the existing gemini bundle test to assert the prefixed model_id
and add a dedicated regression test that verifies the bundle carries the
publisher prefix for non-Anthropic Vertex routes.
Signed-off-by: politerealism <[email protected]>
* style(inference): apply rustfmt to new regression test
Signed-off-by: politerealism <[email protected]>
* fix(inference): address clippy lints in build_vertex_route
- Invert if !is_anthropic to satisfy clippy::if_not_else
- Replace match on Option with map_or_else to satisfy clippy::option_if_let_else
Signed-off-by: politerealism <[email protected]>
---------
Signed-off-by: politerealism <[email protected]>1 parent 6ebf10e commit 5d9b0f0
2 files changed
Lines changed: 210 additions & 6 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
982 | 982 | | |
983 | 983 | | |
984 | 984 | | |
985 | | - | |
986 | | - | |
| 985 | + | |
| 986 | + | |
987 | 987 | | |
988 | 988 | | |
989 | 989 | | |
| |||
2597 | 2597 | | |
2598 | 2598 | | |
2599 | 2599 | | |
| 2600 | + | |
| 2601 | + | |
| 2602 | + | |
| 2603 | + | |
| 2604 | + | |
| 2605 | + | |
| 2606 | + | |
| 2607 | + | |
| 2608 | + | |
| 2609 | + | |
| 2610 | + | |
| 2611 | + | |
| 2612 | + | |
| 2613 | + | |
| 2614 | + | |
| 2615 | + | |
| 2616 | + | |
| 2617 | + | |
| 2618 | + | |
| 2619 | + | |
| 2620 | + | |
| 2621 | + | |
| 2622 | + | |
| 2623 | + | |
| 2624 | + | |
| 2625 | + | |
| 2626 | + | |
| 2627 | + | |
| 2628 | + | |
| 2629 | + | |
| 2630 | + | |
| 2631 | + | |
| 2632 | + | |
| 2633 | + | |
| 2634 | + | |
| 2635 | + | |
| 2636 | + | |
| 2637 | + | |
| 2638 | + | |
| 2639 | + | |
| 2640 | + | |
| 2641 | + | |
| 2642 | + | |
| 2643 | + | |
| 2644 | + | |
| 2645 | + | |
| 2646 | + | |
| 2647 | + | |
| 2648 | + | |
| 2649 | + | |
| 2650 | + | |
| 2651 | + | |
| 2652 | + | |
| 2653 | + | |
| 2654 | + | |
| 2655 | + | |
| 2656 | + | |
| 2657 | + | |
| 2658 | + | |
| 2659 | + | |
| 2660 | + | |
| 2661 | + | |
| 2662 | + | |
| 2663 | + | |
| 2664 | + | |
| 2665 | + | |
| 2666 | + | |
| 2667 | + | |
| 2668 | + | |
| 2669 | + | |
| 2670 | + | |
| 2671 | + | |
| 2672 | + | |
| 2673 | + | |
| 2674 | + | |
| 2675 | + | |
| 2676 | + | |
| 2677 | + | |
| 2678 | + | |
| 2679 | + | |
| 2680 | + | |
| 2681 | + | |
| 2682 | + | |
| 2683 | + | |
2600 | 2684 | | |
2601 | 2685 | | |
2602 | 2686 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
698 | 698 | | |
699 | 699 | | |
700 | 700 | | |
| 701 | + | |
| 702 | + | |
| 703 | + | |
| 704 | + | |
| 705 | + | |
| 706 | + | |
| 707 | + | |
| 708 | + | |
| 709 | + | |
| 710 | + | |
| 711 | + | |
| 712 | + | |
| 713 | + | |
| 714 | + | |
| 715 | + | |
| 716 | + | |
701 | 717 | | |
702 | 718 | | |
703 | 719 | | |
| |||
721 | 737 | | |
722 | 738 | | |
723 | 739 | | |
724 | | - | |
| 740 | + | |
725 | 741 | | |
726 | 742 | | |
727 | 743 | | |
| |||
772 | 788 | | |
773 | 789 | | |
774 | 790 | | |
775 | | - | |
| 791 | + | |
776 | 792 | | |
777 | 793 | | |
778 | 794 | | |
| |||
1140 | 1156 | | |
1141 | 1157 | | |
1142 | 1158 | | |
1143 | | - | |
| 1159 | + | |
1144 | 1160 | | |
1145 | 1161 | | |
1146 | 1162 | | |
| |||
1730 | 1746 | | |
1731 | 1747 | | |
1732 | 1748 | | |
1733 | | - | |
| 1749 | + | |
1734 | 1750 | | |
1735 | 1751 | | |
1736 | 1752 | | |
1737 | 1753 | | |
1738 | 1754 | | |
1739 | 1755 | | |
| 1756 | + | |
| 1757 | + | |
| 1758 | + | |
| 1759 | + | |
| 1760 | + | |
| 1761 | + | |
| 1762 | + | |
| 1763 | + | |
| 1764 | + | |
| 1765 | + | |
| 1766 | + | |
| 1767 | + | |
| 1768 | + | |
| 1769 | + | |
| 1770 | + | |
| 1771 | + | |
| 1772 | + | |
| 1773 | + | |
| 1774 | + | |
| 1775 | + | |
| 1776 | + | |
| 1777 | + | |
| 1778 | + | |
| 1779 | + | |
| 1780 | + | |
| 1781 | + | |
| 1782 | + | |
| 1783 | + | |
| 1784 | + | |
| 1785 | + | |
| 1786 | + | |
| 1787 | + | |
| 1788 | + | |
| 1789 | + | |
| 1790 | + | |
| 1791 | + | |
| 1792 | + | |
| 1793 | + | |
1740 | 1794 | | |
1741 | 1795 | | |
1742 | 1796 | | |
| |||
2613 | 2667 | | |
2614 | 2668 | | |
2615 | 2669 | | |
| 2670 | + | |
| 2671 | + | |
| 2672 | + | |
| 2673 | + | |
| 2674 | + | |
2616 | 2675 | | |
2617 | 2676 | | |
2618 | 2677 | | |
| |||
2651 | 2710 | | |
2652 | 2711 | | |
2653 | 2712 | | |
| 2713 | + | |
| 2714 | + | |
| 2715 | + | |
| 2716 | + | |
| 2717 | + | |
| 2718 | + | |
| 2719 | + | |
| 2720 | + | |
| 2721 | + | |
| 2722 | + | |
| 2723 | + | |
| 2724 | + | |
| 2725 | + | |
| 2726 | + | |
| 2727 | + | |
| 2728 | + | |
| 2729 | + | |
| 2730 | + | |
| 2731 | + | |
| 2732 | + | |
| 2733 | + | |
| 2734 | + | |
| 2735 | + | |
| 2736 | + | |
| 2737 | + | |
| 2738 | + | |
| 2739 | + | |
| 2740 | + | |
| 2741 | + | |
| 2742 | + | |
| 2743 | + | |
| 2744 | + | |
| 2745 | + | |
| 2746 | + | |
| 2747 | + | |
| 2748 | + | |
| 2749 | + | |
| 2750 | + | |
| 2751 | + | |
| 2752 | + | |
| 2753 | + | |
| 2754 | + | |
| 2755 | + | |
| 2756 | + | |
| 2757 | + | |
| 2758 | + | |
| 2759 | + | |
| 2760 | + | |
| 2761 | + | |
| 2762 | + | |
| 2763 | + | |
| 2764 | + | |
| 2765 | + | |
| 2766 | + | |
| 2767 | + | |
| 2768 | + | |
| 2769 | + | |
| 2770 | + | |
| 2771 | + | |
| 2772 | + | |
| 2773 | + | |
2654 | 2774 | | |
2655 | 2775 | | |
2656 | 2776 | | |
| |||
0 commit comments