Skip to content

refactor(ai): ai-proxy and ai-proxy-multi#12030

Merged
shreemaan-abhishek merged 4 commits into
apache:masterfrom
shreemaan-abhishek:refactor-ai-proxy
Mar 11, 2025
Merged

refactor(ai): ai-proxy and ai-proxy-multi#12030
shreemaan-abhishek merged 4 commits into
apache:masterfrom
shreemaan-abhishek:refactor-ai-proxy

Conversation

@shreemaan-abhishek

@shreemaan-abhishek shreemaan-abhishek commented Mar 10, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

Description

This PR accomplishes the following:

  1. Adjust priorities of ai plugins in order to incorporate upcoming ai-rate-limiting plugin.
  2. Refactors ai-proxy and ai-proxy-multi, integrating their proxy logic with APISIX's phase concept to work in conjunction with the ai-rate-limiting plugin under development
  3. Handles AI requests and responses accordingly, also obtaining usage data for AI provider

Checklist

  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change
  • I have updated the documentation to reflect this change
  • I have verified that this change is backward compatible (If not, please discuss on the APISIX mailing list first)

@shreemaan-abhishek
shreemaan-abhishek marked this pull request as ready for review March 11, 2025 07:00
@dosubot dosubot Bot added size:XL This PR changes 500-999 lines, ignoring generated files. plugin tech debt labels Mar 11, 2025
membphis
membphis reviewed Mar 11, 2025
View reviewed changes
Comment thread apisix/plugins/ai-proxy/base.lua
request_table.model = get_model_name_func(conf)
function _M.before_proxy(conf, ctx)
local ai_instance = ctx.picked_ai_instance
local ai_driver = require("apisix.plugins.ai-drivers." .. ai_instance.provider)

@membphis membphis Mar 11, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This may failed, we need to check the return value or use pcall

@shreemaan-abhishek shreemaan-abhishek Mar 11, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this line guarantees safety: https://github.com/apache/apisix/pull/12030/files#diff-6df73662ffc3d9815275bcf80c1ed0cebc86ce2dea275087cd299c6dea3b465aR73

@shreemaan-abhishek
shreemaan-abhishek merged commit 53e5b02 into apache:master Mar 11, 2025
@shreemaan-abhishek
shreemaan-abhishek deleted the refactor-ai-proxy branch March 11, 2025 10:26
laz-xyr pushed a commit to laz-xyr/apisix that referenced this pull request Jun 19, 2025
@shreemaan-abhishek
refactor(ai): ai-proxy and ai-proxy-multi (apache#12030)
a19d3da
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@membphis membphis membphis approved these changes
@nic-6443 nic-6443 nic-6443 approved these changes
@AlinsRan AlinsRan AlinsRan approved these changes

Assignees

No one assigned

Labels

plugin size:XL This PR changes 500-999 lines, ignoring generated files. tech debt

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@shreemaan-abhishek @membphis @nic-6443 @AlinsRan