feat: ai-rag plugin#11568
Conversation
|
|
||
| request_schema.properties.ai_rag.properties.vector_search = vs_req_schema | ||
| request_schema.properties.ai_rag.properties.embeddings = emb_req_schema | ||
|
|
There was a problem hiding this comment.
construct a request_body_schema according to the service providers
| core.log.error("dibag err: ", err) | ||
| core.log.warn("dibag res: ", core.json.encode(res, true)) | ||
|
|
||
| body_tab["ai_rag"] = nil |
There was a problem hiding this comment.
remove ai_rag from request body because their purpose is served and also these values will cause failure when proxying requests to LLM.
There was a problem hiding this comment.
comment the purpose in source code?
| if not body_tab.messages then | ||
| body_tab.messages = {} | ||
| end | ||
| decorate(decorator_conf, body_tab) |
There was a problem hiding this comment.
we need to take the vector search result and append it to the messages field in request body. This messages will be sent to LLM to get the response.
There was a problem hiding this comment.
Based on the code, I think you mean "prepend" instead of "append"?
There was a problem hiding this comment.
ah, yes. My bad, it should be prepend.
6e5c04b to
0c8e6b7
Compare
dosubot
Bot
added
the
size:XL
zhoujiexiong
left a comment
zhoujiexiong
left a comment
There was a problem hiding this comment.
Is documentation related to this PR being prepared?
|
|
||
| if ngx.req.get_method() ~= "POST" then | ||
| ngx.status = 400 | ||
| ngx.say("Unsupported request method: ", ngx.req.get_method()) |
| return | ||
| end | ||
|
|
||
| if header_auth == "key" then |
| ["Content-Type"] = "application/json", | ||
| ["api-key"] = conf.api_key, | ||
| }, | ||
| body = core.json.encode(body) |
| body = core.json.encode(body) | ||
| }) | ||
| if not res or not res.body then | ||
| return nil, err |
There was a problem hiding this comment.
| return nil, err | |
| return nil, STATUS, err |
| return nil, internal_server_error, err | ||
| end | ||
|
|
||
| if type(res_tab.data) ~= "table" or #res_tab.data < 1 then |
There was a problem hiding this comment.
| if type(res_tab.data) ~= "table" or #res_tab.data < 1 then | |
| if type(res_tab.data) ~= "table" or core.table.isempty(res_tab.data) then |
| end | ||
|
|
||
| local embeddings_provider = next(conf.embeddings_provider) | ||
| local embeddings_provider_conf = conf.embeddings_provider[next(conf.embeddings_provider)] |
There was a problem hiding this comment.
| local embeddings_provider_conf = conf.embeddings_provider[next(conf.embeddings_provider)] | |
| local embeddings_provider_conf = conf.embeddings_provider[embeddings_provider] |
| local vs_req_schema = vector_search_driver.request_schema | ||
| local emb_req_schema = embeddings_driver.request_schema | ||
|
|
||
| request_schema.properties.ai_rag.properties.vector_search = vs_req_schema |
There was a problem hiding this comment.
Are there risks with module variables in concurrent scenarios?
There was a problem hiding this comment.
I don't think so, can you elaborate?
There was a problem hiding this comment.
I checked the process again and there is no risk of race condition. Ignore it.
| core.log.error("dibag err: ", err) | ||
| core.log.warn("dibag res: ", core.json.encode(res, true)) | ||
|
|
||
| body_tab["ai_rag"] = nil |
There was a problem hiding this comment.
comment the purpose in source code?
| if not body_tab.messages then | ||
| body_tab.messages = {} | ||
| end | ||
| decorate(decorator_conf, body_tab) |
There was a problem hiding this comment.
Based on the code, I think you mean "prepend" instead of "append"?
| -- limitations under the License. | ||
| -- | ||
| local core = require("apisix.core") | ||
| local internal_server_error = ngx.HTTP_INTERNAL_SERVER_ERROR |
There was a problem hiding this comment.
| local internal_server_error = ngx.HTTP_INTERNAL_SERVER_ERROR | |
| local INTERNAL_SERVER_ERROR = ngx.HTTP_INTERNAL_SERVER_ERROR |
const var, sugg. this style, others elsewhere the same :)
bzp2010
left a comment
bzp2010
left a comment
There was a problem hiding this comment.
There is no documentation and I don't know how to use it.
| local azure_ai_search_schema = { | ||
| type = "object", | ||
| properties = { | ||
| endpoint = { | ||
| type = "string", | ||
| }, | ||
| api_key = { | ||
| type = "string", | ||
| }, | ||
| } | ||
| } | ||
|
|
||
| local azure_openai_embeddings = { | ||
| type = "object", | ||
| properties = { | ||
| endpoint = { | ||
| type = "string", | ||
| }, | ||
| api_key = { | ||
| type = "string", | ||
| }, | ||
| }, | ||
| required = { "endpoint", "api_key" } | ||
| } |
There was a problem hiding this comment.
This should be moved to the provider Lua file.
| end | ||
|
|
||
|
|
||
| _M.__decorate = decorate -- for ai-rag plugin |
There was a problem hiding this comment.
it is unsafe, each plugin should be independent.
if different plugins are using same function, we should push this function to public library
There was a problem hiding this comment.
I found a better way to augument the request body.
|
|
||
| local http = require("resty.http") | ||
| local core = require("apisix.core") | ||
| local decorate = require("apisix.plugins.ai-prompt-decorator").__decorate |
| local azure_openai_embeddings = require("apisix.plugins.ai-rag.embeddings.azure_openai").schema | ||
| local azure_ai_search_schema = require("apisix.plugins.ai-rag.vector-search.azure_ai_search").schema | ||
|
|
||
| local INTERNAL_SERVER_ERROR = ngx.HTTP_INTERNAL_SERVER_ERROR |
There was a problem hiding this comment.
I suggest we keep the prefix HTTP_
|
|
||
| local _M = { | ||
| version = 0.1, | ||
| priority = 1060, -- TODO check with other ai plugins |
There was a problem hiding this comment.
Do we need this comment TODO check with other ai plugins?
There was a problem hiding this comment.
no we don't need this anymore
| -- limitations under the License. | ||
| -- | ||
| local core = require("apisix.core") | ||
| local INTERNAL_SERVER_ERROR = ngx.HTTP_INTERNAL_SERVER_ERROR |
| -- limitations under the License. | ||
| -- | ||
| local core = require("apisix.core") | ||
| local INTERNAL_SERVER_ERROR = ngx.HTTP_INTERNAL_SERVER_ERROR |
membphis
left a comment
membphis
left a comment
There was a problem hiding this comment.
I do not know much about this feature, only check the code style and how to use this plugin
| | **Field** | **Required** | **Type** | **Description** | | ||
| | ----------------------------------------------- | ------------ | -------- | -------------------------------------------- | | ||
| | embeddings_provider | Yes | Object | Configuration for the embeddings provider | | ||
| | embeddings_provider.azure_openai | Yes | Object | Configuration for Azure OpenAI embeddings | |
There was a problem hiding this comment.
I have one question:
do we will support other provider?
There was a problem hiding this comment.
yes, we might have to support other providers if the need be. So I have written code in a modular way.
There was a problem hiding this comment.
Same issue, I think we'd better to add tip in doc for APISIX user, they should know this
| | vector_search_provider | Yes | Object | Configuration for the vector search provider | | ||
| | vector_search_provider.azure_ai_search | Yes | Object | Configuration for Azure AI Search | | ||
| | vector_search_provider.azure_ai_search.endpoint | No | String | Azure AI Search endpoint | | ||
| | vector_search_provider.azure_ai_search.api_key | No | String | Azure AI Search API key | |
There was a problem hiding this comment.
How come the Azure AI Search config details (endpoint and api key) are optional when their parent azure_ai_search is required?
There was a problem hiding this comment.
oops, my bad.
|
|
||
| | **Field** | **Required** | **Type** | **Description** | | ||
| | --------- | ------------ | -------- | ---------------------------- | | ||
| | fields | Yes | String | Fields for the vector search | |
There was a problem hiding this comment.
| | fields | Yes | String | Fields for the vector search | | |
| | fields | Yes | string | Fields for the vector search | |
|
|
||
| The `ai-rag` plugin integrates Retrieval-Augmented Generation (RAG) capabilities with AI models. | ||
| It allows efficient retrieval of relevant documents or information from external data sources and | ||
| augments the AI's responses with that data, improving the accuracy and context of generated outputs. |
There was a problem hiding this comment.
| augments the AI's responses with that data, improving the accuracy and context of generated outputs. | |
| augments the LLM responses with that data, improving the accuracy and contextual relevance of the generated outputs. |
By using the term LLM here you could probably save this entire sentence
| It allows efficient retrieval of relevant documents or information from external data sources and | ||
| augments the AI's responses with that data, improving the accuracy and context of generated outputs. | ||
|
|
||
| **_This plugin must be used in routes that proxy requests to LLMs only._** |
|
|
||
| **_This plugin must be used in routes that proxy requests to LLMs only._** | ||
|
|
||
| **_As of now only Azure OpenAI and Azure AI Search services are supported for generating embeddings and performing vector search respectively, PRs for introducing support for other service providers are welcome._** |
There was a problem hiding this comment.
grammar fix and links
| **_As of now only Azure OpenAI and Azure AI Search services are supported for generating embeddings and performing vector search respectively, PRs for introducing support for other service providers are welcome._** | |
| **_As of now only [Azure OpenAI](https://azure.microsoft.com/en-us/products/ai-services/openai-service) and [Azure AI Search](https://azure.microsoft.com/en-us/products/ai-services/ai-search) services are supported for generating embeddings and performing vector search respectively. PRs for introducing support for other service providers are welcomed._** |
|
|
||
| | **Field** | **Required** | **Type** | **Description** | | ||
| | --------- | ------------ | -------- | ---------------------------- | | ||
| | fields | Yes | String | Fields for the vector search | |
There was a problem hiding this comment.
| | fields | Yes | String | Fields for the vector search | | |
| | fields | Yes | string | Fields for the vector search | |
| } | ||
| }, | ||
| "override": { | ||
| "endpoint": "'"$azure_openai_endpoint"'" |
There was a problem hiding this comment.
the variables in the PUT request should be all caps as well to match the env vars above
| return nil, HTTP_INTERNAL_SERVER_ERROR, err | ||
| end | ||
|
|
||
| if res.status ~= 200 then |
There was a problem hiding this comment.
use ngx const var instead?
| return nil, HTTP_INTERNAL_SERVER_ERROR, err | ||
| end | ||
|
|
||
| if res.status ~= 200 then |
There was a problem hiding this comment.
use ngx const var instead?
| local embeddings_provider_conf = conf.embeddings_provider[next(conf.embeddings_provider)] | ||
| local embeddings_driver = require("apisix.plugins.ai-rag.embeddings." .. embeddings_provider) | ||
|
|
||
| local vector_search_provider = next(conf.vector_search_provider) |
| local vs_req_schema = vector_search_driver.request_schema | ||
| local emb_req_schema = embeddings_driver.request_schema | ||
|
|
||
| request_schema.properties.ai_rag.properties.vector_search = vs_req_schema |
There was a problem hiding this comment.
I checked the process again and there is no risk of race condition. Ignore it.
bzp2010
left a comment
bzp2010
left a comment
There was a problem hiding this comment.
There are no more comments, so if other reviewers agree with it, I'll agree with it.
7 participants
Description
Implement the ai-rag plugin that uses RAG (Retrieval Augmented Generation), to return information about untrained events/facts from LLMs.
Checklist