fix: use OpenAIGenericClient for non-OpenAI endpoints (Gemini, Ollama, vLLM) by Ker102 · Pull Request #1339 · getzep/graphiti

Ker102 · 2026-03-21T05:02:47Z

Problem

When configuring the MCP server's openai LLM provider with a custom api_url pointing to a non-OpenAI endpoint (like Google Gemini's OpenAI-compatible API, Ollama, vLLM, etc.), the LLM client fails with HTTP 404 errors even after fixing the base_url pass-through (#1116).

Root cause: OpenAIClient uses the OpenAI Responses API (/responses) for structured outputs via self.client.responses.parse(). This endpoint is OpenAI-specific — no other provider supports it. OpenAI-compatible endpoints only implement the standard /chat/completions endpoint.

What this PR does

This PR makes two changes to LLMClientFactory.create():

Passes base_url from config to CoreLLMConfig — This is the same fix as fix: pass base_url to OpenAI LLM client for custom endpoint support #1338, included here because the two changes work together.
Auto-detects custom endpoints and uses OpenAIGenericClient — When api_url is set to a non-OpenAI domain (doesn't contain api.openai.com), the factory returns OpenAIGenericClient instead of OpenAIClient. The generic client uses /chat/completions with response_format for structured outputs, which all OpenAI-compatible endpoints support.

The detection is simple and conservative:

custom_url = config.providers.openai.api_url
is_custom_endpoint = custom_url and 'api.openai.com' not in custom_url

If OpenAIGenericClient is not available in the installed version of graphiti-core, it falls back to the standard OpenAIClient behavior (guarded by a try/except import).

How I found this

While working through issue #1116, I fixed the missing base_url and confirmed requests were correctly routed to Gemini. But I was still getting 404 errors. The container logs showed:

HTTP Request: POST .../v1beta/openai/responses "HTTP/1.1 404 Not Found"

After tracing through openai_client.py, I found that _create_structured_completion (line 99) calls self.client.responses.parse() — the Responses API that Gemini doesn't implement. Meanwhile, OpenAIGenericClient._generate_response uses self.client.chat.completions.create() with response_format, which Gemini supports fine.

After applying both fixes:

HTTP Request: POST .../chat/completions "HTTP/1.1 200 OK"
HTTP Request: POST .../embeddings "HTTP/1.1 200 OK"
Successfully processed episode for group modelforge

Testing

Tested with the following configuration:

LLM provider: openai with api_url pointing to https://generativelanguage.googleapis.com/v1beta/openai
LLM model: gemini-2.5-flash
Embedder: openai with gemini-embedding-001

Full write/read cycle verified — episodes are stored and facts are retrievable from the knowledge graph.

Fixes [BUG] OpenAI Provider ignores 'api_base' config and falls back to official API (401 Error with Ollama/Local LLM) #1116
Supersedes fix: pass base_url to OpenAI LLM client for custom endpoint support #1338 (which only had the base_url fix)
Related to # [Feature Request] Fully support Local Model Deployments in MCP Server #1226 (supporting local model deployments)

The OpenAI LLM factory now: 1. Passes base_url from config to CoreLLMConfig (fixes getzep#1116) 2. Auto-detects custom (non-OpenAI) endpoints and uses OpenAIGenericClient instead of OpenAIClient This is needed because OpenAIClient uses the /responses API for structured outputs, which is only supported by api.openai.com. OpenAI-compatible endpoints like Gemini, Ollama, and vLLM only support /chat/completions. OpenAIGenericClient uses /chat/completions with response_format, which works everywhere.

danielchalef · 2026-03-21T05:02:57Z

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

loading1031 · 2026-03-21T05:10:47Z

Update: Tested PR #1339 changes locally with Ollama — confirmed working.

Applied the exact diff from this PR to factories.py and ran end-to-end:

OpenAIGenericClient detected and used: logs show Using OpenAIGenericClient for custom endpoint: http://localhost:11434/v1
Requests go to /chat/completions instead of /responses — all returning 200 OK
add_memory → entity extraction → search_nodes full cycle works

Environment:

LLM: qwen3:8b via Ollama
Embedder: nomic-embed-text via Ollama
Database: FalkorDB (Docker)
graphiti-core: 0.28.2 (already has OpenAIGenericClient)
macOS / Apple Silicon

Ker102 · 2026-03-21T13:53:21Z

@loading1031 Thank you so much for taking the time to test this! 🙏 Really appreciate you applying the diff locally and running a full end-to-end verification with Ollama — that's exactly the kind of real-world validation that gives confidence the fix works across different setups.

Great to see it working smoothly with qwen3:8b + nomic-embed-text on your side too. Your original issue report was super detailed and made it much easier to trace down the root cause, so thank you for that as well!

Hopefully this gets merged soon so everyone hitting #1116 can benefit. 🚀

Ker102 · 2026-03-21T13:54:40Z

I have read the CLA Document and I hereby sign the CLA

Ker102 · 2026-03-21T13:54:52Z

recheck

themavik

the auto-detect logic makes sense. two things: (1) the string check 'api.openai.com' not in custom_url would also match azure endpoints (openai.azure.com) — those don't support /responses either, so that's actually correct behavior. but explicit Azure handling exists in the factory already, so this might cause a double-match. (2) when HAS_OPENAI_GENERIC is False (import fails), custom endpoint users silently get the OpenAI /responses client which will 404. might want a warning log in that case.

themavik

If OpenAIGenericClient is not importable, custom URLs still fall through to OpenAIClient and hit /responses again. Worth a one-line log when the URL is non-OpenAI and the generic client is missing.

@VictorECDSA

feafc422 Bump the uv group across 2 directories with 2 updates (#1363) c4e6923b Upstream Zep internal improvements (#1361) e88c09ca @VictorECDSA has signed the CLA in getzep/graphiti#1356 91fe7e0e @majiayu000 has signed the CLA in getzep/graphiti#1351 c52786d2 @dudo has signed the CLA in getzep/graphiti#1350 d631437d @Ker102 has signed the CLA in getzep/graphiti#1339 73cff2cb @chengjon has signed the CLA in getzep/graphiti#1340 8c617639 @rhlsthrm has signed the CLA in getzep/graphiti#1335 e6424bae @pratyush618 has signed the CLA in getzep/graphiti#1332 6f05647c @bsolomon1124 has signed the CLA in getzep/graphiti#1330 10d91394 @spencer2211 has signed the CLA in getzep/graphiti#1326 1ca14686 Add hiring promotion section to README (#1323) 19e44a97 Bump mcp-server to 1.0.2 and require graphiti-core>=0.28.2 (#1317) 77b16096 Bump graphiti-core version to 0.28.2 (#1315) 7d65d5e7 Harden search filters against Cypher injection (#1312) b10b4889 Restore README title and subtitle (#1314) a9065fa9 Refresh README content and fix image refs (#1313) 5a334ec5 @lvca has signed the CLA in getzep/graphiti#1310 45c8040e @jawherkh has signed the CLA in getzep/graphiti#1309 9eb2c9e8 @kraft87 has signed the CLA in getzep/graphiti#1305 334c8faa @adsharma has signed the CLA in getzep/graphiti#1296 b6f9d874 @StephenBadger has signed the CLA in getzep/graphiti#1295 4b91076a feat: Add GLiNER2 hybrid LLM client (#1284) db54ce09 chore: update Docker images to graphiti-core 0.28.1 (#1292) edc71e8e @devmao has signed the CLA in getzep/graphiti#1289 b4ddc55a @carlos-alm has signed the CLA in getzep/graphiti#1288 aa8e81e3 @giulio-leone has signed the CLA in getzep/graphiti#1280 6fdb352f @aelhajj has signed the CLA in getzep/graphiti#1281 2099603d @avianion has signed the CLA in getzep/graphiti#1278 9eb59f7f @themavik has signed the CLA in getzep/graphiti#1214 98f5b5ff fix: replace edge name with uuid in debug log (#1261) 510bd50d @hanxiao has signed the CLA in getzep/graphiti#1257 17a8ea9e @sprotasovitsky has signed the CLA in getzep/graphiti#1254 9d509a2a @Yifan-233-max has signed the CLA in getzep/graphiti#1245 ef52a2ad chore: regenerate lockfiles to drop diskcache (#1244) 76053036 chore: bump version to 0.28.1 (#1243) bde2f797 fix: replace diskcache with sqlite-based cache to resolve CVE (#1238) git-subtree-dir: graphiti git-subtree-split: feafc422c739f0da166241d4804a9830a294d366

mze3e · 2026-04-16T17:24:35Z

hi any update on this please?

Ker102 · 2026-04-16T17:51:58Z

hi any update on this please?

Im wondering the same. Seems like it hasn't been reviewed yet.

mze3e · 2026-04-16T17:53:28Z

I just checked the official documentation and they’ve added the support already. I tried configuring with my local ollama and it worked ! I guess this ain’t needed anymore

danielchalef · 2026-06-09T14:22:50Z

Thanks for this. The non-OpenAI endpoint routing landed on main: mcp_server/src/services/factories.py now detects non-OpenAI base_urls via is_non_openai_provider() and routes them to OpenAIGenericClient, and passes base_url to the OpenAI LLM client. Closing as superseded by the merged factory refactor.

loading1031 mentioned this pull request Mar 21, 2026

[BUG] OpenAI Provider ignores 'api_base' config and falls back to official API (401 Error with Ollama/Local LLM) #1116

Open

1 of 3 tasks

danielchalef added a commit that referenced this pull request Mar 21, 2026

@Ker102 has signed the CLA in #1339

d631437

lehcode pushed a commit to lehcode/reactive-graphiti that referenced this pull request Mar 22, 2026

@Ker102 has signed the CLA in getzep#1339

dc0a21c

themavik reviewed Mar 22, 2026

View reviewed changes

themavik reviewed Mar 23, 2026

View reviewed changes

danielchalef closed this Jun 9, 2026

zep-cla-assistant Bot locked and limited conversation to collaborators Jun 9, 2026

Navigation Menu

Repository navigation

fix: use OpenAIGenericClient for non-OpenAI endpoints (Gemini, Ollama, vLLM) - #1339

fix: use OpenAIGenericClient for non-OpenAI endpoints (Gemini, Ollama, vLLM)#1339
Ker102 wants to merge 1 commit into
getzep:mainfrom
Ker102:fix/openai-generic-client-for-custom-endpoints

Ker102 commented Mar 21, 2026

Uh oh!

danielchalef commented Mar 21, 2026 •

edited

Loading

Uh oh!

loading1031 commented Mar 21, 2026 •

edited

Loading

Uh oh!

Ker102 commented Mar 21, 2026

Uh oh!

Ker102 commented Mar 21, 2026

Uh oh!

Ker102 commented Mar 21, 2026

Uh oh!

themavik left a comment

Uh oh!

themavik left a comment

Uh oh!

mze3e commented Apr 16, 2026

Uh oh!

Ker102 commented Apr 16, 2026

Uh oh!

mze3e commented Apr 16, 2026

Uh oh!

danielchalef commented Jun 9, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

Ker102 commented Mar 21, 2026

Problem

What this PR does

How I found this

Testing

Related

Uh oh!

danielchalef commented Mar 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

loading1031 commented Mar 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Ker102 commented Mar 21, 2026

Uh oh!

Ker102 commented Mar 21, 2026

Uh oh!

Ker102 commented Mar 21, 2026

Uh oh!

themavik left a comment

Choose a reason for hiding this comment

Uh oh!

themavik left a comment

Choose a reason for hiding this comment

Uh oh!

mze3e commented Apr 16, 2026

Uh oh!

Ker102 commented Apr 16, 2026

Uh oh!

mze3e commented Apr 16, 2026

Uh oh!

danielchalef commented Jun 9, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

danielchalef commented Mar 21, 2026 •

edited

Loading

loading1031 commented Mar 21, 2026 •

edited

Loading