Repository navigation
Index kg_edges by each node so connector deletes stop scanning it - #1639
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Deleting a connector took 3 to 7.6 s on Cloud. The connector's knowledge-graph nodes go with it, and for each node Postgres looks up
kg_edgesbysource_node_idand bytarget_node_id. Neither column had an index of its own, so each node cost two full scans ofkg_edges(about 261k rows). The Instagram delete path runs inside one transaction and hit Prisma's 5 s transaction timeout (Sentry ANYTHINGMCP-CLOUD-BACKEND-4, a connector with 30 nodes): the customer got a 500 and the connector stayed.Fixes #1638. Fixes ANYTHINGMCP-CLOUD-BACKEND-4.
Changes
KgEdge:@@index([sourceNodeId])and@@index([targetNodeId]).CREATE INDEX CONCURRENTLY IF NOT EXISTS. CONCURRENTLY does not lock the table for writes during the blue/green overlap (asdeploy/cloud/release.shasks for big tables), and it cannot run in a transaction block, hence one statement per migration. The index names are Prisma's defaults, so there is no drift.Cloud and self-hosted: same migration, applied by
prisma migrate deployat backend start. The previous release runs fine against the new schema (indexes only).Type
Testing
In
packages/backend, against a new empty database on the local Postgres (port 5433):npx prisma migrate deploy: all migrations applied, including the two new ones (CONCURRENTLY accepted).pg_indexesforkg_edgeslistskg_edges_source_node_id_idxandkg_edges_target_node_id_idx.npx prisma migrate diff --from-config-datasource --to-schema prisma/schema.prisma --exit-code: "No difference detected".On Cloud (read-only):
EXPLAIN ANALYZEof the lookup the cascade makes shows a parallel seq scan ofkg_edges, 40-100 ms per node end.No unit test: schema and SQL only, no code change.
Checklist
Related Issues
Fixes #1638