-
Notifications
You must be signed in to change notification settings - Fork 52
feat: record queries by sqlite #321
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
396ec0b to
71645ae
Compare
8a96a32 to
dbe998e
Compare
3cc03ba to
80ec6e2
Compare
1013474 to
2c02ee5
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR implements query logging functionality for the vchordrq index type using SQLite storage. It adds GUC parameters to control query logging behavior and provides functions and views to access logged query data.
Key changes:
- Added SQLite-based query collection system with configurable sampling and limits
- Introduced three new GUC parameters for controlling query logging behavior
- Added SQL functions and views to retrieve logged queries with proper access control
Reviewed Changes
Copilot reviewed 17 out of 18 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| src/lib.rs | Adds collector module initialization |
| src/collector/*.rs | Implements SQLite-based query logging infrastructure |
| src/index/gucs.rs | Adds new GUC parameters for query logging configuration |
| src/index/functions.rs | Adds function to retrieve logged samples with access control |
| src/sql/finalize.sql | Defines SQL functions and views for query access |
| src/index//scanners/.rs | Integrates query logging into search builders |
| src/index/*/am/mod.rs | Adds sender configuration for query collection |
| tests/vchordrq/recall.slt | Adds comprehensive tests for query logging functionality |
| Cargo.toml | Adds rusqlite dependency |
| scripts/train.py | Reduces training iterations (unrelated optimization) |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
2c02ee5 to
37e09db
Compare
dd14985 to
2dce740
Compare
|
| Latency | dim | minimum | maximum | majority |
|---|---|---|---|---|
| insert at <1000 records + maintain | 3 | 3ms | 5ms | 5ms |
| insert at ~5000 records + maintain | 3 | 5ms | 6ms | 5ms |
| insert at ~10000 records + maintain | 3 | 10ms | 13ms | 10ms |
| insert at <1000 records + maintain | 768 | 3ms | 6ms | 5ms |
| insert at ~5000 records + maintain | 768 | 4ms | 6ms | 5ms |
2c51021 to
cbd7800
Compare
cbd7800 to
a6986b6
Compare
What does this show? Shouldn't we conduct a controlled experiment with variables? |
a6986b6 to
585cd38
Compare
|
Benifit from The latency bottleneck is caused by every |
585cd38 to
8883c12
Compare
|
lgtm. Shall we add a reset interface, to let user recreate the sqlite db if it doesn't work properly after some crash? |
53c1c15 to
76ecc21
Compare
12d2003 to
9b9a8f1
Compare
Signed-off-by: cutecutecat <[email protected]>
9b9a8f1 to
075155e
Compare
We implement a simple strategy that deletes an SQLite database file if corruption is detected upon opening. Then, the next time when VectorChord/src/recorder/worker.rs Line 64 in 9e1bbe1
|
Real-time sampling of SQL queries for recall evaluation and monitoring. Signed-off-by: cutecutecat <[email protected]>
job: +psql_windows
job: +psql_macos
job: +check_debian
job: +psql_alpine
GUC
vchordrq.query_sampling_enable: Whether to enable the query loggervchordrq.query_sampling_max_records: Maximum logged queries per indexvchordrq.query_sampling_rate: Sample rate of query to be loggedFunction
vchordrq_sampled_queries(index: Oid): Show logged queries for the given indexvchordrq_sampled_vectors(index: Oid): Get logged vector samples (only for internal use).View
vchordrq_sampled_queries: Show logged queries for all indexes which you have accessExample
Restriction
vchordg indexis not enabledInternal
Structure
INIT: Connect and create a disk-basedsqlite tableatPG_initINSERT: Insert tosqlite tablewhen query logged, and maintain max rows tomax_logged_queries_per_indexfor each indexDELETE: Delete rows belongs to an index fromsqlite tablewhen aDROP TABLEorDROP INDEXtriggered hookREAD: Read all rows belongs to an PG index when the functionvchordrq_logged_queriescalledSynchronous
INSERT: If cannot get the lock, then skipREAD/DELETE: If cannot get the lock, then waitLatency