CocoIndex provides built-in support for code base chunking, using Tree-sitter to keep syntax boundary. In this example, we will build real-time index for codebase using CocoIndex.
We appreciate a star ⭐ at CocoIndex Github if this is helpful.
Tree-sitter is a parser generator tool and an incremental parsing library. It is available in Rust 🦀 - GitHub. CocoIndex has built-in Rust integration with Tree-sitter to efficiently parse code and extract syntax trees for various programming languages. Check out the list of supported languages here - in the language section.
- We will ingest CocoIndex codebase.
- For each file, perform chunking (Tree-sitter) and then embedding.
- We will save the embeddings and the metadata in Postgres with PGVector.
We will match against user-provided text by a SQL query, reusing the embedding operation in the indexing flow.
Install Postgres if you don't have one.
-
Install dependencies:
pip install -e . -
Update index:
cocoindex update main
-
Run:
python main.py
I used CocoInsight (Free beta now) to troubleshoot the index generation and understand the data lineage of the pipeline. It just connects to your local CocoIndex server, with Zero pipeline data retention. Run the following command to start CocoInsight:
cocoindex server -ci main
Then open the CocoInsight UI at https://cocoindex.io/cocoinsight.
