Code Review

The pull request successfully addresses the reported issue by correctly propagating the workspaceId to the document loader's init() options, which resolves the $vars being empty during the "Process" step. The changes to _splitIntoChunks() and previewChunks() effectively ensure that the workspaceId is available where needed. Additionally, the removal of the unused docId parameter in deleteVectorStoreFromStore is a good cleanup.

However, several informative comment blocks and inline comments were removed, which could impact the long-term maintainability and understanding of the codebase. While the code remains functional, the absence of these explanations might make it harder for future developers to grasp the intent and logic behind certain operations. It is recommended to restore or rephrase these comments to ensure code clarity.

Code Review

This pull request correctly addresses the issue of $vars not being available in custom loaders by passing the workspaceId through the _splitIntoChunks function. This resolves the authentication failures during the 'Process' step.

However, the PR also includes several refactorings that seem unrelated to the primary fix. I've identified a few potential issues in these changes:

• A breaking API change in the deleteVectorStoreFromStore endpoint, which no longer supports deletion by docId.
• A redundant database write operation within _insertIntoVectorStoreWorkerThread.
• The potential loss of docId in the metadata of documents being upserted to the vector store, which could impact filtering and deletion capabilities.

Please review the detailed comments for suggestions on how to address these points.

return new Document({
    pageContent: chunk.pageContent,
    metadata: {
        ...JSON.parse(chunk.metadata),
        docId: chunk.docId
    }
})

// Configuration already saved by insertIntoVectorStore, just retrieve the entity
const entity = await appDataSource.getRepository(DocumentStore).findOneBy({
    id: data.storeId,
    workspaceId: workspaceId
})
if (!entity) {
    throw new InternalFlowiseError(StatusCodes.NOT_FOUND, `Document store ${data.storeId} not found`)
}

Code Review

This pull request effectively resolves a critical bug where variables were not being read in custom document loaders due to a missing workspaceId. The fix is well-implemented by propagating the workspaceId through the previewChunks and _splitIntoChunks functions, ensuring it's available during the loader's initialization. The changes are clean, targeted, and correctly address the issue described. The detailed problem description and testing notes are also very helpful.