feat: acls#835
Merged
Merged
Conversation
I noticed during onboarding that even once documents were ingested the frontend continued to poll the backend for task status. This is unnecessary, keeps the backend server busy when it doesn't need to be and fills the logs with polling. This addition stops the onboarding cards from polling when they are completed.
I found the page at /admin, it appears to be an abandoned page for uploading files that has been superseded by /knowledge, so I removed it.
Add end-to-end sync support for cloud connectors. Frontend: introduce useSyncConnector/useSyncAllConnectors hooks and wire sync buttons into the knowledge page and actions dropdown with pending state and toast feedback; invalidate search queries after sync. Backend: add get_synced_file_ids_for_connector and change connector_sync to only re-sync files already indexed in OpenSearch (prefer document_id, fallback to filename) to avoid reimporting deleted items; implement sync_all_connectors to batch-sync existing cloud connector files and emit telemetry. Infrastructure: expose DOCUMENT_ID and SOURCE_URL env vars in docker-compose and Helm values/templates and add them to LANGFLOW variables. Langflow flow: update flows/ingestion_flow.json to add secret inputs and dynamic document_id/source_url form fields and adjust node visibility/positions and defaults. This ensures targeted re-syncs of previously indexed cloud files and provides UI controls to trigger syncs.
Introduce OPENSEARCH_INDEX_NAME (default: documents) to .env.example and wire it into docker-compose so services (including Langflow) receive the index name; update the integration workflow to watch flows/**. Large documentation revisions: clarify onboarding/setup/prereqs, OAuth redirect URIs and webhook guidance, list supported ingestion file types, consolidate troubleshooting links, and reorganize/configure many environment and Langflow-related settings in the configuration reference. Also include multiple minor frontend/backend updates and add SharePoint utils and TUI startup checks to support connector/startup behavior.
Include allowed_users/allowed_groups in search results and frontend types/UI; update chunks page to display access control. Harden OpenSearch interactions: use filename.keyword in aggregations, safer _source access, and stronger logging. Add knn_vector mapping checks/handling to avoid known OpenSearch k-NN NullPointerException, verify dimensions, and skip or warn on mapping issues. Improve embedding generation with tenacity retries and exponential backoff, add IBM/watsonx rate-limit handling (sequential with delays), and better concurrency controls. Misc: hide long embedding vectors from _source, small logging and stability improvements throughout the multimodel OpenSearch component and search service.
Add support for document ACLs (allowed_users, allowed_groups) end-to-end: expose ALLOWED_USERS and ALLOWED_GROUPS env vars and include them in LANGFLOW_VARIABLES_TO_GET_FROM_ENVIRONMENT (docker-compose.yml); extend the ingestion flow UI to accept dynamic allowed_users/allowed_groups inputs and update node metadata/positions; normalize metadata in OpenSearch ingestion components so allowed_users/allowed_groups arriving as JSON strings are parsed into lists (flows/components/opensearch.py and flows/components/opensearch_multimodel.py). Also import (and comment) vector_store_connection in multimodel component, add/adjust raw_search/tool metadata and update related timestamps/code_hash. Minor frontend, connector, service and helm/value changes to propagate the new ACL fields and related defaults.
Update useSyncAllConnectors and useSyncConnector to remove the delayed setTimeout that invalidated the "search" query and instead immediately call queryClient.invalidateQueries({ queryKey: ["tasks"], exact: false }). This ensures new sync jobs show up in the task list right away instead of waiting for a delayed search refetch. (file: frontend/app/api/mutations/useSyncConnector.ts)
Collaborator
|
follow up PR: #949 |
Delete redundant/duplicated flows/components/opensearch_multimodel.py and clean up opensearch_multimodal.py: remove duplicate json import, import vector_store_connection decorator, and apply @vector_store_connection to the OpenSearch multimodal vector store component class. This consolidates multimodel logic and ensures the component uses the shared vector store connection decorator.
Collaborator
|
awaiting the integration tests. |
Collaborator
|
@mpawlow Can you test aand review the chat and basic ingestion? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request implements comprehensive extraction, propagation, and storage of Access Control List (ACL) information for documents ingested from Google Drive, OneDrive, and SharePoint connectors. It introduces connector-specific logic to fetch detailed user and group permissions from each provider's API and ensures that this ACL data is consistently passed through the document processing pipeline and indexed in OpenSearch. This enables fine-grained access control and auditing for ingested documents.
The most important changes are:
Connector-specific ACL Extraction:
_extract_google_drive_acltogoogle_drive/connector.pyto fetch and parse user/group permissions from the Google Drive API for each file, and propagate this ACL intoConnectorDocumentinstances. [1] [2] [3] [4]_extract_onedrive_acltoonedrive/connector.pyto retrieve permissions from the Microsoft Graph API for OneDrive items, and use this ACL in document creation. [1] [2]_extract_sharepoint_acltosharepoint/connector.pyto obtain permissions from the Microsoft Graph API for SharePoint files, and use this ACL in document creation. [1] [2]Pipeline and Metadata Propagation:
service.pyandprocessors.py) to accept and propagate theaclfield from connectors through to chunk indexing, ensuring ACLs are stored with each chunk in OpenSearch. [1] [2] [3] [4]Efficient ACL Indexing and Updates:
_update_connector_metadatainservice.pyto call a dedicatedupdate_document_aclutility, optimizing ACL updates using hashing to skip unchanged ACLs and updating only when necessary. Other metadata is now updated via a singleupdate_by_querycall for efficiency.These changes collectively provide end-to-end support for extracting, storing, and updating document-level ACLs from external storage providers, improving security and compliance in the document indexing pipeline.