Skip to content

feat: acls#835

Merged
edwinjosechittilappilly merged 25 commits into
mainfrom
connector-acls
Feb 11, 2026
Merged

feat: acls#835
edwinjosechittilappilly merged 25 commits into
mainfrom
connector-acls

Conversation

@phact

@phact phact commented Jan 22, 2026

Copy link
Copy Markdown
Collaborator

This pull request implements comprehensive extraction, propagation, and storage of Access Control List (ACL) information for documents ingested from Google Drive, OneDrive, and SharePoint connectors. It introduces connector-specific logic to fetch detailed user and group permissions from each provider's API and ensures that this ACL data is consistently passed through the document processing pipeline and indexed in OpenSearch. This enables fine-grained access control and auditing for ingested documents.

The most important changes are:

Connector-specific ACL Extraction:

  • Added _extract_google_drive_acl to google_drive/connector.py to fetch and parse user/group permissions from the Google Drive API for each file, and propagate this ACL into ConnectorDocument instances. [1] [2] [3] [4]
  • Added _extract_onedrive_acl to onedrive/connector.py to retrieve permissions from the Microsoft Graph API for OneDrive items, and use this ACL in document creation. [1] [2]
  • Added _extract_sharepoint_acl to sharepoint/connector.py to obtain permissions from the Microsoft Graph API for SharePoint files, and use this ACL in document creation. [1] [2]

Pipeline and Metadata Propagation:

  • Modified the document processing pipeline (service.py and processors.py) to accept and propagate the acl field from connectors through to chunk indexing, ensuring ACLs are stored with each chunk in OpenSearch. [1] [2] [3] [4]

Efficient ACL Indexing and Updates:

  • Refactored _update_connector_metadata in service.py to call a dedicated update_document_acl utility, optimizing ACL updates using hashing to skip unchanged ACLs and updating only when necessary. Other metadata is now updated via a single update_by_query call for efficiency.

These changes collectively provide end-to-end support for extracting, storing, and updating document-level ACLs from external storage providers, improving security and compliance in the document indexing pipeline.

phact and others added 23 commits January 26, 2026 15:14
I noticed during onboarding that even once documents were ingested the frontend continued to poll the backend for task status. This is unnecessary, keeps the backend server busy when it doesn't need to be and fills the logs with polling. This addition stops the onboarding cards from polling when they are completed.
I found the page at /admin, it appears to be an abandoned page for uploading files that has been superseded by /knowledge, so I removed it.
Add end-to-end sync support for cloud connectors. Frontend: introduce useSyncConnector/useSyncAllConnectors hooks and wire sync buttons into the knowledge page and actions dropdown with pending state and toast feedback; invalidate search queries after sync. Backend: add get_synced_file_ids_for_connector and change connector_sync to only re-sync files already indexed in OpenSearch (prefer document_id, fallback to filename) to avoid reimporting deleted items; implement sync_all_connectors to batch-sync existing cloud connector files and emit telemetry. Infrastructure: expose DOCUMENT_ID and SOURCE_URL env vars in docker-compose and Helm values/templates and add them to LANGFLOW variables. Langflow flow: update flows/ingestion_flow.json to add secret inputs and dynamic document_id/source_url form fields and adjust node visibility/positions and defaults. This ensures targeted re-syncs of previously indexed cloud files and provides UI controls to trigger syncs.
Introduce OPENSEARCH_INDEX_NAME (default: documents) to .env.example and wire it into docker-compose so services (including Langflow) receive the index name; update the integration workflow to watch flows/**. Large documentation revisions: clarify onboarding/setup/prereqs, OAuth redirect URIs and webhook guidance, list supported ingestion file types, consolidate troubleshooting links, and reorganize/configure many environment and Langflow-related settings in the configuration reference. Also include multiple minor frontend/backend updates and add SharePoint utils and TUI startup checks to support connector/startup behavior.
Include allowed_users/allowed_groups in search results and frontend types/UI; update chunks page to display access control. Harden OpenSearch interactions: use filename.keyword in aggregations, safer _source access, and stronger logging. Add knn_vector mapping checks/handling to avoid known OpenSearch k-NN NullPointerException, verify dimensions, and skip or warn on mapping issues. Improve embedding generation with tenacity retries and exponential backoff, add IBM/watsonx rate-limit handling (sequential with delays), and better concurrency controls. Misc: hide long embedding vectors from _source, small logging and stability improvements throughout the multimodel OpenSearch component and search service.
Add support for document ACLs (allowed_users, allowed_groups) end-to-end: expose ALLOWED_USERS and ALLOWED_GROUPS env vars and include them in LANGFLOW_VARIABLES_TO_GET_FROM_ENVIRONMENT (docker-compose.yml); extend the ingestion flow UI to accept dynamic allowed_users/allowed_groups inputs and update node metadata/positions; normalize metadata in OpenSearch ingestion components so allowed_users/allowed_groups arriving as JSON strings are parsed into lists (flows/components/opensearch.py and flows/components/opensearch_multimodel.py). Also import (and comment) vector_store_connection in multimodel component, add/adjust raw_search/tool metadata and update related timestamps/code_hash. Minor frontend, connector, service and helm/value changes to propagate the new ACL fields and related defaults.
Update useSyncAllConnectors and useSyncConnector to remove the delayed setTimeout that invalidated the "search" query and instead immediately call queryClient.invalidateQueries({ queryKey: ["tasks"], exact: false }). This ensures new sync jobs show up in the task list right away instead of waiting for a delayed search refetch. (file: frontend/app/api/mutations/useSyncConnector.ts)
@edwinjosechittilappilly edwinjosechittilappilly changed the title acls feat: acls Feb 11, 2026

@edwinjosechittilappilly edwinjosechittilappilly left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@edwinjosechittilappilly

Copy link
Copy Markdown
Collaborator

follow up PR: #949

Delete redundant/duplicated flows/components/opensearch_multimodel.py and clean up opensearch_multimodal.py: remove duplicate json import, import vector_store_connection decorator, and apply @vector_store_connection to the OpenSearch multimodal vector store component class. This consolidates multimodel logic and ensures the component uses the shared vector store connection decorator.

@edwinjosechittilappilly edwinjosechittilappilly left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@edwinjosechittilappilly

Copy link
Copy Markdown
Collaborator

awaiting the integration tests.

@edwinjosechittilappilly

Copy link
Copy Markdown
Collaborator

@mpawlow Can you test aand review the chat and basic ingestion?
I have tested the ACLs in the PR

@edwinjosechittilappilly edwinjosechittilappilly left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@edwinjosechittilappilly edwinjosechittilappilly merged commit 87d6a8c into main Feb 11, 2026
4 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants