feat: Add ACL support and harden OpenSearch handling#948
Merged
edwinjosechittilappilly merged 4 commits intoFeb 11, 2026
Conversation
Include allowed_users/allowed_groups in search results and frontend types/UI; update chunks page to display access control. Harden OpenSearch interactions: use filename.keyword in aggregations, safer _source access, and stronger logging. Add knn_vector mapping checks/handling to avoid known OpenSearch k-NN NullPointerException, verify dimensions, and skip or warn on mapping issues. Improve embedding generation with tenacity retries and exponential backoff, add IBM/watsonx rate-limit handling (sequential with delays), and better concurrency controls. Misc: hide long embedding vectors from _source, small logging and stability improvements throughout the multimodel OpenSearch component and search service.
Add support for document ACLs (allowed_users, allowed_groups) end-to-end: expose ALLOWED_USERS and ALLOWED_GROUPS env vars and include them in LANGFLOW_VARIABLES_TO_GET_FROM_ENVIRONMENT (docker-compose.yml); extend the ingestion flow UI to accept dynamic allowed_users/allowed_groups inputs and update node metadata/positions; normalize metadata in OpenSearch ingestion components so allowed_users/allowed_groups arriving as JSON strings are parsed into lists (flows/components/opensearch.py and flows/components/opensearch_multimodel.py). Also import (and comment) vector_store_connection in multimodel component, add/adjust raw_search/tool metadata and update related timestamps/code_hash. Minor frontend, connector, service and helm/value changes to propagate the new ACL fields and related defaults.
Update useSyncAllConnectors and useSyncConnector to remove the delayed setTimeout that invalidated the "search" query and instead immediately call queryClient.invalidateQueries({ queryKey: ["tasks"], exact: false }). This ensures new sync jobs show up in the task list right away instead of waiting for a delayed search refetch. (file: frontend/app/api/mutations/useSyncConnector.ts)
edwinjosechittilappilly
merged commit a43c527
into
connector-acls-sharepoint
6 checks passed
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds end-to-end ACL plumbing (connector → Langflow ingestion → OpenSearch → API/search results → UI) and improves OpenSearch query/mapping handling to be more robust.
Changes:
- Propagate ACL fields (
allowed_users,allowed_groups, plus owner) through ingestion and return them in search results. - Harden OpenSearch handling (use
filename.keywordfor aggregations; improve raw search behavior and embedding/ingest resilience in the Langflow components/flow). - Improve SharePoint connector ACL extraction and apply ACL even when using cached download URLs.
Reviewed changes
Copilot reviewed 13 out of 14 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| src/utils/acl_utils.py | Small refactor to make ACL-update decision clearer before bulk updating chunks. |
| src/services/search_service.py | Adjust aggregation field to filename.keyword; harden hit parsing; include ACL fields in returned chunks. |
| src/services/langflow_file_service.py | Add ACL parameters and serialize them into Langflow global vars via headers. |
| src/connectors/sharepoint/connector.py | Update Graph permissions URL pattern and broaden parsing to v2 permission fields; extract ACL for cached files too. |
| src/connectors/langflow_connector_service.py | Extract ACL from connector documents (when present) and pass through to ingestion. |
| helm/openrag/values.yaml | Expose ALLOWED_USERS/ALLOWED_GROUPS to Langflow flows via env var list. |
| helm/openrag/templates/langflow/deployment.yaml | Provide default Langflow env values for ALLOWED_USERS/ALLOWED_GROUPS. |
| frontend/app/knowledge/chunks/page.tsx | Display ACL/owner information in the chunk/file detail UI. |
| frontend/app/api/queries/useGetSearchQuery.ts | Extend types/mapping to include ACL fields on chunks/files. |
| frontend/app/api/mutations/useSyncConnector.ts | Invalidate tasks immediately after starting a sync so new jobs appear in the task list. |
| flows/ingestion_flow.json | Wire ACL variables into the ingestion flow; update OpenSearch component configuration/metadata. |
| flows/components/opensearch_multimodel.py | Harden OpenSearch multi-model component behavior (raw search input handling, mapping updates, retries, ACL normalization, agg field). |
| flows/components/opensearch.py | Normalize ACL fields during ingest; switch filename agg to filename.keyword. |
| docker-compose.yml | Add defaults and env exposure for ALLOWED_USERS/ALLOWED_GROUPS in local Langflow setup. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| import json | ||
| import time | ||
| import uuid | ||
| import json |
There was a problem hiding this comment.
Duplicate import json was introduced (it now appears twice at the top of the file). Remove the redundant import to avoid confusion and keep imports clean.
Suggested change
| import json |
| @@ -1,4 +1,5 @@ | |||
| import copy | |||
| import json | |||
There was a problem hiding this comment.
json is imported but not used anywhere in this module. If it’s not needed, please remove the import; otherwise, consider adding the missing usage (or leaving a brief comment explaining why it’s required).
Suggested change
| import json |
| <div className="space-y-1"> | ||
| {fileData.allowed_users.map((user, idx) => ( | ||
| <div | ||
| key={user ?? idx} |
There was a problem hiding this comment.
The React key for allowed users/groups uses key={user ?? idx} / key={group ?? idx}. If the ACL list contains duplicate strings (which is plausible for permissions data), keys will collide and can cause incorrect list reconciliation. Consider using a key that is always unique per item (e.g., include the index in addition to the value).
Suggested change
| key={user ?? idx} | |
| key={`${user ?? "user"}-${idx}`} |
Comment on lines
+468
to
480
| # Get access token - use same approach as _make_graph_request | ||
| access_token = self.oauth.get_access_token() | ||
|
|
||
| if not access_token: | ||
| logger.warning(f"No access token available for ACL extraction: {file_id}") | ||
| return DocumentACL() | ||
|
|
||
| # Determine the correct path for permissions API call | ||
| # Use the same URL pattern as _get_file_metadata_by_id and list_files | ||
| site_info = self._parse_sharepoint_url() | ||
| if site_info: | ||
| site_id = site_info.get("site_id") | ||
| drive_id = site_info.get("drive_id") | ||
| if site_id and drive_id: | ||
| permissions_url = f"{self._graph_base_url}/sites/{site_id}/drives/{drive_id}/items/{file_id}/permissions" | ||
| else: | ||
| # Fallback to user drive | ||
| permissions_url = f"{self._graph_base_url}/me/drive/items/{file_id}/permissions" | ||
| permissions_url = f"{self._graph_base_url}/sites/{site_info['host_name']}:/sites/{site_info['site_name']}:/drive/items/{file_id}/permissions" | ||
| else: |
There was a problem hiding this comment.
In _extract_sharepoint_acl, the comment says this uses the same approach as _make_graph_request, but the permissions fetch is still done via a raw httpx.AsyncClient().get(...) call (outside the shared helper). Using _make_graph_request here would centralize auth headers, consistent timeouts, and error handling for Graph calls, reducing the risk of a long-hanging request or inconsistent behavior.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces improvements and bug fixes for OpenSearch integration, focusing on access control normalization, index mapping robustness, and logging clarity. The changes enhance how ACL fields are handled during ingestion, improve error handling for dynamic field mapping, and clean up logging and documentation for better maintainability.
Access Control and Metadata Handling:
allowed_usersandallowed_groupsfields in document metadata, ensuring they are always lists even if provided as JSON strings, both inflows/components/opensearch.pyandflows/components/opensearch_multimodel.py. [1] [2]OpenSearch Index Mapping Robustness:
Environment and Configuration:
docker-compose.ymlto set default values forDOCUMENT_IDandSOURCE_URL, addALLOWED_USERSandALLOWED_GROUPSenvironment variables, and include them in the environment variable propagation list for Langflow. [1] [2]Logging and Documentation Improvements:
flows/components/opensearch_multimodel.py, and simplify the class description for the OpenSearch multimodal component. [1] [2] [3] [4] [5] [6]Bug Fixes:
.keywordfield for filenames, ensuring correct aggregation results.