feat: add encryption to the config files and add IBM secrets manager#1179
Merged
Conversation
github-actions
Bot
added
backend
edwinjosechittilappilly
previously requested changes
Mar 18, 2026
edwinjosechittilappilly
left a comment
edwinjosechittilappilly
left a comment
Collaborator
There was a problem hiding this comment.
@lucaseduoli Please move the Test Files to the Test Folders.
Also Please Ensure the Encrypt and decrypt is also implemented for IBM COS and AWS S3 connectors also.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces a new encryption layer for secrets persisted to disk (config files, connection storage, and OAuth token caches), with support for retrieving a master secret from IBM Secrets Manager and auto-upgrading legacy plaintext secrets when a master secret is available.
Changes:
- Add
src/utils/encryption.pywith AES-256-GCM secret envelopes plus encrypted read/write helpers for token/cache files. - Encrypt/decrypt provider API keys in
ConfigManagerand connection secrets inConnectionManager, with auto-upgrade behavior. - Refactor Google Drive / OneDrive / SharePoint OAuth token-file handling and AuthService token persistence to use encrypted file utilities; add IBM Secrets Manager SDK dependency.
Reviewed changes
Copilot reviewed 11 out of 12 changed files in this pull request and generated 13 comments.
Show a summary per file
| File | Description |
|---|---|
| uv.lock | Locks IBM Secrets Manager SDK + dependencies. |
| pyproject.toml | Adds ibm-secrets-manager-sdk runtime dependency. |
| src/utils/encryption.py | New encryption + IBM Secrets Manager-backed master-secret retrieval + encrypted file helpers. |
| src/config/config_manager.py | Decrypt provider api_key on load; encrypt on save; auto-upgrade legacy plaintext keys. |
| src/connectors/connection_manager.py | Encrypt/decrypt connector config secrets; attempts auto-upgrade of legacy plaintext values. |
| src/connectors/google_drive/oauth.py | Switch token file IO to encrypted read/write utilities with upgrade behavior. |
| src/connectors/onedrive/oauth.py | Switch MSAL cache IO to encrypted read/write utilities with upgrade behavior. |
| src/connectors/sharepoint/oauth.py | Switch MSAL cache IO to encrypted read/write utilities with upgrade behavior. |
| src/services/auth_service.py | Writes OAuth token files via encrypted file utility. |
| test_encryption.py | Adds ad-hoc encryption/config/connection checks (currently outside pytest’s executed paths). |
| test_ibm_encryption.py | Adds ad-hoc master-secret fallback/cache tests (currently outside pytest’s executed paths). |
| test_oauth_encryption.py | Adds ad-hoc OAuth upgrade checks (not structured as pytest tests). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
github-actions
Bot
added
enhancement
github-actions
Bot
added
tests
enhancement
github-actions
Bot
added
enhancement
github-actions
Bot
added
enhancement
github-actions
Bot
added
enhancement
This comment has been minimized.
This comment has been minimized.
github-actions
Bot
added
documentation
This comment has been minimized.
This comment has been minimized.
Collaborator
|
@lucaseduoli can you add it for IBM COS and AWS S3 keys also ? Also can you double check if the IBM AUTH .... ENv variable is used? |
zzzming
requested changes
Mar 19, 2026
zzzming
left a comment
zzzming
left a comment
Collaborator
There was a problem hiding this comment.
Can you exit when the encryption key cannot be read successfully? It should fail on the start. introduce a env to force quit until all the prerequisite is met
github-actions
Bot
added
documentation
Contributor
|
Build successful! ✅ |
Collaborator
Author
IBM Auth is used, and its added for IBM COS and AWS S3, since they use connection_manager's update_connection, that is encrypted |
zzzming
approved these changes
Mar 19, 2026
edwinjosechittilappilly
pushed a commit
that referenced
this pull request
Mar 24, 2026
…1179) * added encryption based on env defined key * added ibm secrets manager and salt + kdf * changed ibm secrets manager access type and added constants * change test folder and used pytest * added ibm cos and aws keys as secrets * fixed cache not being saved * fixed error not raising * fixed tenant id not expecting the matching tenant * added preserve editred * added decryption failed test * added ibm encryption under the feature flag * Added openrag encryption key to environment, configuration and tui * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <[email protected]> * changed descriptions * added tenant id as env * fixed tests * enforce prerequisites --------- Co-authored-by: Copilot Autofix powered by AI <[email protected]>
lucaseduoli
added a commit
that referenced
this pull request
Mar 24, 2026
…1179) * added encryption based on env defined key * added ibm secrets manager and salt + kdf * changed ibm secrets manager access type and added constants * change test folder and used pytest * added ibm cos and aws keys as secrets * fixed cache not being saved * fixed error not raising * fixed tenant id not expecting the matching tenant * added preserve editred * added decryption failed test * added ibm encryption under the feature flag * Added openrag encryption key to environment, configuration and tui * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <[email protected]> * changed descriptions * added tenant id as env * fixed tests * enforce prerequisites --------- Co-authored-by: Copilot Autofix powered by AI <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces comprehensive encryption and secure storage for sensitive secrets and credentials across the codebase. The most important changes include encrypting provider API keys and connection secrets, upgrading legacy unencrypted secrets to AES-256-GCM, and refactoring OAuth token handling to use encrypted files. These enhancements improve security and ensure backward compatibility by automatically upgrading old secrets.
Encryption of sensitive secrets and credentials
src/config/config_manager.py. Existing unencrypted keys are automatically upgraded to AES-256-GCM if a master secret is available. [1] [2] [3] [4]src/connectors/connection_manager.py. Legacy unencrypted secrets are automatically upgraded. [1] [2]OAuth token handling refactoring
Dependency updates
ibm-secrets-manager-sdktopyproject.tomlto support secure secret management.Code cleanup
aiofilesimports in files that now use encrypted file utilities. [1] [2] [3] [4]