feat: Add Amazon S3 connector and UI#1080
Merged
Merged
Conversation
Introduce a full Amazon S3 / S3-compatible connector and integrate it into the app. Backend: add new S3 connector implementation and auth helpers (src/connectors/aws_s3), register API routes for defaults/configure/list/bucket-status, wire S3 into connection manager and connector registry, and include AWS S3 in sync_all. Frontend: add S3 settings form/dialog, React Query hooks (defaults, configure, bucket status), connect S3 UI into connector cards, cloud picker, and upload flow with an S3 bucket view and direct-sync support. CLI/TUI: add S3-related env fields and config prompts. Misc: small UI icon usages and query invalidation added to keep state in sync.
github-actions
Bot
added
frontend
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds full Amazon S3 (and S3-compatible storage) connector support to OpenRAG, closely following the established IBM COS connector pattern. It includes a Python backend connector, REST API endpoints, TUI configuration fields, frontend settings dialogs, React Query hooks, and bucket status/sync views.
Changes:
- New
src/connectors/aws_s3package withS3Connector, HMAC-based authentication viaboto3, and helper functions for credential resolution with env-var fallback. - Four new API endpoints (
/connectors/aws_s3/defaults,/configure,/{connection_id}/buckets,/{connection_id}/bucket-status) plus registration insrc/main.pyand the connection manager. - Frontend S3 settings dialog, bucket status query hooks, bucket sync view in the upload page, and AWS logo integration in connector cards and knowledge page.
Reviewed changes
Copilot reviewed 18 out of 18 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
src/connectors/aws_s3/__init__.py |
Module init exporting S3Connector |
src/connectors/aws_s3/auth.py |
Credential resolution and boto3 client/resource factory |
src/connectors/aws_s3/connector.py |
S3 connector implementing BaseConnector (list files, download, ACL) |
src/connectors/__init__.py |
Adds S3Connector to package exports |
src/connectors/connection_manager.py |
Registers aws_s3 connector type in factory and availability check |
src/api/connectors.py |
S3 API endpoints (defaults, configure, list buckets, bucket status) and adds aws_s3 to sync_all_connectors |
src/main.py |
Registers the four S3 API routes |
src/tui/managers/env_manager.py |
Adds aws_s3_endpoint and aws_region config fields and env-var loading |
src/tui/config_fields.py |
Adds S3 Endpoint URL and AWS Region TUI config fields |
frontend/app/api/mutations/useS3ConfigureMutation.ts |
React Query mutation for S3 configure endpoint |
frontend/app/api/queries/useS3DefaultsQuery.ts |
React Query hook for S3 defaults |
frontend/app/api/queries/useS3BucketStatusQuery.ts |
React Query hook for S3 bucket status |
frontend/app/settings/_components/s3-settings-dialog.tsx |
S3 configuration dialog with test/save flow |
frontend/app/settings/_components/s3-settings-form.tsx |
S3 credential/endpoint form with bucket selector |
frontend/app/settings/_components/connector-cards.tsx |
Adds S3 dialog and icon mapping |
frontend/app/knowledge/page.tsx |
Adds AWS logo for S3 knowledge sources |
frontend/app/upload/[provider]/page.tsx |
Adds S3BucketView and S3 to direct-sync providers |
frontend/components/cloud-picker/file-item.tsx |
Adds AWS icon for aws_s3 provider |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
Comment on lines
+60
to
+61
| aws_s3_endpoint: str = "" | ||
| aws_region: str = "" |
There was a problem hiding this comment.
The new environment variables AWS_S3_ENDPOINT and AWS_REGION are added to EnvConfig (line 60-61) and the load mapping (line 205-206), but they are missing from the optional_vars list in the save_env method (around line 535-545). This means these values will be loaded from an existing .env file but will be silently dropped when the env file is re-saved through the TUI.
Add the two new entries to the optional_vars list in the save_env method, e.g. after the AWS_SECRET_ACCESS_KEY entry:
("AWS_S3_ENDPOINT", self.config.aws_s3_endpoint) and ("AWS_REGION", self.config.aws_region).
| """Amazon S3 / S3-compatible storage connector for OpenRAG.""" | ||
|
|
||
| import mimetypes | ||
| import os |
There was a problem hiding this comment.
The os module is imported but never used in this file. The S3 connector delegates all credential resolution to auth.py, so this import can be removed.
Comment on lines
+200
to
+364
| function S3BucketView({ | ||
| connector, | ||
| syncMutation, | ||
| addTask, | ||
| onBack, | ||
| onDone, | ||
| }: { | ||
| connector: any; | ||
| syncMutation: ReturnType<typeof useSyncConnector>; | ||
| addTask: (id: string) => void; | ||
| onBack: () => void; | ||
| onDone: () => void; | ||
| }) { | ||
| const queryClient = useQueryClient(); | ||
| const { data: buckets, isLoading, refetch } = useS3BucketStatusQuery( | ||
| connector.connectionId, | ||
| { enabled: true }, | ||
| ); | ||
|
|
||
| const [syncingBucket, setSyncingBucket] = useState<string | null>(null); | ||
|
|
||
| const invalidateBucketStatus = () => { | ||
| queryClient.invalidateQueries({ queryKey: ["s3-bucket-status", connector.connectionId] }); | ||
| }; | ||
|
|
||
| const syncAll = () => { | ||
| syncMutation.mutate( | ||
| { | ||
| connectorType: connector.type, | ||
| body: { | ||
| connection_id: connector.connectionId!, | ||
| selected_files: [], | ||
| sync_all: true, | ||
| }, | ||
| }, | ||
| { | ||
| onSuccess: (result) => { | ||
| invalidateBucketStatus(); | ||
| if (result.task_ids?.length) { | ||
| addTask(result.task_ids[0]); | ||
| onDone(); | ||
| } else { | ||
| toast.info("No files found in any bucket."); | ||
| } | ||
| }, | ||
| onError: (err) => { | ||
| toast.error(err instanceof Error ? err.message : "Sync failed"); | ||
| }, | ||
| }, | ||
| ); | ||
| }; | ||
|
|
||
| const syncBucket = (bucketName: string) => { | ||
| setSyncingBucket(bucketName); | ||
| syncMutation.mutate( | ||
| { | ||
| connectorType: connector.type, | ||
| body: { | ||
| connection_id: connector.connectionId!, | ||
| selected_files: [], | ||
| bucket_filter: [bucketName], | ||
| }, | ||
| }, | ||
| { | ||
| onSuccess: (result) => { | ||
| setSyncingBucket(null); | ||
| invalidateBucketStatus(); | ||
| if (result.task_ids?.length) { | ||
| addTask(result.task_ids[0]); | ||
| onDone(); | ||
| } else { | ||
| toast.info(`No files found in bucket "${bucketName}".`); | ||
| } | ||
| }, | ||
| onError: (err) => { | ||
| setSyncingBucket(null); | ||
| toast.error(err instanceof Error ? err.message : "Sync failed"); | ||
| }, | ||
| }, | ||
| ); | ||
| }; | ||
|
|
||
| return ( | ||
| <> | ||
| <div className="mb-8 flex gap-2 items-center"> | ||
| <Button variant="ghost" onClick={onBack} size="icon"> | ||
| <ArrowLeft size={18} /> | ||
| </Button> | ||
| <h2 className="text-xl text-[18px] font-semibold"> | ||
| Add from {connector.name} | ||
| </h2> | ||
| </div> | ||
|
|
||
| <div className="max-w-3xl mx-auto space-y-4"> | ||
| <div className="flex items-center justify-between"> | ||
| <p className="text-sm text-muted-foreground"> | ||
| Select a bucket to ingest, or sync everything at once. | ||
| </p> | ||
| <div className="flex gap-2"> | ||
| <Button | ||
| variant="ghost" | ||
| size="icon" | ||
| onClick={() => refetch()} | ||
| disabled={isLoading} | ||
| title="Refresh bucket status" | ||
| > | ||
| <RefreshCw size={16} className={isLoading ? "animate-spin" : ""} /> | ||
| </Button> | ||
| <Button | ||
| className="bg-foreground text-background hover:bg-foreground/90 font-semibold" | ||
| onClick={syncAll} | ||
| disabled={syncMutation.isPending} | ||
| > | ||
| {syncMutation.isPending && !syncingBucket | ||
| ? "Ingesting…" | ||
| : "Sync All Buckets"} | ||
| </Button> | ||
| </div> | ||
| </div> | ||
|
|
||
| {isLoading ? ( | ||
| <div className="flex justify-center py-8"> | ||
| <div className="animate-spin rounded-full h-6 w-6 border-b-2 border-primary" /> | ||
| </div> | ||
| ) : !buckets?.length ? ( | ||
| <div className="rounded-lg border p-6 text-center text-muted-foreground text-sm"> | ||
| No buckets found. Check your S3 credentials and endpoint. | ||
| </div> | ||
| ) : ( | ||
| <div className="rounded-lg border divide-y"> | ||
| {buckets.map((bucket) => ( | ||
| <div | ||
| key={bucket.name} | ||
| className="flex items-center justify-between px-4 py-3" | ||
| > | ||
| <div className="flex items-center gap-3"> | ||
| <div> | ||
| <p className="font-medium text-sm">{bucket.name}</p> | ||
| {bucket.ingested_count > 0 && ( | ||
| <p className="text-xs text-muted-foreground"> | ||
| {bucket.ingested_count} document{bucket.ingested_count !== 1 ? "s" : ""} ingested | ||
| </p> | ||
| )} | ||
| </div> | ||
| </div> | ||
| <Button | ||
| variant="outline" | ||
| size="sm" | ||
| onClick={() => syncBucket(bucket.name)} | ||
| disabled={syncMutation.isPending} | ||
| > | ||
| {syncingBucket === bucket.name | ||
| ? "Ingesting…" | ||
| : bucket.ingested_count > 0 | ||
| ? "Re-sync" | ||
| : "Ingest"} | ||
| </Button> | ||
| </div> | ||
| ))} | ||
| </div> | ||
| )} | ||
| </div> | ||
| </> | ||
| ); | ||
| } |
There was a problem hiding this comment.
S3BucketView is nearly identical to IBMCOSBucketView above (same props, same UI, same sync logic). The only differences are the status query hook (useS3BucketStatusQuery vs useIBMCOSBucketStatusQuery) and the query key for invalidation. Consider extracting a shared BucketSyncView component parameterized by the status query hook and query key to avoid maintaining two copies of the same ~160-line component.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces full Amazon S3 connector support to the frontend, including configuration, bucket selection, and status querying, similar to the existing IBM COS integration. It adds new React Query hooks, S3-specific settings dialogs and forms, and ensures S3 is handled throughout the app wherever connectors are managed or displayed.
Amazon S3 Integration
useS3ConfigureMutationfor configuring S3 connections (frontend/app/api/mutations/useS3ConfigureMutation.ts)useS3DefaultsQueryfor fetching S3 configuration defaults (frontend/app/api/queries/useS3DefaultsQuery.ts)useS3BucketStatusQueryfor retrieving bucket ingestion status (frontend/app/api/queries/useS3BucketStatusQuery.ts)frontend/app/settings/_components/s3-settings-dialog.tsx,frontend/app/settings/_components/s3-settings-form.tsx) [1] [2]frontend/app/settings/_components/connector-cards.tsx) [1] [2] [3] [4]UI/UX Enhancements
frontend/app/knowledge/page.tsx,frontend/app/settings/_components/connector-cards.tsx) [1] [2] [3]Connector Handling and Upload Flow
aws_s3in the list of direct-sync providers for upload and ingestion, ensuring S3 buckets are handled like IBM COS buckets (frontend/app/upload/[provider]/page.tsx) (frontend/app/upload/[provider]/page.tsxR12, frontend/app/upload/[provider]/page.tsxL23-R24)These changes collectively enable users to securely configure, test, and manage Amazon S3 connectors within the app, with a user experience and code structure consistent with other supported cloud providers.