Skip to content

feat(sdk): add Amazon Redshift support to Workflow Insight#722

Merged
ParidelPooya merged 2 commits into
mainfrom
feat/insight-vscode-redshift
Jul 15, 2026
Merged

feat(sdk): add Amazon Redshift support to Workflow Insight#722
ParidelPooya merged 2 commits into
mainfrom
feat/insight-vscode-redshift

Conversation

@ParidelPooya

Copy link
Copy Markdown
Contributor

Adds Amazon Redshift as a queryable destination in the Workflow Insight VS Code extension, alongside the existing DynamoDB / Aurora / S3-Athena / CloudWatch Logs destinations. Redshift is modeled on the Aurora path (SQL over the same record_json + scalar-columns table) with the Redshift-specific differences handled:

  • New src/redshift.ts: runRedshiftQuery + fetchRedshiftRecord over the Redshift Data API. Unlike the RDS Data API, the Redshift Data API is asynchronous, so it executes then polls DescribeStatement to completion and pages through GetStatementResult, normalizing to the shared result shape.
  • schema.ts: new "redshift" DestinationType and a dedicated NL->SQL prompt (record schema, dialect, few-shots) covering SUPER/PartiQL navigation. JSON keys are camelCase, so the prompt requires double-quoted SUPER paths (record_json."functionName", o."durationMs") and array UNNEST via a joined table alias.
  • config.ts / package.json settings: redshift workgroup/cluster, database, schema, table, dbUser, and secretArn (Serverless or provisioned). Wired through extension.ts (query + record-fetch routing) and destinationTest.ts (SELECT 1 connectivity probe). Settings webview updated (SettingsModal, types, starter query). Adds @aws-sdk/client-redshift-data.

Example CDK stack + exporter fixes (validated end-to-end against a live Redshift Serverless workgroup):

  • RedshiftExporter: rewrite the upsert MERGE to project the row in the USING subquery and join on src.execution_arn. The previous form joined on a bound parameter, which Redshift plans as a NestedLoop join and rejects with "NestedLoop join is not supported in MERGE" — so every export silently failed.
  • redshift-create-table handler: run multiple statements (splits on ';') so the provisioning SQL can CREATE TABLE and GRANT in one custom resource.
  • stack.ts: GRANT ALL on the table TO PUBLIC (the table is owned by the create-table role's DB user; the exporter and query identities are different users and were otherwise denied), set enable_case_sensitive_identifier=true on the workgroup so camelCase SUPER keys are reachable, and pass the Redshift env vars to the example function so it emits records via RedshiftExporter.

Tests: unit tests for runRedshiftQuery/fetchRedshiftRecord (polling, pagination, failure, SUPER parse) and extended config/destinationTest coverage.

…tension

Adds Amazon Redshift as a queryable destination in the Workflow Insight VS Code
extension, alongside the existing DynamoDB / Aurora / S3-Athena / CloudWatch
Logs destinations. Redshift is modeled on the Aurora path (SQL over the same
record_json + scalar-columns table) with the Redshift-specific differences
handled:

- New src/redshift.ts: runRedshiftQuery + fetchRedshiftRecord over the Redshift
  Data API. Unlike the RDS Data API, the Redshift Data API is asynchronous, so
  it executes then polls DescribeStatement to completion and pages through
  GetStatementResult, normalizing to the shared result shape.
- schema.ts: new "redshift" DestinationType and a dedicated NL->SQL prompt
  (record schema, dialect, few-shots) covering SUPER/PartiQL navigation. JSON
  keys are camelCase, so the prompt requires double-quoted SUPER paths
  (record_json."functionName", o."durationMs") and array UNNEST via a joined
  table alias.
- config.ts / package.json settings: redshift workgroup/cluster, database,
  schema, table, dbUser, and secretArn (Serverless or provisioned). Wired
  through extension.ts (query + record-fetch routing) and destinationTest.ts
  (SELECT 1 connectivity probe). Settings webview updated (SettingsModal, types,
  starter query). Adds @aws-sdk/client-redshift-data.

Example CDK stack + exporter fixes (validated end-to-end against a live Redshift
Serverless workgroup):

- RedshiftExporter: rewrite the upsert MERGE to project the row in the USING
  subquery and join on src.execution_arn. The previous form joined on a bound
  parameter, which Redshift plans as a NestedLoop join and rejects with
  "NestedLoop join is not supported in MERGE" — so every export silently failed.
- redshift-create-table handler: run multiple statements (splits on ';') so the
  provisioning SQL can CREATE TABLE and GRANT in one custom resource.
- stack.ts: GRANT ALL on the table TO PUBLIC (the table is owned by the
  create-table role's DB user; the exporter and query identities are different
  users and were otherwise denied), set enable_case_sensitive_identifier=true on
  the workgroup so camelCase SUPER keys are reachable, and pass the Redshift
  env vars to the example function so it emits records via RedshiftExporter.

Tests: unit tests for runRedshiftQuery/fetchRedshiftRecord (polling,
pagination, failure, SUPER parse) and extended config/destinationTest coverage.
@ParidelPooya
ParidelPooya marked this pull request as ready for review July 15, 2026 20:55
@ParidelPooya
ParidelPooya merged commit be54c31 into main Jul 15, 2026
16 checks passed
@ParidelPooya
ParidelPooya deleted the feat/insight-vscode-redshift branch July 15, 2026 20:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants