Skip to content

feat(sdk): add Amazon OpenSearch support to Workflow Insight extension#723

Merged
ParidelPooya merged 2 commits into
mainfrom
feat/insight-vscode-opensearch
Jul 15, 2026
Merged

feat(sdk): add Amazon OpenSearch support to Workflow Insight extension#723
ParidelPooya merged 2 commits into
mainfrom
feat/insight-vscode-opensearch

Conversation

@ParidelPooya

Copy link
Copy Markdown
Contributor

Adds Amazon OpenSearch as a queryable destination in the Workflow Insight VS Code extension, alongside DynamoDB / Aurora / Redshift / S3-Athena / CloudWatch Logs. OpenSearch isn't SQL-native and Amazon OpenSearch Service requires SigV4-signed HTTPS, so this uses the OpenSearch SQL plugin for tabular queries.

  • New src/opensearch.ts: runOpenSearchQuery (POST _plugins/_sql, normalizes {schema, datarows} to the shared result shape), fetchOpenSearchRecord (direct GET /_doc/ for the row drill-down), and pingOpenSearch (connectivity probe). All requests are SigV4-signed (service "es") with @smithy/signature-v4 + @aws-crypto/sha256-js over native fetch, using the extension's resolved credentials.
  • schema.ts: new "opensearch" DestinationType and an OpenSearch-SQL prompt (record schema, dialect, few-shots). The index name is hyphenated so it is backtick-quoted; string fields are queried by plain name (the SQL plugin does NOT expose the .keyword multifield); the operations array-of-objects cannot be unnested via the SQL plugin.
  • config.ts / package.json settings: opensearchEndpoint + opensearchIndex. Wired through extension.ts (query + record-fetch routing, index as table name) and destinationTest.ts (SigV4 ping). Settings webview updated (SettingsModal, types, starter query). Adds @smithy/signature-v4 + @aws-crypto/sha256-js deps.

Example CDK stack + exporter fixes (validated end-to-end against a live OpenSearch Service domain):

  • OpenSearchExporter: send the SigV4-signed header set verbatim instead of merging a capital-cased "Content-Type" on top of the signer's lowercase "content-type". Sending both makes fetch collapse them case-insensitively into a value that no longer matches the canonical request SigV4 signed, so the domain rejected every index request with 403 Forbidden (silent, since the plugin runs exporters best-effort). Also include the response body in the thrown error so future failures are diagnosable.
  • stack.ts: pass the domain endpoint to the example function (hoisted so the env var can be set) and output it, so the example function's OpenSearchExporter indexes records.

Tests: unit tests for runOpenSearchQuery/fetchOpenSearchRecord/pingOpenSearch (mocked signer + fetch: normalization, error surfacing, 404, _source flatten) and extended config/destinationTest coverage.

…extension

Adds Amazon OpenSearch as a queryable destination in the Workflow Insight VS
Code extension, alongside DynamoDB / Aurora / Redshift / S3-Athena / CloudWatch
Logs. OpenSearch isn't SQL-native and Amazon OpenSearch Service requires
SigV4-signed HTTPS, so this uses the OpenSearch SQL plugin for tabular queries.

- New src/opensearch.ts: runOpenSearchQuery (POST _plugins/_sql, normalizes
  {schema, datarows} to the shared result shape), fetchOpenSearchRecord (direct
  GET <index>/_doc/<executionArn> for the row drill-down), and pingOpenSearch
  (connectivity probe). All requests are SigV4-signed (service "es") with
  @smithy/signature-v4 + @aws-crypto/sha256-js over native fetch, using the
  extension's resolved credentials.
- schema.ts: new "opensearch" DestinationType and an OpenSearch-SQL prompt
  (record schema, dialect, few-shots). The index name is hyphenated so it is
  backtick-quoted; string fields are queried by plain name (the SQL plugin does
  NOT expose the .keyword multifield); the operations array-of-objects cannot be
  unnested via the SQL plugin.
- config.ts / package.json settings: opensearchEndpoint + opensearchIndex. Wired
  through extension.ts (query + record-fetch routing, index as table name) and
  destinationTest.ts (SigV4 ping). Settings webview updated (SettingsModal,
  types, starter query). Adds @smithy/signature-v4 + @aws-crypto/sha256-js deps.

Example CDK stack + exporter fixes (validated end-to-end against a live
OpenSearch Service domain):

- OpenSearchExporter: send the SigV4-signed header set verbatim instead of
  merging a capital-cased "Content-Type" on top of the signer's lowercase
  "content-type". Sending both makes fetch collapse them case-insensitively into
  a value that no longer matches the canonical request SigV4 signed, so the
  domain rejected every index request with 403 Forbidden (silent, since the
  plugin runs exporters best-effort). Also include the response body in the
  thrown error so future failures are diagnosable.
- stack.ts: pass the domain endpoint to the example function (hoisted so the env
  var can be set) and output it, so the example function's OpenSearchExporter
  indexes records.

Tests: unit tests for runOpenSearchQuery/fetchOpenSearchRecord/pingOpenSearch
(mocked signer + fetch: normalization, error surfacing, 404, _source flatten)
and extended config/destinationTest coverage.

@yaythomas yaythomas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really nice addition — the dialect prompt is clearly written from live validation (the .keyword non-resolution and array-of-objects limitations are exactly the sharp edges of the SQL plugin), and the OpenSearchExporter header fix is a great catch with a first-class explanatory comment. A few small comments inline, nothing blocking.

Comment thread packages/aws-durable-execution-sdk-js-insight-vscode/src/schema.ts Outdated
Comment thread packages/aws-durable-execution-sdk-js-insight-vscode/src/opensearch.ts Outdated
Comment thread packages/aws-durable-execution-sdk-js-insight-vscode/src/opensearch.ts Outdated
Comment thread packages/aws-durable-execution-sdk-js-insight-vscode/src/opensearch.ts Outdated
Comment thread packages/aws-durable-execution-sdk-js-insight-vscode/src/opensearch.ts Outdated
- schema.ts: align the DestinationType header comment with the prompt — the SQL
  plugin can't resolve the .keyword subfield, so filter/group on the plain field
  name (the header still said .keyword).
- opensearch.ts: centralize the endpoint trailing-slash normalization in
  signedFetch (was repeated in all three callers), which now takes a leading-
  slash path; add a query-string guard so a future unsigned ?param fails loudly
  instead of returning an opaque 403; harmonize error-body truncation to a single
  ERR_DETAIL_MAX (was 800/500/300).
- Add countOpenSearchDocs (signed GET <index>/_count) and a second connection-
  test probe that reports the index doc count, soft-passing a 404 (index is
  created on first export) so a typo'd index name is visible without hard-failing
  a fresh domain.
- Document the SQL plugin's default ~200-row cap (plugins.query.size_limit) in
  the endpoint setting description and the dialect prompt, so unbounded list
  queries aren't silently truncated.

Tests updated: countOpenSearchDocs unit tests + destinationTest index-probe
coverage (count, 404 soft-pass, ping-failure).
@ParidelPooya
ParidelPooya marked this pull request as ready for review July 15, 2026 22:58
@ParidelPooya
ParidelPooya merged commit 9882ac6 into main Jul 15, 2026
16 checks passed
@ParidelPooya
ParidelPooya deleted the feat/insight-vscode-opensearch branch July 15, 2026 22:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants