Skip to content

Commit 1f5edfa

Browse files
authored
Read Amazon report files on the server and add finance tools (#1645)
* Read Amazon report files on the server and add finance tools A REST tool can now follow a download link in its response (endpointMapping.download): fetched server-side through fetchOutbound (SSRF-guarded, no credentials, size caps), gunzipped, parsed as TSV/CSV/JSON and returned one page at a time. amazon_seller_download_report_document uses it. Add amazon_seller_list_transactions, amazon_seller_get_finance_summary and amazon_seller_list_service_fee_events, since listFinancialEvents has no event-type filter. The OpenAPI importer now reads Swagger 2.0 'in: body' parameters. Tools imported from Swagger 2.0 specs (Amazon SP-API) sent no body and no Content-Type, which Amazon answers with 400. * Regenerate llms-full.txt for the new download section * Cap decompressed report downloads at 32 MB
1 parent 73cbbc5 commit 1f5edfa

12 files changed

Lines changed: 923 additions & 11 deletions

‎docs/tool-definition.md‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -819,6 +819,51 @@ out of the result. A tool can set its own values on the endpoint mapping:
819819
- REST connectors only. Tools without these keys behave exactly as before, on
820820
Cloud and self-hosted alike.
821821

822+
### Download links in the response (`download`)
823+
824+
Some APIs answer with a short-lived link to the data instead of the data
825+
(Amazon SP-API report documents: `{ "url": "https://…amazonaws.com/…",
826+
"compressionAlgorithm": "GZIP" }`). An AI client cannot open that link. With a
827+
`download` block the tool makes its API call as usual, then fetches the link on
828+
the server and returns the parsed content:
829+
830+
```json
831+
"endpointMapping": {
832+
"method": "GET",
833+
"path": "/reports/2021-06-30/documents/{reportDocumentId}",
834+
"download": {
835+
"urlField": "url",
836+
"compressionField": "compressionAlgorithm",
837+
"format": "auto",
838+
"allowedHosts": ["amazonaws.com"],
839+
"offset": "$offset",
840+
"limit": "$limit",
841+
"columns": "$columns",
842+
"filterColumn": "$filterColumn",
843+
"filterValue": "$filterValue"
844+
}
845+
}
846+
```
847+
848+
- `urlField` (required): dot path of the link in the API response.
849+
- `allowedHosts` (required): hosts the link may point to; a domain also matches
850+
its subdomains. Only `https` links are fetched.
851+
- `compressionField`: dot path of a marker whose value `GZIP` means gunzip. Gzip
852+
is also recognised by its first bytes.
853+
- `format`: `auto` (default) picks JSON, tab- or comma-separated text from the
854+
content; or `tsv`, `csv`, `json`, `text`.
855+
- `offset`, `limit`, `columns`, `filterColumn`, `filterValue`: references to tool
856+
parameters. Delimited files come back as one page of rows keyed by the header
857+
(`limit` default 50, at most 500), filtered by one column (several values
858+
comma-separated, case-insensitive) and cut to the listed columns, with
859+
`columns`, `totalRows`, `matchedRows` and `nextOffset`. JSON comes back whole
860+
under `data`; other text as the first 20 000 characters.
861+
- The link is fetched without the connector's credentials, with the same SSRF
862+
checks as every outbound call (each redirect included), at most 25 MB
863+
compressed and 32 MB decompressed.
864+
- REST connectors only, on Cloud and self-hosted alike. Tools without the block
865+
behave exactly as before.
866+
822867
### By Connector Type
823868

824869
| Connector | method | path | queryParams | bodyMapping | headers |

‎llms-full.txt‎

Lines changed: 45 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎packages/backend/src/adapters/catalog.spec.ts‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -79,7 +79,9 @@ const REST_HTTP_METHODS = new Set(['GET', 'POST', 'PUT', 'PATCH', 'DELETE']);
7979
function argumentsSent(em: Record<string, unknown>): Set<string> {
8080
const sent = new Set<string>();
8181
for (const m of String(em.path ?? '').matchAll(/\{([^{}]+)\}/g)) sent.add(m[1]);
82-
const fields: unknown[] = [em.queryParams, em.headers];
82+
// `download` reads its paging and filter arguments on the server, after the
83+
// call: they are used, just not sent upstream (download-follow.util.ts).
84+
const fields: unknown[] = [em.queryParams, em.headers, em.download];
8385
const sendsBody = REQUEST_BODY_METHODS.has(String(em.method).toUpperCase());
8486
if (sendsBody) fields.push(em.bodyMapping);
8587
for (const field of fields) {

0 commit comments

Comments
 (0)