Skip to content

split block by max size#1065

Merged
yokofly merged 1 commit intodevelopfrom
split-chunk-for-http-sink
Dec 16, 2025
Merged

split block by max size#1065
yokofly merged 1 commit intodevelopfrom
split-chunk-for-http-sink

Conversation

@yokofly
Copy link
Copy Markdown
Collaborator

@yokofly yokofly commented Dec 16, 2025

PR checklist:

  • Did you run ClangFormat ?
  • Did you separate headers to a different section in existing community code base ?
  • Did you surround proton: starts/ends for new code in existing community code base ?

Please write user-readable short description of the changes:
For HTTP databricks sample, insert fails when more than one row in chunk; even for max_insert_block_size is explicitly set
https://docs.timeplus.com/databricks-external

CREATE EXTERNAL STREAM http_databricks_t1 (product string, quantity int)
SETTINGS
type = 'http',
http_header_Authorization='Bearer $TOKEN',
url = 'https://$HOST.cloud.databricks.com/api/2.0/sql/statements/',
data_format = 'Template',
format_template_resultset_format='{"warehouse_id":"$WAREHOUSE_ID","statement": "INSERT INTO sales (product, quantity) VALUES (:product, :quantity)", "parameters": [${data}]}',
format_template_row_format='{ "name": "product", "value": ${product:JSON}, "type": "STRING" },{ "name": "quantity", "value": ${quantity:JSON}, "type": "INT" }',
format_template_rows_between_delimiter=''

INSERT INTO http_databricks_t1(product, quantity) VALUES('test',95)('test2', 96);

@yokofly yokofly merged commit c5a7931 into develop Dec 16, 2025
9 of 11 checks passed
@yokofly yokofly deleted the split-chunk-for-http-sink branch December 16, 2025 02:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants