Skip to content

clickhouse-destination: add format directive to configuration#828

Merged
OverOrion merged 2 commits intoaxoflow:mainfrom
bshifter:clickhouse-dest-json-format
Oct 22, 2025
Merged

clickhouse-destination: add format directive to configuration#828
OverOrion merged 2 commits intoaxoflow:mainfrom
bshifter:clickhouse-dest-json-format

Conversation

@bshifter
Copy link
Member

@bshifter bshifter commented Oct 17, 2025

This update allows clickhouse-destination to support multiple JSON structures for ClickHouse ingestion.

  • Supports the existing JSONEachRow format (default).
  • Adds a new JSONCompactEachRow format, which is more compact but less human-readable.
  • Invalid configuration values are rejected with an error, e.g.:
    Error parsing within destination: invalid format value 'invalid-format', possible values: [JSONEachRow, JSONCompactEachRow].

example:

destination {
  clickhouse (
    ...
    json-var(json("$my_filterx_json_var"))
    format("JSONCompactEachRow")
    ...
  );
};

JSONEachRow (each JSON object per line, more readable):

{"id":1,"name":"foo","value":42}
{"id":2,"name":"bar","value":17}

JSONCompactEachRow (compact array-based row representation):

[1,"foo",42]
[2,"bar",17]

@bshifter bshifter force-pushed the clickhouse-dest-json-format branch from ea7ed68 to 64a26c8 Compare October 17, 2025 11:11
@github-actions
Copy link
Contributor

github-actions bot commented Oct 17, 2025

This Pull Request introduces config grammar changes

axoflow/a66ef8b1c2e16b75ca98e6ab4fa386fc3517c27a -> bshifter/clickhouse-dest-json-format

Details
--- a/destination
+++ b/destination

 clickhouse(
+    format(<string>)
 )

MrAnno
MrAnno previously approved these changes Oct 17, 2025
MrAnno
MrAnno previously approved these changes Oct 20, 2025
Copy link
Contributor

@MrAnno MrAnno left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please write a news entry for the users.

@MrAnno
Copy link
Contributor

MrAnno commented Oct 20, 2025

The CI is failing because std::optional is a C++17 feature.

@bshifter bshifter force-pushed the clickhouse-dest-json-format branch from 47085b0 to 9e9bec1 Compare October 20, 2025 10:57
@bshifter bshifter requested a review from MrAnno October 20, 2025 10:58
@bshifter bshifter force-pushed the clickhouse-dest-json-format branch from 9e9bec1 to 577671a Compare October 20, 2025 15:13
This update allows clickhouse-destination to support multiple JSON
structures for ClickHouse ingestion.

In addition to the standard JSONEachRow format, a new JSONCompactEachRow
format is now available. It produces more compact output, reducing size,
though at the cost of human readability.

Signed-off-by: shifter <[email protected]>
@bshifter bshifter force-pushed the clickhouse-dest-json-format branch from 577671a to 0907a8c Compare October 21, 2025 08:12
@bshifter bshifter changed the title clickhouse-destination: add json-format directive to configuration clickhouse-destination: add format directive to configuration Oct 21, 2025
@OverOrion OverOrion merged commit 5b284cd into axoflow:main Oct 22, 2025
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants