Skip to content

perf(http): speed up varchar parsing in CSV import#5985

Merged
bluestreak01 merged 5 commits intomasterfrom
puzpuzpuz_csv_import_varchar
Jul 25, 2025
Merged

perf(http): speed up varchar parsing in CSV import#5985
bluestreak01 merged 5 commits intomasterfrom
puzpuzpuz_csv_import_varchar

Conversation

@puzpuzpuz
Copy link
Copy Markdown
Contributor

@puzpuzpuz puzpuzpuz commented Jul 23, 2025

Adds fast path to double quote replacement in VarcharAdapter.

Parsing + WAL writes for https://datasets.clickhouse.com/hits_compatible/hits.csv.gz sent to the POST /imp endpoint:

  • before: 480s
  • after: 410s

@puzpuzpuz puzpuzpuz self-assigned this Jul 23, 2025
@puzpuzpuz puzpuzpuz added Performance Performance improvements REST API Issues or changes relating to the HTTP endpoints labels Jul 23, 2025
@puzpuzpuz puzpuzpuz marked this pull request as ready for review July 23, 2025 09:50
@glasstiger
Copy link
Copy Markdown
Contributor

[PR Coverage check]

😍 pass : 22 / 22 (100.00%)

file detail

path covered line new line coverage
🔵 io/questdb/std/str/Utf8s.java 2 2 100.00%
🔵 io/questdb/cutlass/text/types/VarcharAdapter.java 14 14 100.00%
🔵 io/questdb/std/str/DirectUtf8Sink.java 3 3 100.00%
🔵 io/questdb/cutlass/text/AbstractTextLexer.java 2 2 100.00%
🔵 io/questdb/cutlass/text/CopyTask.java 1 1 100.00%

@puzpuzpuz
Copy link
Copy Markdown
Contributor Author

@bluestreak01 thanks for the review!

@bluestreak01 bluestreak01 merged commit d432c7f into master Jul 25, 2025
34 checks passed
@bluestreak01 bluestreak01 deleted the puzpuzpuz_csv_import_varchar branch July 25, 2025 08:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Performance Performance improvements REST API Issues or changes relating to the HTTP endpoints

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants