Skip to content

Load tokenizer regexes on first ruleset instantiation#446

Merged
Anilm3 merged 2 commits into
masterfrom
anilm3/lazy-initialise-tokenizer-regexes
Aug 29, 2025
Merged

Load tokenizer regexes on first ruleset instantiation#446
Anilm3 merged 2 commits into
masterfrom
anilm3/lazy-initialise-tokenizer-regexes

Conversation

@Anilm3

@Anilm3 Anilm3 commented Aug 28, 2025 •
edited
Loading

Copy link
Copy Markdown
Collaborator

As the title suggests, this PR changes the initialisation process of the tokenizer regular expressions to avoid the static initialisation cost. These regular expressions are now being preloaded / instantiated on the first ruleset instantiation, which also prevent impacting request latency.

@Anilm3
Anilm3 marked this pull request as ready for review August 28, 2025 15:45
@Anilm3
Anilm3 requested a review from a team as a code owner August 28, 2025 15:45
@codecov-commenter

codecov-commenter commented Aug 28, 2025 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 42.24138% with 67 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.07%. Comparing base (4492d25) to head (8291cf2).
⚠️ Report is 36 commits behind head on master.

Files with missing lines Patch % Lines
src/tokenizer/mysql.cpp 29.62% 13 Missing and 6 partials ⚠️
src/tokenizer/pgsql.cpp 34.61% 10 Missing and 7 partials ⚠️
src/tokenizer/generic_sql.cpp 35.71% 5 Missing and 4 partials ⚠️
src/tokenizer/shell.cpp 38.46% 4 Missing and 4 partials ⚠️
src/tokenizer/sqlite.cpp 38.46% 4 Missing and 4 partials ⚠️
src/tokenizer/sql_base.cpp 57.14% 4 Missing and 2 partials ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##           master     #446      +/-   ##
==========================================
- Coverage   85.36%   85.07%   -0.29%     
==========================================
  Files         179      178       -1     
  Lines        9085     9166      +81     
  Branches     4001     4043      +42     
==========================================
+ Hits         7755     7798      +43     
- Misses        517      545      +28     
- Partials      813      823      +10
Flag Coverage Δ
waf_test 85.07% <42.24%> (-0.29%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions

github-actions Bot commented Aug 28, 2025 •
edited
Loading

Copy link
Copy Markdown

Artifact Size Comparison 📦

Artifact Previous Release This PR Difference
darwin-arm64::libddwaf.a 86105528 86305256 0.00%
darwin-arm64::libddwaf.a.stripped 4442896 4468832 0.00%
darwin-arm64::libddwaf.dylib 1920368 1938624 0.00%
darwin-universal::libddwaf.a 173548224 173973016 0.00%
darwin-universal::libddwaf.a.stripped 9465600 9522592 0.00%
darwin-universal::libddwaf.dylib 4050288 4068544 0.00%
darwin-x86_64::libddwaf.a 87442648 87667712 0.00%
darwin-x86_64::libddwaf.a.stripped 5022656 5053712 0.00%
darwin-x86_64::libddwaf.dylib 2107008 2108752 0.00%
linux-aarch64::libddwaf.a 70896324 71243350 0.00%
linux-aarch64::libddwaf.a.stripped 11575224 11627594 0.00%
linux-aarch64::libddwaf.so 2427912 2434424 0.00%
linux-armv7::libddwaf.a 62633514 62943352 0.00%
linux-armv7::libddwaf.a.stripped 10576914 10625340 0.00%
linux-armv7::libddwaf.so 2108812 2116508 0.00%
linux-i386::libddwaf.a 60811086 61128590 0.00%
linux-i386::libddwaf.a.stripped 9140778 9180290 0.00%
linux-i386::libddwaf.so 2346940 2350020 0.00%
linux-x86_64::libddwaf.a 71366196 71714046 0.00%
linux-x86_64::libddwaf.a.stripped 11397112 11445674 0.00%
linux-x86_64::libddwaf.so 2616768 2622768 0.00%
windows-arm64::ddwaf.dll 4735488 ∞
windows-arm64::ddwaf.lib 11698 ∞
windows-arm64::ddwaf_static.lib 56060578 ∞
windows-win32::ddwaf.dll 3298304 3335168 0.01%
windows-win32::ddwaf.lib 11922 11922 0.00%
windows-win32::ddwaf_static.lib 47761216 48018914 0.00%
windows-x64::ddwaf.dll 4030976 4062208 0.00%
windows-x64::ddwaf.lib 11698 11698 0.00%
windows-x64::ddwaf_static.lib 55348460 55633280 0.00%
windows-x86_64::libddwaf.a 6263042 6283102 0.00%
windows-x86_64::libddwaf.dll 18798493 18811081 0.00%
windows-x86_64::libddwaf.dll.a 31948 31948 0.00%

@pr-commenter

pr-commenter Bot commented Aug 28, 2025 •
edited
Loading

Copy link
Copy Markdown

Benchmarks clang

Benchmark execution time: 2025-08-29 13:23:12

Comparing candidate commit 8291cf2 in PR branch anilm3/lazy-initialise-tokenizer-regexes with baseline commit 4492d25 in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 1 metrics, 0 unstable metrics.

@pr-commenter

pr-commenter Bot commented Aug 28, 2025 •
edited
Loading

Copy link
Copy Markdown

Benchmarks gcc

Benchmark execution time: 2025-08-29 13:23:18

Comparing candidate commit 8291cf2 in PR branch anilm3/lazy-initialise-tokenizer-regexes with baseline commit 4492d25 in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 1 metrics, 0 unstable metrics.

@Anilm3
Anilm3 force-pushed the anilm3/lazy-initialise-tokenizer-regexes branch from e765aab to 89b66af Compare August 28, 2025 16:13
@Anilm3
Anilm3 force-pushed the anilm3/lazy-initialise-tokenizer-regexes branch from 89b66af to 4db447a Compare August 28, 2025 16:18
@pr-commenter

pr-commenter Bot commented Aug 28, 2025 •
edited
Loading

Copy link
Copy Markdown

Benchmarks clang-pgo

Benchmark execution time: 2025-08-29 13:37:03

Comparing candidate commit 8291cf2 in PR branch anilm3/lazy-initialise-tokenizer-regexes with baseline commit 4492d25 in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 1 metrics, 0 unstable metrics.

@Anilm3 Anilm3 changed the title Lazy-initialise tokenizer regexes Load tokenizer regexes on first ruleset instantiation Aug 29, 2025
@Anilm3
Anilm3 force-pushed the anilm3/lazy-initialise-tokenizer-regexes branch from 389c696 to 49662fd Compare August 29, 2025 10:16
@Anilm3
Anilm3 force-pushed the anilm3/lazy-initialise-tokenizer-regexes branch from 49662fd to 8291cf2 Compare August 29, 2025 13:00
@Anilm3
Anilm3 merged commit 2ccf343 into master Aug 29, 2025
57 checks passed
@Anilm3
Anilm3 deleted the anilm3/lazy-initialise-tokenizer-regexes branch August 29, 2025 15:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@RomainMuller RomainMuller RomainMuller approved these changes
@cataphract cataphract cataphract approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@Anilm3 @codecov-commenter @RomainMuller @cataphract