Skip to content

Version 5.0.0

Choose a tag to compare

@lemire lemire released this 28 Sep 12:28
· 164 commits to master since this release
f5d9dd0

Main changes (Major release)

  1. Static reflection is no longer guarded and is an officially supported feature. When your
    compiler has reflection enabled (e.g., g++ -std=c++26 -freflection with
    GCC 16), simdjson detects it by itself and the reflection-based functions
    become available.

  2. When deserializing a C++ structure through reflection, simdjson now uses
    key selectors (see below) by default: it reads the object in a single pass,
    whatever the order of the keys. You can return to the previous approach
    (one lookup per member) with -DSIMDJSON_DISABLE_KEY_SELECTOR_REFLECTION=1.

  3. A positive integer in [2^64, 10^20) is now
    reported as a big integer (BIGINT_NUMBER), like other overflowing
    integers, instead of a malformed number. When big integers are parsed as
    strings, a token such as 123456789123456789123x is now rejected.

One major change are the key selectors. A common task is to extract a few fields from a JSON object. In simdjson 5.0 (C++20 or better), you can name the keys at compile time and visit the object once:

using namespace simdjson;
auto json = R"({ "name": "Daniel", "age": 42, "city": "Montreal" })"_padded;
ondemand::parser parser;
auto doc = parser.iterate(json);

std::string_view name, city;
uint64_t age = 0;
auto result = doc.get_object().for_each<"name", "city", "age">(name, city, age);
// name == "Daniel", city == "Montreal", age == 42

The keys can appear in any order. At compile time, simdjson builds a perfect
hash function for your set of keys. At run time, recognizing a key takes a hash
computed from a couple of bytes and one comparison. You can also pass one
callback per key instead of variables. The iteration stops as soon as all keys
have been found.

We added annotation for the data structures for automated (C++26) serialization and deserialization: rename, rename_all,
alias, skip, default_value, flatten, deny_unknown_fields,
transparent, and so forth.

struct [[= simdjson::rename_all<simdjson::case_style::camel_case>]] User {
  std::string first_name;
  int64_t user_id;
  [[= simdjson::rename<"KEY">]] int api_key;
};
// {"firstName":"Ann","userId":7,"KEY":8}

We often get many JSON documents in one file or one network message. simdjson
has long supported streams of documents separated by white space (NDJSON). In
5.0 we added:

  • RFC 7464 JSON text sequences
    (each document preceded by the record separator character) and
    comma-separated documents;
  • stream_format::newline_delimited: you promise that each document sits on
    its own line. When you only read part of a document, simdjson jumps to the
    next line instead of walking over the rest of the document.
  • simdjson::slice_at, which cuts a stream into blocks at document
    boundaries so you can parse the blocks on as many threads as you like. The
    built-in threaded mode uses at most two threads.

We also fixed several bugs in document_stream, found in an
audit by Francisco Geiman Thiesen.

The are many other smaller features.

  • NaN and infinity: JSON does not allow NaN or Infinity, but many
    systems produce them anyway. If you define SIMDJSON_ENABLE_NAN_INF, simdjson
    parses them, and serializes them.
  • Narrow types: get_uint8(), get_int8(), get_uint16(), get_int16()
    check the range for you. With C++23, get_float32() and get_float64()
    return std::float32_t and std::float64_t. The binary32 value is rounded
    once, directly from the decimal string, not through a double.
  • The DOM API can parse a buffer that has no padding
    (parser.parse_unpadded(...)). It is slower than the regular function, but
    it never reads past the end of your buffer and it does not copy your data.
  • With C++17, simdjson::padded_input adds padding only when it is needed:
    when your string ends near a page boundary.
  • C++20 ranges: you can pipe On-Demand arrays and objects into
    std::views::transform and other adaptors.
  • DOM arrays support reverse iteration (rbegin(), rend()) with no
    allocation.
  • On-Demand objects offer get_current_position() and revert_position(): if
    you miss an optional field, you can go back to where you were instead of
    rescanning the whole object.
  • char8_t (u8) variants of the string accessors in C++20.
  • Better pretty printing with the FracturedJson
    style, including tables.
  • Memory-mapped files under Windows.
  • We support for the memory-safe compiler Fil-C.

Performance

The simdjson 5.0 release improved performance compared to simdjson 4.0 in some key cases. Let me review some of them.

I built both versions withGCC 16.1 (-O3, CMake Release) and ran them on an Intel Xeon Gold 6548N (Emerald Rapids), pinned to one core.

Let us start with DOM parsing of our standard files (GB/s):

file 4.0 5.0 speedup
twitter 4.78 4.82 1.0
citm_catalog 4.82 4.79 1.0
github_events 5.33 5.30 1.0
canada 1.10 1.21 1.1
marine_ik 1.25 1.38 1.1
mesh 1.17 1.26 1.1
numbers 1.13 1.41 1.3
twitterescaped 1.59 2.85 1.8
update-center 3.96 3.84 1.0
apache_builds 4.95 4.78 1.0

Files full of numbers (canada, marine_ik, mesh, numbers) are 8% to 25% faster.
And a file full of escaped Unicode characters (twitterescaped) is almost twice
as fast: among other changes, we now decode consecutive \uXXXX sequences without going back to the
string scanner between them.

We also serialize faster. Printing floating-point numbers used to be a bottleneck. We replaced the ancient Grisu2
by Dragonbox, and removed calls to memcpy and
memmove from the hot path.

file 4.0 5.0 speedup
twitter 0.94 0.96 1.0
citm_catalog 1.07 1.08 1.0
gsoc-2018 1.10 1.25 1.1
canada 0.31 0.52 1.7
marine_ik 0.28 0.37 1.3
mesh 0.34 0.47 1.4
numbers 0.32 0.49 1.6

List

  • Get reflection working without warnings under GCC 16 by @lemire in #2658
  • Fix failing just_ascii test by @jaja360 in #2661
  • literals should be guarded by @lemire in #2669
  • adding a C++17 padded_input for convenience. by @lemire in #2667
  • amalgamate should provide nicer error messages by @lemire in #2672
  • Various guards by @lemire in #2673
  • Add support for RFC 7464 JSON text sequences and comma-delimited documents by @jaja360 in #2664
  • finishing up the new streaming by @lemire in #2674
  • modernizing the benchmarks by @lemire in #2679
  • adding memory-file mapping to Windows + better doc by @lemire in #2676
  • Add std::ranges support for On-Demand API (variant) by @lemire in #2678
  • Fix undefined behavior in document_stream parsing by @metsw24-max in #2680
  • fix: ondemand for wildcard matches by @lemire in #2686
  • new optimization to the ARM NEON kernel by @lemire in #2692
  • builds with Fil-C by @lemire in #2695
  • Align source() output for comma-delimited streams with json_sequence behavior by @jmestwa-coder in #2699
  • builder: coalesce per-field separator+key+colon into one constexpr write by @FranciscoThiesen in #2698
  • fixing issue 2703 by @lemire in #2705
  • builder: force-inline atom templates and replace integer writer by @FranciscoThiesen in #2707
  • Add support for parsing NaN and Infinity as requested in #1540, #2414, and #2540 by @codeinred in #2696
  • adding gcc16 by @lemire in #2712
  • Add support for writing NaN/Infinity if SIMDJSON_ENABLE_NAN_INF is enabled by @codeinred in #2711
  • builder: position-as-local writer for reflection serializer (+55% CITM) by @FranciscoThiesen in #2708
  • JSON Pointer array index overflow handling by @jmestwa-coder in #2713
  • fix -Wunneeded-internal-declaration warning by @codeinred in #2714
  • avoid unnecessary pointer member by @lemire in #2715
  • [skip ci] Add section to basics.md documenting the SIMDJSON_ENABLE_NAN_INF flag by @aylindmello in #2718
  • add simdjson-dart binding to docs by @xaldarof in #2719
  • docs: use HTTPS for arXiv link in README by @wyf027 in #2726
  • docs: use HTTPS for Boost license link in README by @wyf027 in #2725
  • docs: use HTTPS for YouTube embed links in README by @wyf027 in #2724
  • adding silent flag to amalgamation. by @lemire in #2721
  • perf(builder): add LSX fast_needs_escaping for LoongArch by @wyf027 in #2727
  • Bug fix for get_string() crash when parse incomplete json string (in option disabled my default) by @wankunde in #2720
  • simplify the tests for memory mapping under Windows. by @lemire in #2729
  • add: cpp26 annotations support for rename and skip by @hiteshmk05 in #2730
  • docs: clarify raw_json() consumption and rewind option by @wyf027 in #2734
  • let us use lower-case words for layout_mode by @lemire in #2740
  • adding clang 20 to our CI coverage. by @lemire in #2739
  • ci: fix warning about deprecated node.js by @pauldreik in #2744
  • DOM without padding (experimental) by @lemire in #2775
  • Compile-time key selectors: single-pass, order-independent field extraction by @lemire in #2776
  • grisu2 to dragonbox as underlying logic for to_chars by @fior512 in #2777
  • using __builtin_popcountll in the hope that it will pick up better optimizations by @lemire in #2779
  • to_chars.cpp, 2 digits at a time and avoiding reversal loop by @fior512 in #2781
  • to_chars.cpp, fast path to avoid i128 emulation when possible by @fior512 in #2780
  • Add LASX (LoongArch 256-bit SIMD) fast path for JSON string builder by @vsaraikin in #2778
  • Fix incorrect doc comments by @lemire in #2784
  • Treat 20-digit positive overflows as big integers by @Yusufihsangorgel in #2793
  • Remove dead structural_iterator class by @vpelikh in #2792
  • Remove unused add_overflow from all architecture backends by @vpelikh in #2790
  • README update, 'Real-world usage' projects that changed their name by @fior512 in #2797
  • arm64: test comparison masks with shrn+fcmp instead of a horizontal reduce by @lemire in #2796
  • Faster write_string_escaped: single-pass block escaping (SSE2, NEON) by @lemire in #2795
  • Fix throw on unterminated bracket-quoted JSONPath keys in wildcard walks by @lemire in #2801
  • Remove unused escape helpers and add write_string_escaped tests by @lemire in #2804
  • Detect C++26 static reflection instead of requiring a build flag by @lemire in #2798
  • perf: eliminate libc memcpy/memset/memmove calls in to_chars hotpath by @fior512 in #2805
  • fixing a typo in if-logic of perfdiff.cpp by @fior512 in #2813
  • Remove ineffective parse skip-flag benchmarks and dead scripts by @lemire in #2816
  • Add FreeBSD CI by @neilpang in #2811
  • fix: prevent parse_unpadded 1-byte OOB on truncated nested JSON by @lemire in #2817
  • fix: LTO workaround by @fior512 in #2818
  • Fix failing just_ascii test (em dashes in comments) by @lemire in #2821
  • perf: remove remaining variable-size memmove/memset in to_chars by @lemire in #2808
  • Add newline_delimited, slice_at, and skip documents by delimiter by @lemire in #2803
  • docs: fix cmake -B build directory in HACKING.md by @ezralicodes in #2823
  • Reject malformed big integers with trailing non-structural characters by @Avogar in #2820
  • Speed up the json2msgpack on-demand benchmark by @lemire in #2819
  • Fix ambiguous atom()/append() on optional serialization (#2827) by @lemire in #2828
  • Add 32-bit binary float support to the On Demand API by @lemire in #2825
  • New benchmark for int64, double and number for both ondemand and dom by @fior512 in #2807
  • Add get_float to the number accessor benchmark by @lemire in #2829
  • [skip ci] remove dead CI configs and the stale fuzz job docs by @advitrocks9 in #2830
  • Vendor fast_float without old-style casts by @lemire in #2833
  • feat(ondemand): protect object iteration via state-based container locking by @Pastoray in #2809
  • cmake: do not enable SIMDJSON_DEVELOPMENT_CHECKS in Release by @lemire in #2838
  • ondemand: json_type::unknown has no ostream arm by @advitrocks9 in #2835
  • fix at_path bug by @lemire in #2841
  • Add a JSONPath fuzz target by @taimoor789 in #2837
  • Add C++20 u8 variants of string access methods (#2731) by @AlbertoFVisconti in #2786
  • fix wildcard path recursion depth by @lemire in #2844
  • doc: recommend restricting max_depth to protect recursive user code by @lemire in #2845
  • fix for #2848 by @lemire in #2849
  • Add object::get_current_position()/revert_position() to avoid full rescans on optional-field misses by @taimoor789 in #2846
  • Require POWER8 vector support for ppc64 SIMD code by @pkubaj in #2851
  • More secure document stream by @lemire in #2853
  • Fix DOM at_path error propagation by @lilong555 in #2855
  • Remove unused competition dependencies (cjson, jsmn, gason, fastjson,… by @pratap9572singh in #2862
  • Allow tag_invoke to throw without std::terminate (#2858) by @lemire in #2872
  • arm64: write structural indexes in groups of 4 then 2 by @lemire in #2869
  • Fix line break handling for fractured json formatting by @maxbachmann in #2870
  • builder: restore pos + n <= cap in writer::ensure (citm serialization +14-20%) by @lemire in #2874
  • Fix GCC 11 -Wmaybe-uninitialized false positive in document_stream_tests by @lemire in #2875
  • Speed up integer parsing by reading four digits at a time by @lemire in #2873
  • Arm64 sve runtime detection by @lemire in #2867
  • Add C++20 char8_t string accessors to the On-Demand API (#2731) by @lemire in #2843
  • Speed up float parsing by branching on the first integer digit by @lemire in #2876
  • Serde-style annotations: default, alias, flatten, rename_all, deny_unknown_fields, ... (#2660) by @lemire in #2877
  • Closer match options for FracturedJson by @maxbachmann in #2878
  • Add prettify benchmark by @Cuda-Chen in #2868
  • Add allocation-free reverse iteration for DOM arrays by @lilong555 in #2857
  • Add On-Demand getters for narrower integers and std::float32_t/std::float64_t by @lemire in #2881
  • remove some overloads by @lemire in #2856
  • perf: stage2: decode adjacent Unicode escapes without rescanning by @mpurbay-arm in #2882
  • FracturedJson table formatting by @maxbachmann in #2879

New Contributors

Full Changelog: v4.6.1...v5.0.0