Skip to content

Commit c8240bd

Browse files
authored
GH-50830: [C++][Parquet] Use JsonWriter for LogicalType::ToJSON() (#50877)
### Rationale for this change This change was made because it refactored ToJSON() to build JSON through a proper writer instead of hand-built strings, which removes a correctness risk. It also aligns the codebase's ongoing transition away from manual JSON construction. ### What changes are included in this PR? This PR focused on changing the 8 ToJSON() functions from hand built strings to using JsonWriter. In addition, the test files (schema_test.cc, reader_test.cc) were also updated to reflect these changes. Finally, WriteCrsKeyAndValue were no longer needed so it was deleted. ### Are these changes tested? These changes are tested and verified. There was a parquet-schema-test in which 41/41 testcases passed. There was a parquet-reader-test in which 149/154 passed and 5 were skipped as they were unrelated. A Full ctest -R "^parquet-": 11/11 suites passed. And finally the pre-commit (C++ Format + C++ Lint) came out clean. ### Are there any user-facing changes? Yes, there are user-facing changes. ToJSON() now produces compact JSON instead of spaced JSON. It's the same valid JSON, but different exact bytes: - Before: {"Type": "Decimal", "precision": 10, "scale": 4} - After: {"Type":"Decimal","precision":10,"scale":4} ### AI Disclosure Per the AI-generated code guidance: the 8 ToJSON() transitions to JsonWriter, test file changes, the merge conflict resolutions, and the test were produced with Claude Code, and reviewed and verified by me. Correctness was checked by rebuilding and running the relevant test after every single function conversion, not just once at the end. In addition, catching two mistakes early via failing tests (the Decimal attempt that dropped fields, the Int attempt that did the same). * GitHub Issue: #50830 Authored-by: Nathan Chung <[email protected]> Signed-off-by: Antoine Pitrou <[email protected]>
1 parent b12755a commit c8240bd

7 files changed

Lines changed: 395 additions & 299 deletions

File tree

‎cpp/src/arrow/json/json_writer_internal.cc‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -197,6 +197,12 @@ Result<std::string_view> JsonWriter::GetString() const {
197197
return view;
198198
}
199199

200+
Result<std::string> JsonWriter::GetPrettyString(
201+
const simdjson::fractured_json_options& options) const {
202+
ARROW_ASSIGN_OR_RAISE(std::string_view json, GetString());
203+
return simdjson::fractured_json_string(json, options);
204+
}
205+
200206
void JsonWriter::Clear() {
201207
builder_.clear();
202208
needs_comma_ = false;
@@ -218,4 +224,9 @@ void JsonWriter::BoolField(std::string_view key, bool value) {
218224
Bool(value);
219225
}
220226

227+
void JsonWriter::IntField(std::string_view key, int32_t value) {
228+
Key(key);
229+
Int(value);
230+
}
231+
221232
} // namespace arrow::json

‎cpp/src/arrow/json/json_writer_internal.h‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,9 +58,13 @@ class ARROW_EXPORT JsonWriter {
5858

5959
void StringField(std::string_view key, std::string_view value);
6060
void BoolField(std::string_view key, bool value);
61+
void IntField(std::string_view key, int32_t value);
6162

6263
Result<std::string_view> GetString() const;
6364

65+
Result<std::string> GetPrettyString(
66+
const simdjson::fractured_json_options& options = {}) const;
67+
6468
void Clear();
6569

6670
private:

‎cpp/src/arrow/json/json_writer_internal_test.cc‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -290,4 +290,47 @@ TEST(JsonWriter, WriteValueAllNumberTypes) {
290290
R"({"signed":-42,"unsigned":18446744073709551615,"double":2.5,"big":184467440737095516161234567890})");
291291
}
292292

293+
TEST(JsonWriter, IntField) {
294+
JsonWriter writer;
295+
296+
writer.StartObject();
297+
writer.IntField("a", 42);
298+
writer.EndObject();
299+
300+
ASSERT_OK_AND_ASSIGN(std::string_view json, writer.GetString());
301+
302+
EXPECT_EQ(json, R"({"a":42})");
303+
}
304+
305+
TEST(JsonWriter, GetPrettyString) {
306+
JsonWriter writer;
307+
308+
writer.StartObject();
309+
writer.Key("a");
310+
writer.Int(42);
311+
writer.Key("b");
312+
writer.String("hello");
313+
writer.EndObject();
314+
315+
ASSERT_OK_AND_ASSIGN(std::string pretty, writer.GetPrettyString());
316+
317+
// Pretty output should differ from the compact form (padded spacing, at minimum),
318+
// even though a small object like this may still be rendered on one line.
319+
EXPECT_NE(pretty, R"({"a":42,"b":"hello"})");
320+
321+
// But it should still parse back to the same values.
322+
sj::parser parser;
323+
simdjson::padded_string padded(pretty);
324+
sj::document doc;
325+
ASSERT_EQ(parser.iterate(padded).get(doc), simdjson::SUCCESS);
326+
327+
int64_t a_value;
328+
ASSERT_EQ(doc["a"].get(a_value), simdjson::SUCCESS);
329+
EXPECT_EQ(a_value, 42);
330+
331+
std::string_view b_value;
332+
ASSERT_EQ(doc["b"].get(b_value), simdjson::SUCCESS);
333+
EXPECT_EQ(b_value, "hello");
334+
}
335+
293336
} // namespace arrow::json

‎cpp/src/parquet/printer.cc‎

Lines changed: 105 additions & 110 deletions
Original file line numberDiff line numberDiff line change
@@ -21,9 +21,11 @@
2121
#include <cstdio>
2222
#include <memory>
2323
#include <ostream>
24+
#include <sstream>
2425
#include <string>
2526
#include <vector>
2627

28+
#include "arrow/json/json_writer_internal.h"
2729
#include "arrow/util/key_value_metadata.h"
2830
#include "arrow/util/string.h"
2931

@@ -254,16 +256,18 @@ void ParquetFilePrinter::DebugPrint(std::ostream& stream, std::list<int> selecte
254256
void ParquetFilePrinter::JSONPrint(std::ostream& stream, std::list<int> selected_columns,
255257
const char* filename) {
256258
const FileMetaData* file_metadata = fileReader->metadata().get();
257-
stream << "{\n";
258-
stream << " \"FileName\": \"" << filename << "\",\n";
259-
stream << " \"Version\": \"" << ParquetVersionToString(file_metadata->version())
260-
<< "\",\n";
261-
stream << " \"CreatedBy\": \"" << file_metadata->created_by() << "\",\n";
262-
stream << " \"TotalRows\": \"" << file_metadata->num_rows() << "\",\n";
263-
stream << " \"NumberOfRowGroups\": \"" << file_metadata->num_row_groups() << "\",\n";
264-
stream << " \"NumberOfRealColumns\": \""
265-
<< file_metadata->schema()->group_node()->field_count() << "\",\n";
266-
stream << " \"NumberOfColumns\": \"" << file_metadata->num_columns() << "\",\n";
259+
::arrow::json::JsonWriter writer;
260+
writer.StartObject();
261+
writer.StringField("FileName", filename);
262+
writer.StringField("Version", ParquetVersionToString(file_metadata->version()));
263+
writer.StringField("CreatedBy", file_metadata->created_by());
264+
writer.StringField("TotalRows", std::to_string(file_metadata->num_rows()));
265+
writer.StringField("NumberOfRowGroups",
266+
std::to_string(file_metadata->num_row_groups()));
267+
writer.StringField(
268+
"NumberOfRealColumns",
269+
std::to_string(file_metadata->schema()->group_node()->field_count()));
270+
writer.StringField("NumberOfColumns", std::to_string(file_metadata->num_columns()));
267271

268272
if (selected_columns.empty()) {
269273
for (int i = 0; i < file_metadata->num_columns(); i++) {
@@ -277,161 +281,152 @@ void ParquetFilePrinter::JSONPrint(std::ostream& stream, std::list<int> selected
277281
}
278282
}
279283

280-
stream << " \"Columns\": [\n";
281-
int c = 0;
284+
writer.Key("Columns");
285+
writer.StartArray();
282286
for (auto i : selected_columns) {
283287
const ColumnDescriptor* descr = file_metadata->schema()->Column(i);
284-
stream << " { \"Id\": \"" << i << "\","
285-
<< " \"Name\": \"" << descr->path()->ToDotString() << "\","
286-
<< " \"PhysicalType\": \""
287-
<< TypeToString(descr->physical_type(), descr->type_length()) << "\","
288-
<< " \"ConvertedType\": \"" << ConvertedTypeToString(descr->converted_type())
289-
<< "\","
290-
<< " \"LogicalType\": " << (descr->logical_type())->ToJSON() << " }";
291-
c++;
292-
if (c != static_cast<int>(selected_columns.size())) {
293-
stream << ",\n";
294-
}
288+
writer.StartObject();
289+
writer.StringField("Id", std::to_string(i));
290+
writer.StringField("Name", descr->path()->ToDotString());
291+
writer.StringField("PhysicalType",
292+
TypeToString(descr->physical_type(), descr->type_length()));
293+
writer.StringField("ConvertedType", ConvertedTypeToString(descr->converted_type()));
294+
writer.Key("LogicalType");
295+
writer.RawValue(descr->logical_type()->ToJSON());
296+
writer.EndObject();
295297
}
298+
writer.EndArray();
296299

297-
stream << "\n ],\n \"RowGroups\": [\n";
300+
writer.Key("RowGroups");
301+
writer.StartArray();
298302
for (int r = 0; r < file_metadata->num_row_groups(); ++r) {
299-
stream << " {\n \"Id\": \"" << r << "\", ";
303+
writer.StartObject();
304+
writer.StringField("Id", std::to_string(r));
300305

301306
auto group_reader = fileReader->RowGroup(r);
302307
std::unique_ptr<RowGroupMetaData> group_metadata = file_metadata->RowGroup(r);
303308

304-
stream << " \"TotalBytes\": \"" << group_metadata->total_byte_size() << "\", ";
305-
stream << " \"TotalCompressedBytes\": \"" << group_metadata->total_compressed_size()
306-
<< "\", ";
309+
writer.StringField("TotalBytes", std::to_string(group_metadata->total_byte_size()));
310+
writer.StringField("TotalCompressedBytes",
311+
std::to_string(group_metadata->total_compressed_size()));
307312
auto row_group_sorting_columns = group_metadata->sorting_columns();
308313
if (!row_group_sorting_columns.empty()) {
309-
stream << " \"SortColumns\": [\n";
310-
for (size_t i = 0; i < row_group_sorting_columns.size(); i++) {
311-
stream << " {\"column_idx\": " << row_group_sorting_columns[i].column_idx
312-
<< ", \"descending\": " << row_group_sorting_columns[i].descending
313-
<< ", \"nulls_first\": " << row_group_sorting_columns[i].nulls_first
314-
<< "}";
315-
if (i + 1 != row_group_sorting_columns.size()) {
316-
stream << ",";
317-
}
318-
stream << '\n';
314+
writer.Key("SortColumns");
315+
writer.StartArray();
316+
for (const auto& sorting_column : row_group_sorting_columns) {
317+
writer.StartObject();
318+
writer.IntField("column_idx", sorting_column.column_idx);
319+
writer.IntField("descending", sorting_column.descending);
320+
writer.IntField("nulls_first", sorting_column.nulls_first);
321+
writer.EndObject();
319322
}
320-
stream << " ], ";
323+
writer.EndArray();
321324
}
322-
stream << " \"Rows\": \"" << group_metadata->num_rows() << "\",\n";
325+
writer.StringField("Rows", std::to_string(group_metadata->num_rows()));
323326

324-
// Print column metadata
325-
stream << " \"ColumnChunks\": [\n";
326-
int c1 = 0;
327+
writer.Key("ColumnChunks");
328+
writer.StartArray();
327329
for (auto i : selected_columns) {
328330
auto column_chunk = group_metadata->ColumnChunk(i);
329331
std::shared_ptr<Statistics> stats = column_chunk->statistics();
330-
331332
const ColumnDescriptor* descr = file_metadata->schema()->Column(i);
332-
stream << " {\"Id\": \"" << i << "\", \"Values\": \""
333-
<< column_chunk->num_values() << "\", "
334-
<< "\"StatsSet\": ";
333+
334+
writer.StartObject();
335+
writer.StringField("Id", std::to_string(i));
336+
writer.StringField("Values", std::to_string(column_chunk->num_values()));
335337
if (column_chunk->is_stats_set()) {
336-
stream << R"("True", "Stats": {)";
338+
writer.StringField("StatsSet", "True");
339+
writer.Key("Stats");
340+
writer.StartObject();
337341
if (stats->HasNullCount()) {
338-
stream << R"("NumNulls": ")" << stats->null_count() << "\"";
342+
writer.StringField("NumNulls", std::to_string(stats->null_count()));
339343
}
340344
if (stats->HasDistinctCount()) {
341-
stream << ", "
342-
<< R"("DistinctValues": ")" << stats->distinct_count() << "\"";
345+
writer.StringField("DistinctValues", std::to_string(stats->distinct_count()));
343346
}
344347
if (stats->HasMinMax()) {
345348
std::string min = stats->EncodeMin(), max = stats->EncodeMax();
346-
stream << ", "
347-
<< R"("Max": ")"
348-
<< FormatStatValue(descr->physical_type(), max, descr->logical_type())
349-
<< "\", "
350-
<< R"("Min": ")"
351-
<< FormatStatValue(descr->physical_type(), min, descr->logical_type())
352-
<< "\"";
349+
writer.StringField(
350+
"Max", FormatStatValue(descr->physical_type(), max, descr->logical_type()));
351+
writer.StringField(
352+
"Min", FormatStatValue(descr->physical_type(), min, descr->logical_type()));
353353
if (stats->is_max_value_exact().has_value()) {
354-
stream << ", "
355-
<< R"("IsMaxValueExact": ")"
356-
<< (stats->is_max_value_exact().value() ? "True" : "False") << "\"";
354+
writer.StringField("IsMaxValueExact",
355+
stats->is_max_value_exact().value() ? "True" : "False");
357356
} else {
358-
stream << ", "
359-
<< R"("IsMaxValueExact": "unknown")";
357+
writer.StringField("IsMaxValueExact", "unknown");
360358
}
361359
if (stats->is_min_value_exact().has_value()) {
362-
stream << ", "
363-
<< R"("IsMinValueExact": ")"
364-
<< (stats->is_min_value_exact().value() ? "True" : "False") << "\"";
360+
writer.StringField("IsMinValueExact",
361+
stats->is_min_value_exact().value() ? "True" : "False");
365362
} else {
366-
stream << ", "
367-
<< R"("IsMinValueExact": "unknown")";
363+
writer.StringField("IsMinValueExact", "unknown");
368364
}
369365
}
370-
stream << " },";
366+
writer.EndObject();
371367
} else {
372-
stream << "\"False\",";
368+
writer.StringField("StatsSet", "False");
373369
}
374-
stream << "\n \"Compression\": \""
375-
<< ::arrow::internal::AsciiToUpper(
376-
Codec::GetCodecAsString(column_chunk->compression()))
377-
<< R"(", "Encodings": )";
378-
stream << "\"";
370+
371+
writer.StringField("Compression",
372+
::arrow::internal::AsciiToUpper(
373+
Codec::GetCodecAsString(column_chunk->compression())));
374+
375+
std::ostringstream encodings_stream;
379376
if (column_chunk->encoding_stats().empty()) {
380377
for (auto encoding : column_chunk->encodings()) {
381-
stream << EncodingToString(encoding) << " ";
378+
encodings_stream << EncodingToString(encoding) << " ";
382379
}
383380
} else {
384-
PrintPageEncodingStats(stream, column_chunk->encoding_stats());
381+
PrintPageEncodingStats(encodings_stream, column_chunk->encoding_stats());
385382
}
386-
stream << "\"";
387-
stream << ", "
388-
<< R"("UncompressedSize": ")" << column_chunk->total_uncompressed_size()
389-
<< R"(", "CompressedSize": ")" << column_chunk->total_compressed_size()
390-
<< "\"";
383+
writer.StringField("Encodings", encodings_stream.str());
384+
385+
writer.StringField("UncompressedSize",
386+
std::to_string(column_chunk->total_uncompressed_size()));
387+
writer.StringField("CompressedSize",
388+
std::to_string(column_chunk->total_compressed_size()));
391389

392390
if (column_chunk->bloom_filter_offset()) {
393-
// Output BloomFilter {offset, length}
394-
stream << ", \"BloomFilter\": {"
395-
<< R"("offset": ")" << column_chunk->bloom_filter_offset().value() << "\"";
391+
writer.Key("BloomFilter");
392+
writer.StartObject();
393+
writer.StringField("offset",
394+
std::to_string(column_chunk->bloom_filter_offset().value()));
396395
if (column_chunk->bloom_filter_length()) {
397-
stream << R"(, "length": ")" << column_chunk->bloom_filter_length().value()
398-
<< "\"";
396+
writer.StringField("length",
397+
std::to_string(column_chunk->bloom_filter_length().value()));
399398
}
400-
stream << "}";
399+
writer.EndObject();
401400
}
402401

403402
if (column_chunk->GetColumnIndexLocation()) {
404403
auto location = column_chunk->GetColumnIndexLocation().value();
405-
// Output ColumnIndex {offset, length}
406-
stream << ", \"ColumnIndex\": {"
407-
<< R"("offset": ")" << location.offset;
408-
stream << R"(", "length": ")" << location.length;
409-
stream << "\"}";
404+
writer.Key("ColumnIndex");
405+
writer.StartObject();
406+
writer.StringField("offset", std::to_string(location.offset));
407+
writer.StringField("length", std::to_string(location.length));
408+
writer.EndObject();
410409
}
411410

412411
if (column_chunk->GetOffsetIndexLocation()) {
413412
auto location = column_chunk->GetOffsetIndexLocation().value();
414-
// Output OffsetIndex {offset, length}
415-
stream << ", \"OffsetIndex\": {"
416-
<< R"("offset": ")" << location.offset << "\"";
417-
stream << R"(, "length": ")" << location.length << "\"";
418-
stream << "}";
413+
writer.Key("OffsetIndex");
414+
writer.StartObject();
415+
writer.StringField("offset", std::to_string(location.offset));
416+
writer.StringField("length", std::to_string(location.length));
417+
writer.EndObject();
419418
}
420419

421-
// end of a ColumnChunk
422-
stream << " }";
423-
c1++;
424-
if (c1 != static_cast<int>(selected_columns.size())) {
425-
stream << ",\n";
426-
}
427-
}
428-
429-
stream << "\n ]\n }";
430-
if ((r + 1) != static_cast<int>(file_metadata->num_row_groups())) {
431-
stream << ",\n";
420+
writer.EndObject();
432421
}
422+
writer.EndArray();
423+
writer.EndObject();
433424
}
434-
stream << "\n ]\n}\n";
425+
writer.EndArray();
426+
writer.EndObject();
427+
428+
PARQUET_ASSIGN_OR_THROW(std::string pretty_json, writer.GetPrettyString());
429+
stream << pretty_json << "\n";
435430
}
436431

437432
} // namespace parquet

0 commit comments

Comments
 (0)