Sometimes you need to export a lot of data to JSON to a file. Maybe it\u2019s \u201cexport all data to JSON\u201d, or the GDPR \u201cRight to portability\u201d, where you effectively need to do the same.<\/p>\n
And as with any big dataset, you can\u2019t just fit it all in memory and write it to a file. It takes a while, it reads a lot of entries from the database and you need to be careful not to make such exports overload the entire system, or run out of memory.<\/p>\n
Luckily, it\u2019s fairly straightforward to do that, with the help of Jackson\u2019s The code does a few things: The main work is done by Jackson\u2019s SequenceWriter, and the (kind of obvious) point to take home is \u2013 don\u2019t assume your data will fit in memory. It almost never does, so do everything in batches and incremental writes.<\/p>\nSequenceWriter<\/code> and optionally of piped streams. Here\u2019s how it would look like:<\/p>\nprivate ObjectMapper jsonMapper = new ObjectMapper();\r\n private ExecutorService executorService = Executors.newFixedThreadPool(5);\r\n\r\n @Async\r\n public ListenableFuture<Boolean> export(UUID customerId) {\r\n try (PipedInputStream in = new PipedInputStream();\r\n PipedOutputStream pipedOut = new PipedOutputStream(in);\r\n GZIPOutputStream out = new GZIPOutputStream(pipedOut)) {\r\n \r\n Stopwatch stopwatch = Stopwatch.createStarted();\r\n\r\n ObjectWriter writer = jsonMapper.writer().withDefaultPrettyPrinter();\r\n\r\n try(SequenceWriter sequenceWriter = writer.writeValues(out)) {\r\n sequenceWriter.init(true);\r\n \r\n Future<?> storageFuture = executorService.submit(() ->\r\n storageProvider.storeFile(getFilePath(customerId), in));\r\n\r\n int batchCounter = 0;\r\n while (true) {\r\n List<Record> batch = readDatabaseBatch(batchCounter++);\r\n for (Record record : batch) {\r\n sequenceWriter.write(entry);\r\n }\r\n }\r\n\r\n \/\/ wait for storing to complete\r\n storageFuture.get();\r\n } \r\n\r\n logger.info(\"Exporting took {} seconds\", stopwatch.stop().elapsed(TimeUnit.SECONDS));\r\n\r\n return AsyncResult.forValue(true);\r\n } catch (Exception ex) {\r\n logger.error(\"Failed to export data\", ex);\r\n return AsyncResult.forValue(false);\r\n }\r\n }<\/pre>\n\n
writeValues(out)<\/code> sounds like you are instructing the writer to write something now; instead it configures it to use the particular stream later.<\/li>\nSequenceWriter<\/code> is initialized with true<\/code>, which means \u201cwrap in array\u201d. You are writing many identical records, so they should represent an array in the final JSON.<\/li>\nPipedOutputStream<\/code> and PipedInputStream<\/code> to link the SequenceWriter<\/code> to a an InputStream<\/code> which is then passed to a storage service. If we were explicitly working with files, there would be no need for that \u2013 simply passing a FileOutputStream<\/code> would do. However, you may want to store the file differently, e.g. in Amazon S3, and there the putObject call requires an InputStream from which to read data and store it in S3. So, in effect, you are writing to an OutputStream which is directly written to an InputStream, which, when attampted to be read from, gets everything written to another OutputStream<\/li>\n