We want to be able to store JSON log events in Pinot so that they can be queried efficiently and so that we can reduce storage costs. Part of this involves encoding unstructured message fields in the log event using a new log compressor called CLP. The other part is to transform the log event to fit a table's schema (e.g., extracting nested fields and storing them in a column). We think this can be done with a custom StreamMessageDecoder and a few UDFs.

We've written more about the motivation and proposal here.