Allow "brimcap load" of log files in non-auto-detected formats #80
Description
Right now brimcap load seems dependent on the log outputs of analyzers being auto-detect-able. This works fine for the main cases of Zeek and Suricata since their default outputs are Zeek TSV and NDJSON, respectively. However, while working on #72 I happened to try working with a NetFlow analyzer's CSV output. Using these configs with Brimcap v0.0.3, we can see how it fails because we don't currently auto-detect CSV.
$ cat nfdump-wrapper-csv.sh
#!/bin/bash
TMPFILE=$(mktemp)
cat - > "$TMPFILE"
nfpcapd -r "$TMPFILE" -l .
rm "$TMPFILE"
for file in nfcapd.*
do
nfdump -r $file -o csv | ghead -n -3 > ${file}.csv
done
$ cat nfdump-csv.yml
analyzers:
- cmd: nfdump-wrapper-csv.sh
globs: ["*.csv"]
$ brimcap load -root "$HOME/Library/Application Support/Brim/data/brimcap-root" -config nfdump-csv.yml -p testpool ~/pcap/wrccdc.pcap
100.0% 500.0MB/500.0MB records=0
Post "http://localhost:9867/pool/1sArPVhV4gBbiH5B1E8NytXJ34G/log": format detection error
tzng: line 1: bad format
zeek: line 1: bad types/fields definition in zeek header
zjson: line 1: invalid character 's' in literal true (expecting 'r')
zson: identifier "ts" must be enum and requires decorator
zng: zng type ID out of range
parquet: auto-detection not supported
zst: auto-detection not supported
If we wanted to support this, I can think of two ways to proceed.
- Allow the specification of input format in
brimcap load(similar to thezq -ioptions) - Add auto-detection support for all formats at the Zed layer (Auto-detect all input formats for files super#2517)