Update "Custom Brimcap Config" wiki article#340
Conversation
docs/Custom-Brimcap-Config.md
Outdated
| and [Suricata](https://suricata.readthedocs.io/en/latest/install.html#install-binary-packages). | ||
| We'll use Linux Ubuntu 20.04 as our OS platform. On such a host, the following |
There was a problem hiding this comment.
Strictly speaking, we can't say that GA releases of both Zeek and Suricata are newer than the ones with Brimcap, since we're now up-to-date on Zeek. Therefore I dropped that part of the text.
docs/Custom-Brimcap-Config.md
Outdated
| 2. Other changes are made to the default configuration of the Brimcap-bundled | ||
| Zeek, such as enabling | ||
| [Community ID Flow Hashing](https://docs.zeek.org/en/master/customizations.html#community-id). | ||
| See the [build-zeek release automation](https://github.com/brimdata/build-zeek/blob/main/.github/workflows/release.yml) | ||
| for details on how this and other customizations are handled. |
There was a problem hiding this comment.
We used to use a Zeek package for Community ID, but now it's included out-of-the-box with Zeek so I've been taking advantage of that in the Zeek builds that come out of the new build-zeek repo. This gave me a nice excuse to link to the build-zeek repo, since it's a better starting place than what we had before for users that want to try their hand at making their own custom Zeek builds, including on Windows.
| to be invoked when you open or drag pcap files into Zui. | ||
|
|
||
|  | ||
|  |
There was a problem hiding this comment.
The Zui menu used to be called "Preferences" in some OSes and "Settings" on others, but thankfully we've standardized on "Settings" across the board now.
| $ cat zeek-wrapper.sh | ||
| #!/bin/bash | ||
| exec /opt/zeek/bin/zeek -C -r - --exec "event zeek_init() { Log::disable_stream(PacketFilter::LOG); Log::disable_stream(LoadedScripts::LOG); }" local | ||
| exec /opt/zeek/bin/zeek -C -r - --exec "event zeek_init() { Log::disable_stream(PacketFilter::LOG); Log::disable_stream(LoadedScripts::LOG); Log::disable_stream(Telemetry::LOG); }" local |
There was a problem hiding this comment.
The newer Zeek releases have these telemetry logs that are suited primarily to gathering perf info about running deployments (e.g., live capture environments) but they still get generated when processing pcaps. Their volume dwarfs the amount of actual analyzed events when small pcaps are processed, and it seems doubtful their contents would be essential viewing for the pcap use case. I've been excluding them via the Zeek Runners that are bundled with the builds from the new build-zeek repo, so I do the same here.
| As we did with Zeek and Suricata, we create a [wrapper script](https://github.com/brimdata/brimcap/blob/main/examples/nfdump-wrapper.sh) to act as our | ||
| Brimcap analyzer. It works in two phases, first creating binary NetFlow records | ||
| and then converting them to CSV. `nfpcapd` only accepts a true pcap file input | ||
| and then converting them to NDJSON. `nfpcapd` only accepts a true pcap file input |
There was a problem hiding this comment.
When I ran the latest nfdump today for the first time in a while, it seems they recently mucked up their CSV support (they now output one of their fields onto a separate line, which made Zed's CSV reader choke), but the good news is that they added an NDJSON output format that I'm more than happy to recommend instead. I'd only been using nfdump's CSV output because their JSON output was always a giant array in the past which meant bumping into brimdata/super#3865. Now that they can output NDJSON (which they call json-log since it's more compatible with log-centric tools like Splunk and Logstash) we can take advantage of that.
Co-authored-by: Noah Treuhaft <[email protected]>
Co-authored-by: Noah Treuhaft <[email protected]>
2 participants
Now that the build-zeek repo brings us up to date with Zeek, I wanted to give this article in the Brimcap wiki a test run to see if anything needed to be changed. I ended up fixing and improving a handful of things. I'll put in-line comments with explanations for several changes.