Wiki article and example YAML/scripts for custom Zeek/Suricata/NetFlow#72
Wiki article and example YAML/scripts for custom Zeek/Suricata/NetFlow#72
Conversation
| 1. The `type alert` defines the names, [data types](https://github.com/brimdata/zed/blob/main/docs/formats/zson.md#33-primitive-values), | ||
| and hierarchical locations of expected fields in the input records. Here we've | ||
| defined a single "wide" shape for _all_ alerts we've known Suricata to | ||
| generate, which is convenient because it allows Brim to easily display them in | ||
| columnar format. |
There was a problem hiding this comment.
I know it's been debated at times whether it's appropriate to use these wide schemas, particularly when it might result in records with excessive nulls because of alerts that don't use several fields. However, I've continued to err on the side of "wide"-ness because we've had community users report it as a bug when data they're accustomed to seeing in columnar layouts in Brim suddenly lose their column headers because additional schemas have appeared. We did discuss once as a group some UX we could add in Brim to call a user's attention to the multiple schemas to make it easy for the user to see the data under each schema separately (e.g. perhaps under a separate "tab" per schema), and when that exists, I'd be more open to letting through more diverse schemas.
| Our pool is now ready to be queried in Brim. | ||
|
|
||
|  |
There was a problem hiding this comment.
Note the histogram is empty, which is unsurprising since I think we're still wired to only populate this if the field named _path is present. We've discussed in the past that it might be helpful to be able to specify a primary "segmentation axis" for the data in a pool (e.g. _path for Zeek) that could be used to make stacked bars when present, and in its absence, just give non-stacked "count" bars? (see brimdata/zui#1541)
|
The markdown link checker is an expected failure because the links in the docs are relative to where this will ultimately be published in the wiki. |
jameskerr
left a comment
jameskerr
left a comment
There was a problem hiding this comment.
As I was reading about the output files and globs, I wasn't sure what directory those files would end up in? The current directory?
Otherwise very clear and high quality. The ergonomics are not quite there in the tooling yet. There are a lot of workarounds.
Co-authored-by: James Kerr <[email protected]>
Co-authored-by: Noah Treuhaft <[email protected]>
Co-authored-by: Noah Treuhaft <[email protected]>
Co-authored-by: Noah Treuhaft <[email protected]>
Co-authored-by: Noah Treuhaft <[email protected]>
|
Thanks for the feedback, @jameskerr. Some thoughts:
I'm glad you spoke up about this. I was on the fence about whether to get into the topic in this article, but since you spotted its absence, I went ahead and added some text that mentions how it goes to a temporary directory by default, and then a new "Debug" section that describes how to use the
That's exactly what I was trying to communicate, so thanks for saying that! 😄 Indeed, I'm glad to see that these things are feasible with a little elbow grease. But before I feel confident pointing users at it and implying that we're ready for them to go as far as they want with it, it'll indeed be good to see more bug fixes and UX improvements start to arrive. As mentioned at our team sync meeting, I'm keen to hold this PR open in a draft state and use it like a checklist as we march toward bundling the next release. |
This reverts commit 41ac5bb.
3 participants
The motivation for this article is described in #64. Having written the initial version and tested it extensively, I'm inclined to wait on fixes for some of the bugs that were found along the way, as that way the final version will have fewer warts. Beyond the bugs, the end-to-end shown here also illustrates the need for some of the other enhancements waiting in our backlog. I'm starting this out as a draft PR and will add comments at the relevant spots.
Closes #64
Closes #8