Here's a simple configuration excerpt:

destination d_kafka {
  kafka(properties(metadata.broker.list("localhost:9092"))
        topic("syslog")
        payload("$(format-json --scope all-nv-pairs --scope core)")
        partition(field("$PROGRAM"))
#       partition(random)
#       partition("static-key")
  );
};

A few questions, while I'm reviewing:

• Would it be possible to use the crc32() function from zlib (which we already use at least indirectly, via librdkafka), instead of shipping a crc32 implementation in modules/kafka/crc32.c?

• While I like the way properties are done, with the current implementation, both name and value must be a simple string. Would it make sense to use value-pairs() instead? That would make it much more flexible, and enable us to use a syntax used elsewhere too.

  This way, the config would look exactly the same (properties is a good name, that can be kept), but allow for using templates in the value part. The Riemann destination in the Incubator does something similar: it reuses the value-pairs infrastructure, but allows the same style of config you have with properties(...).

  The downside of that is performance, though. The value-pairs stuff isn't exactly the fastest thing, but $(format-json) uses that too, anyway, and if you have only a few properties, it doesn't make much of a difference.

• For partition(), I'd use a template, to simplify the syntax a bit. Then, if you'd want a static-key, you'd just write a static string there, without variables. If you want $PROGRAM, you'd write partition("$PROGRAM"), and for random... well, we'd likely need a template function for that, or $(uuid), or $UNIXTIME or $(sha1 $UNIXTIME) or something similar. In any case, I think it would be a bit more flexible to use a template there, and the code would become a tiny bit simpler too, as you'd only have to handle one case: your current field(...) case.

• I don't think you need to have neither BalaBit, nor Tihamer as copyright holders, in the few files where they are. Copying the skeleton does not, in my opinion, count as having to keep their copyrights.

As discussed on IRC, a better target for this module is the syslog-ng incubator, thanks for the review!