We can have more partitions (IO jobs) (from one or multiple parallelize() calls) than number of cores/IO workers available.

The LogScheduler correctly spreads the messages between partitions, but we can only run a limited amount of IO jobs at the same time.

The work() method is implemented in a way that during its run, new batches can be added to its workload. Also, it does not return if there is even one batch available for processing. If there is a big load of messages, IO jobs run indefinitely, or at least until new batches are no longer added to its queue.

If we have some IO jobs scheduled, but not ran, new batches will still be added to the non-running jobs, but we only process batches from the running jobs.

This happens until log-iw-size() number of messages are accumulated in the non-running jobs, which will cause the running jobs to not receive any more logs, thus eventually return, causing the previously non-running jobs to run finally. So the only thing that saves us from complete starvation is log-iw-size() and backpressure, which is not the intended behavior.

We need to put a hard limit to the number of logs processed by one job run, so ivykis can work its magic and cycle between the scheduled IO jobs. We now count the number of logs processed in one work() run, and if it reaches a configurable limit, we finish the batch and return from work(). The new log-fetch-limit() option of parallelize() can change this limit from its default 1000 value.

I have also added some level 4 metrics that can help debug parallelize() errors.

syslogng_parallelized_assigned_events_total{id="/home/alltilla/repos/axosyslog/build/install/etc/syslog-ng.conf:6:3",partition_index="0"} 614700
syslogng_parallelized_assigned_events_total{id="/home/alltilla/repos/axosyslog/build/install/etc/syslog-ng.conf:6:3",partition_index="1"} 614661
syslogng_parallelized_assigned_events_total{id="/home/alltilla/repos/axosyslog/build/install/etc/syslog-ng.conf:6:3",partition_index="2"} 614600
syslogng_parallelized_assigned_events_total{id="/home/alltilla/repos/axosyslog/build/install/etc/syslog-ng.conf:6:3",partition_index="3"} 614600
syslogng_parallelized_processed_events_total{id="/home/alltilla/repos/axosyslog/build/install/etc/syslog-ng.conf:6:3",partition_index="0"} 614513
syslogng_parallelized_processed_events_total{id="/home/alltilla/repos/axosyslog/build/install/etc/syslog-ng.conf:6:3",partition_index="1"} 614500
syslogng_parallelized_processed_events_total{id="/home/alltilla/repos/axosyslog/build/install/etc/syslog-ng.conf:6:3",partition_index="2"} 614500
syslogng_parallelized_processed_events_total{id="/home/alltilla/repos/axosyslog/build/install/etc/syslog-ng.conf:6:3",partition_index="3"} 614500