Motivation

Multiple services require data to be batched before further processing. Where batches can be collected based on:

• The number of records observed/collected (Batch count)
• A specified time-interval (Batch window)
• The number of underlying bytes collected (Batch size)
• Some other artibrary condition (i.e when a certain attribute is encountered then create a new batch)

To solve this problem, the new batcher utility class aims to standardize how we collect and return data needing to be batched. This has the following benefits:

• Standardizes a batching approach that can be re-used across multiple components
• Extendability allows us future versions to define custom batch policies that can be triggered on artibrary data.

Changes

• Adds a new Batcher utility that allows us to define a batch policy and collect and batch records based:
  • The number of records collected (batch count) and;
  • The duration for which a batch is being collected for (batch window)

Testing

• Added tests/unit/utils/test_batch_policy.py