All notable changes to this project will be documented in this file. For help with updating to new Bytewax versions, please see the migration guide.
Add any extra change notes here and we'll put them in the release notes on GitHub when we make a new release.
-
The
db_dirattribute of {py:obj}~bytewax.recovery.RecoveryConfigis astragain, as in v0.21.1. The PyO3 0.25 upgrade had made it apathlib.Path, which broke code such asconfig.db_dir + "/x". The constructor still accepts astror apathlib.Path. -
Adds
bytewax.__version__, exposing the installed package version from Python code, e.g. when sanity-checking an environment or reporting a bug. It is read from the package metadata, so importing the package stays cheap and does not load the compiled extension.bytewaxis now a regular package instead of a namespace package. Add-ons that install modules into it, such asbytewax.duckdb, are still found, also from anothersys.pathentry than bytewax; only a path added tosys.pathafterimport bytewaxis no longer searched for them. -
Security: the keys of eight eMudhra intermediate CAs were compromised in October 2026. The release Docker images now upgrade Debian's
ca-certificatespackage (to20250419~deb12u2or later), which no longer trusts them; images built on base images that are no longer updated, such as the one for Python 3.9, still trusted them. The lock files for tests and builds pincertifi2026.10.23, which drops theemSign ECC Root CA - G3root they chain to. Bytewax bundles no CA certificates itself: in your own environments, updatecertifito 2026.10.19 or later and your system'sca-certificates. -
{py:obj}
~bytewax.connectors.kafka.KafkaSourceno longer blocks forever at startup when the brokers are unreachable. It now gives up fetching topic metadata after 30 seconds and fails with aRuntimeErrornaming the brokers and topic. See "Startup and Unreachable Brokers" in the Kafka guide. -
Behavior change {py:obj}
~bytewax.connectors.kafka.KafkaSinknow fails the dataflow with aRuntimeErrorwhen a message isn't delivered (e.g. unreachable brokers pastmessage.timeout.ms, or a missing topic the brokers don't auto-create). This is intended: previously such messages were dropped silently, even with recovery enabled; now resuming re-sends them (at-least-once), and messages still queued when the dataflow fails are dropped rather than waited on. When the producer's local queue is full, the sink waits for it to drain instead of crashing withBufferError(reported in #522 by @shubhanshu02); a single message larger than the whole queue fails with a clear error. A non-callableon_deliveryinadd_configis rejected with aTypeErrorwhen the sink is built; it used to raise aKafkaExceptionthere. The docs now state that this at-least-once guarantee requires running with recovery and brokers acknowledging writes (notacks=0). -
Fixes
bytewax.connectors.kafka.serdefailing to import withconfluent-kafka>=2.7unless theconfluent-kafka[schemaregistry]extra was installed.PlainAvroSerializerandPlainAvroDeserializerno longer need Schema Registry dependencies; they still accept aconfluent_kafka.schema_registry.Schema. -
Fixes
python -m bytewax.runon Windows with an absolute path likeC:\proj\flow.py:flow: the drive letter's colon was taken as themodule:dataflowseparator, so the module was looked up asC. -
Fixes a bug where {py:obj}
~bytewax.testing.TestingSource.PAUSEand already-triggered {py:obj}~bytewax.testing.TestingSource.ABORTitems were not counted in the source's resume state, causing resumed executions to rewind and re-emit items that had already been processed. -
Fixes {py:obj}
~bytewax.inputs.batch_asyncnever closing the event loop it creates, nor the source async iterator, when it finishes or its partition is closed. Each use leaked file descriptors until garbage collection, and the source's cleanup (e.g. closing a websocket) didn't run. A loop passed in withloop=is still left open. Reported in #536 by @Chittaranjans. -
Fixes {py:obj}
~bytewax.tracing.setup_tracinghanging forever instead of raising aRuntimeErrorwhen tracing can't be set up: when it is called a second time in the same process, or when the tracing config is invalid (e.g. a malformed OTLP endpoint URL). Tracing can only be set up once per process. -
This is the last release that supports Python 3.8 and 3.9, which are end-of-life upstream. The next release will require Python 3.10 or newer.
-
Adds support for Python 3.13 and 3.14, with wheels for Linux, macOS and Windows (#564). The Rust core now uses PyO3 0.25 instead of 0.21. There are no wheels for the free-threaded builds (
3.13t,3.14t). -
Fixes {py:obj}
~bytewax.connectors.kafka.KafkaSourcehanging at startup when anoauth_cbis set inadd_config(e.g. with"sasl.mechanism": "OAUTHBEARER"): the callback was never called while fetching topic metadata (#541, #547). -
Linux x86_64 wheels are now
manylinux2014(manylinux_2_17) wheels and need glibc 2.17 or newer, like the aarch64 and armv7 wheels already did. The v0.21.1 x86_64 wheels needed glibc 2.31, so they did not install on e.g. RHEL 8 or Debian 10. -
The
bytewax/bytewaxDocker images are now based on Debian 12 (bookworm) instead of Debian 11 (bullseye), and are also published for Python 3.13 and 3.14. -
Building Bytewax from source now uses Rust 1.86.0 (pinned in
rust-toolchain.toml) instead of 1.74.1. -
Fixes the type hints of the
latestream of {py:obj}~bytewax.operators.windowing.join_window: late values (sub-keyed by window ID) are(side, value)2-tuples, wheresideis the index of the stream the value came from, not bare values. The docstring now says so. Code annotated with the old type needs updating; it did not match what the stream carries at runtime. -
Fixes the type hints of {py:obj}
~bytewax.operators.windowing.collect_windownot accepting theorderedargument, so type checkers rejectedcollect_window(..., ordered=False)even though it works at runtime. -
{py:obj}
~bytewax.operators.windowing.collect_windownow raises its intendedTypeErrorexplaining that onlylist,set, anddictare supported whenintois another type. A formatting bug replaced that message with "unsupported format string passed to type.format". Theintodocs now list the supported types, theupdocs no longer call it a "stream of items to count", and the:returns:docs no longer say the values are always in timestamp order, which isn't true withordered=False.
-
join_windowoperator now supports using stream-order via theorderedparameter. -
Windowing operators now correctly respect
now_getterandto_system_utcinEventClock. -
Fixes an issue where the runtime would not properly report the correct exception being raised by dataflow code.
-
Fixes a bug where window closing will be delayed if using event time and all values for a key fall into a single window and event timestamps are within
wait_for_system_durationof each other. -
Fixes a bug where using
EventClockon macOS systems would randomly assert.
-
{py:obj}
~bytewax.inputs.SimplePollingSourcenow allows you to retain state to support at-least-once delivery. -
Fixes a bug when using {py:obj}
~bytewax.operators.windowing.SlidingWindowerwhere values would be assigned to an extra window if their timestamps were near the end of the correct window. -
Upstream type hints on {py:obj}
bytewax.connectors.kafka.operators.serialize_key, {py:obj}~bytewax.connectors.kafka.operators.serialize_value, and {py:obj}~bytewax.connectors.kafka.operators.serializehave been made more broad to support all Kafka serializers, like {py:obj}confluent_kafka.serialization.StringSerializer. -
Breaking change - Fixes a bug which caused two of the same types of windowing operators in a dataflow to spuriously result in a
ValueError. This fix invalidates any recovery data for all windowing operators; it is recommended to delete and re-create the recovery store if you are using windowing operators. -
Fixes a performance issue where {py:obj}
bytewax.operators.StatefulBatchLogic.notify_at(and thus many of the other stateful operators'notify_atderived from it) was being called superfluously.
- Fixes a bug when using
{py:obj}
~bytewax.operators.windowing.EventClockwhere in-order but "slow" data results in watermark assertion errors.
-
Adds a dataflow structure visualizer. Run
python -m bytewax.visualize. -
Breaking change The internal format of recovery databases has been changed from using
JsonPickleto Python's built-in {py:obj}pickle. Recovery stores that used the old format will not be usable after upgrading. -
Breaking change The
unaryoperator andUnaryLogichave been renamed tostatefulandStatefulLogicrespectively. -
Adds a
stateful_batchoperator to allow for lower-level batch control while managing state. -
StatefulLogic.on_notify,StatefulLogic.on_eof, andStatefulLogic.notify_atare now optional overrides. The defaults retain the state and emit nothing. -
Breaking change Windowing operators have been moved from
bytewax.operators.windowintobytewax.operators.windowing. -
Breaking change
ClockConfigs have hadConfigdropped from their name and are justClocks. E.g. If you previouslyfrom bytewax.operators.window import SystemClockConfignowfrom bytewax.operators.windowing import SystemClock. -
Breaking change
WindowConfigs have been renamed toWindowers. E.g. If you previouslyfrom bytewax.operators.window import SessionWindownowfrom bytewax.operators.windowing import SessionWindower. -
Breaking change All windowing operators now return a set of streams {py:obj}
~bytewax.operators.windowing.WindowOut. {py:obj}~bytewax.operators.windowing.WindowMetadatanow is branched into its own stream and is no longer part of the single downstream. All window operator emitted items are labeled with the unique window ID they came from to facilitate joining the data later. -
Breaking change {py:obj}
~bytewax.operators.windowing.fold_windownow requires amergeargument. This handles whenever the session windower determines that two windows must be merged because a new item bridged a gap. -
Breaking change The
join_namedandjoin_window_namedoperators have been removed because they did not support returning proper type information. Use {py:obj}~bytewax.operators.joinor {py:obj}~bytewax.operators.windowing.join_windowinstead, which have been enhanced to properly type their downstream values. -
Breaking change {py:obj}
~bytewax.operators.joinand {py:obj}~bytewax.operators.windowing.join_windowhave had theirproductargument replaced withinsert_mode. You now can specify more nuanced kinds of join modes. -
Python interfaces are now provided for custom clocks and windowers. Subclass {py:obj}
~bytewax.operators.windowing.Clock(and a corresponding {py:obj}~bytewax.operators.windowing.ClockLogic) or {py:obj}~bytewax.operators.windowing.Windower(and a corresponding {py:obj}~bytewax.operators.windowing.WindowerLogic) to define your own senses of time and window definitions. -
Adds a {py:obj}
~bytewax.operators.windowing.windowoperator to allow you to write more flexible custom windowing operators. -
Session windows now work correctly with out-of-order data and joins.
-
{py:obj}
~bytewax.operators.windowing.WindowMetadatanow contains a {py:obj}~bytewax.operators.windowing.WindowMetadata.merged_idsfield with any window IDs that were merged into this window. -
All windowing operators now process items in timestamp order. The most visible change that this results in is that the {py:obj}
~bytewax.operators.windowing.collect_windowoperator now emits collections with values in timestamp order. -
Adds a {py:obj}
~bytewax.operators.filter_map_valueoperator. -
Adds a {py:obj}
~bytewax.operators.enrich_cachedoperator for easier joining with an external data source. -
Adds a {py:obj}
~bytewax.operators.key_rmconvenience operator to remove keys from a {py:obj}~bytewax.operators.KeyedStream.
- Fixes a bug where using a system clock on certain architectures causes items to be dropped from windows.
-
Multiple operators have been reworked to avoid taking and releasing Python's global interpreter lock while iterating over multiple items. Windowing operators, stateful operators and operators like
branchwill see significant performance improvements.Thanks to @damiondoesthings for helping us track this down!
-
Breaking change
FixedPartitionedSource.build_part,DynamicSource.build,FixedPartitionedSink.build_partandDynamicSink.buildnow take an additionalstep_idargument. This argument can be used when labeling custom Python metrics. -
Custom Python metrics can now be collected using the
prometheus-clientlibrary. -
Breaking change The schema registry interface has been removed. You can still use schema registries, but you need to instantiate the (de)serializers on your own. This allows for more flexibility. See the
confluent_serdeandredpanda_serdeexamples for how to use the new interface. -
Fixes bug where items would be incorrectly marked as late in sliding and tumbling windows in cases where the timestamps are very far from the
align_toparameter of the windower. -
Adds
stateful_flat_mapoperator. -
Breaking change Removes
builderargument fromstateful_map. Instead, the initial state value is alwaysNoneand you can call your previous builder by hand in themapper. -
Breaking change Improves performance by removing the
now: datetimeargument fromFixedPartitionedSource.build_part,DynamicSource.build, andUnaryLogic.on_item. If you need the current time, use:
from datetime import datetime, timezone
now = datetime.now(timezone.utc)- Breaking change Improves performance by removing the
sched: datetimeargument fromStatefulSourcePartition.next_batch,StatelessSourcePartition.next_batch,UnaryLogic.on_notify. You should already have the scheduled next awake time in whatever instance variable you returned in{Stateful,Stateless}SourcePartition.next_awakeorUnaryLogic.notify_at.
-
Fixes a bug that prevented the deletion of old state in recovery stores.
-
Better error messages on invalid epoch and backup interval parameters.
-
Fixes bug where dataflow will hang if a source's
next_awakeis set far in the future.
-
Changes the default batch size for
KafkaSourcefrom 1 to 1000 to match the Kafka input operator. -
Fixes an issue with the
count_windowoperator: #364.
-
Support for schema registries, through
bytewax.connectors.kafka.registry.RedpandaSchemaRegistryandbytewax.connectors.kafka.registry.ConfluentSchemaRegistry. -
Custom Kafka operators in
bytewax.connectors.kafka.operators:input,output,deserialize_key,deserialize_value,deserialize,serialize_key,serialize_valueandserialize. -
Breaking change
KafkaSourcenow emits a specialKafkaSourceMessageto allow access to all data on consumed messages.KafkaSinknow consumesKafkaSinkMessageto allow setting additional fields on produced messages. -
Non-linear dataflows are now possible. Each operator method returns a handle to the
Streams it produces; add further steps via calling operator functions on those returned handles, not the rootDataflow. See the migration guide for more info. -
Auto-complete and type hinting on operators, inputs, outputs, streams, and logic functions now works.
-
A ton of new operators:
collect_final,count_final,count_window,flatten,inspect_debug,join,join_named,max_final,max_window,merge,min_final,min_window,key_on,key_assert,key_split,merge,unary. Documentation for all operators are inbytewax.operatorsnow. -
New operators can be added in Python, made by grouping existing operators. See
bytewax.dataflowmodule docstring for more info. -
Breaking change Operators are now stand-alone functions;
import bytewax.operators as opand use e.g.op.map("step_id", upstream, lambda x: x + 1). -
Breaking change All operators must take a
step_idargument now. -
Breaking change
foldandreduceoperators have been renamed tofold_finalandreduce_final. They now only emit on EOF and are only for use in batch contexts. -
Breaking change
batchoperator renamed tocollect, so as to not be confused with runtime batching. Behavior is unchanged. -
Breaking change
outputoperator does not forward downstream its items. Add operators on the upstream handle instead. -
next_batchon input partitions can now return anyIterable, not just aList. -
inspectoperator now has a default inspector that prints out items with the step ID. -
collect_windowoperator now can collect intosets anddicts. -
Adds a
get_fs_idargument to{Dir,File}Sourceto allow handling non-identical files per worker. -
Adds a
TestingSource.EOFandTestingSource.ABORTsentinel values you can use to test recovery. -
Breaking change Adds a
datetimeargument toFixedPartitionSource.build_part,DynamicSource.build_part,StatefulSourcePartition.next_batch, andStatelessSourcePartition.next_batch. You can now use this to update yournext_awaketime easily. -
Breaking change Window operators now emit
WindowMetadataobjects downstream. These objects can be used to introspect the open_time and close_time of windows. This changes the output type of windowing operators from:(key, values)to(key, (metadata, values)). -
Breaking change IO classes and connectors have been renamed to better reflect their semantics and match up with documentation.
-
Moves the ability to start multiple Python processes with the
-por--processesto thebytewax.testingmodule. -
Breaking change
SimplePollingSourcemoved frombytewax.connectors.periodictobytewax.inputssince it is an input helper. -
SimplePollingSource'salign_toargument now works.
-
Adds the
batchoperator to Dataflows. CallingDataflow.batchwill batch incoming items until either a batch size has been reached or a timeout has passed. -
Adds the
SimplePollingInputsource. Subclass this input source to periodically source new input for a dataflow. -
Re-adds GLIBC 2.27 builds to support older linux distributions.
-
Breaking change Recovery system re-worked. Kafka-based recovery removed. SQLite recovery file format changed; existing recovery DB files can not be used. See the module docstring for
bytewax.recoveryfor how to use the new recovery system. -
Dataflow execution supports rescaling over resumes. You can now change the number of workers and still get proper execution and recovery.
-
epoch-intervalhas been renamed tosnapshot-interval -
The
list-partsmethod ofPartitionedInputhas been changed to return aList[str]and should only reflect the available inputs that a given worker has access to. You no longer need to return the complete set of partitions for all workers. -
The
nextmethod ofStatefulSourceandStatelessSourcehas been changed tonext_batchand should return aListof elements, or the empty list if there are no elements to return.
-
Added new cli parameter
backup-interval, to configure the length of time to wait before "garbage collecting" older recovery snapshots. -
Added
next_awaketo input classes, which can be used to schedule when the next call tonext_batchshould occur. Usenext_awakeinstead oftime.sleep. -
Added
bytewax.inputs.batcher_asyncto bridge async Python libraries in Bytewax input sources. -
Added support for linux/aarch64 and linux/armv7 platforms.
KafkaRecoveryConfighas been removed as a recovery store.
- Add support for Windows builds - thanks @zzl221000!
- Adds a CSVInput subclass of FileInput
- Add a cooldown for activating workers to reduce CPU consumption.
- Add support for Python 3.11.
-
Breaking change Reworked the execution model.
run_mainandcluster_mainhave been moved tobytewax.testingas they are only supposed to be used when testing or prototyping. Production dataflows should be ran by calling thebytewax.runmodule withpython -m bytewax.run <dataflow-path>:<dataflow-name>. Seepython -m bytewax.run -hfor all the possible options. The functionality offered byspawn_clusterare now only offered by thebytewax.runscript, sospawn_clusterwas removed. -
Breaking change
{Sliding,Tumbling}Window.start_athas been renamed toalign_toand both now require that argument. It's not possible to recover windowing operators without it. -
Fixes bugs with windows not closing properly.
-
Fixes an issue with SQLite-based recovery. Previously you'd always get an "interleaved executions" panic whenever you resumed a cluster after the first time.
-
Add
SessionWindowfor windowing operators. -
Add
SlidingWindowfor windowing operators. -
Breaking change Rename
TumblingWindowConfigtoTumblingWindow -
Add
filter_mapoperator. -
Breaking change New partition-based input and output API. This removes
ManualInputConfigandManualOutputConfig. Seebytewax.inputsandbytewax.outputsfor more info. -
Breaking change
Dataflow.captureoperator is renamed toDataflow.output. -
Breaking change
KafkaInputConfigandKafkaOutputConfighave been moved tobytewax.connectors.kafka.KafkaInputandbytewax.connectors.kafka.KafkaOutput. -
Deprecation warning The
KafkaRecoverystore is being deprecated in favor ofSqliteRecoveryConfig, and will be removed in a future release.
- Breaking change Fixes issue with multi-worker recovery. If the cluster crashed before all workers had completed their first epoch, the cluster would resume from the incorrect position. This requires a change to the recovery store. You cannot resume from recovery data written with an older version.
-
Dataflow continuation now works. If you run a dataflow over a finite input, all state will be persisted via recovery so if you re-run the same dataflow pointing at the same input, but with more data appended at the end, it will correctly continue processing from the previous end-of-stream.
-
Fixes issue with multi-worker recovery. Previously resume data was being routed to the wrong worker so state would be missing.
-
Breaking change The above two changes require that the recovery format has been changed for all recovery stores. You cannot resume from recovery data written with an older version.
-
Adds an introspection web server to dataflow workers.
-
Adds
collect_windowoperator.
- Added Google Colab support.
- Added tracing instrumentation and configurations for tracing backends.
-
Fixes bug where window is never closed if recovery occurs after last item but before window close.
-
Recovery logging is reduced.
-
Breaking change Recovery format has been changed for all recovery stores. You cannot resume from recovery data written with an older version.
-
Adds a
DynamoDBandBigqueryoutput connector.
-
Performance improvements.
-
Support SASL and SSL for
bytewax.inputs.KafkaInputConfig.
-
KafkaInputConfig now accepts additional properties. See
bytewax.inputs.KafkaInputConfig. -
Support for a pre-built Kafka output component. See
bytewax.outputs.KafkaOutputConfig.
-
Added the
fold_windowoperator, works likereduce_windowbut allows the user to build the initial accumulator for each key in abuilderfunction. -
Output is no longer specified using an
output_builderfor the entire dataflow, but you supply an "output config" per capture. Seebytewax.outputsfor more info. -
Input is no longer specified on the execution entry point (like
run_main), it is instead using theDataflow.inputoperator. -
Epochs are no longer user-facing as part of the input system. Any custom Python-based input components you write just need to be iterators and emit items. Recovery snapshots and backups now happen periodically, defaulting to every 10 seconds.
-
Recovery format has been changed for all recovery stores. You cannot resume from recovery data written with an older version.
-
The
reduce_epochoperator has been replaced withreduce_window. It takes a "clock" and a "windower" to define the kind of aggregation you want to do. -
runandrun_clusterhave been removed and the remaining execution entry points moved intobytewax.execution. You can now get similar prototyping functionality withbytewax.execution.run_mainandbytewax.execution.spawn_clusterusingTesting{Input,Output}Configs. -
Dataflowhas been moved intobytewax.dataflow.Dataflow.
-
Input is no longer specified using an
input_builder, but now aninput_configwhich allows you to use pre-built input components. Seebytewax.inputsfor more info. -
Preliminary support for a pre-built Kafka input component. See
bytewax.inputs.KafkaInputConfig. -
Keys used in the
(key, value)2-tuples to route data for stateful operators (likestateful_mapandreduce_epoch) must now be strings. Because of thisbytewax.exhashis no longer necessary and has been removed. -
Recovery format has been changed for all recovery stores. You cannot resume from recovery data written with an older version.
-
Slight changes to
bytewax.recovery.RecoveryConfigconfig options due to recovery system changes. -
bytewax.run()andbytewax.run_cluster()no longer takerecovery_configas they don't support recovery.
-
Adds
bytewax.AdvanceToandbytewax.Emitto control when processing happens. -
Adds
bytewax.run_main()as a way to test input and output builders without starting a cluster. -
Adds a
bytewax.testingmodule with helpers for testing. -
bytewax.run_cluster()andbytewax.spawn_cluster()now take amp_ctxargument to allow you to change the multiprocessing behavior. E.g. from "fork" to "spawn". Defaults now to "spawn". -
Adds dataflow recovery capabilities. See
bytewax.recovery. -
Stateful operators
bytewax.Dataflow.reduce()andbytewax.Dataflow.stateful_map()now require astep_idargument to handle recovery. -
Execution entry points now take configuration arguments as kwargs.
-
Capture operator no longer takes arguments. Items that flow through those points in the dataflow graph will be processed by the output handlers setup by each execution entry point. Every dataflow requires at least one capture.
-
Executor.build_and_run()is replaced with four entry points for specific use cases:-
run()for exeuction in the current process. It returns all captured items to the calling process for you. Use this for prototyping in notebooks and basic tests. -
run_cluster()for execution on a temporary machine-local cluster that Bytewax coordinates for you. It returns all captured items to the calling process for you. Use this for notebook analysis where you need parallelism. -
spawn_cluster()for starting a machine-local cluster with more control over input and output. Use this for standalone scripts where you might need partitioned input and output. -
cluster_main()for starting a process that will participate in a cluster you are coordinating manually. Use this when starting a Kubernetes cluster.
-
-
Adds
bytewax.parsemodule to help with reading command line arguments and environment variables for the above entrypoints. -
Renames
bytewax.inptobytewax.inputs.