State Rebuilds: Kafka Streams vs. Apache Flink

1. Overview

Stateful stream processors accumulate state from event streams over time. That state drives every aggregation, join, and windowed computation an application performs. When we change business logic, fix a bug, evolve a schema, or correct historical data, the accumulated state may no longer be valid. At that point we need a rebuild, replaying input events until state and outputs reflect reality again.

Rebuilds are expensive, they generate heavy replay traffic, push sustained backpressure through the pipeline, spike infrastructure load, and stretch recovery windows.

In this article, we’ll compare how Kafka Streams and Apache Flink trigger, execute, and where possible avoid rebuilds in production.

2. Rebuilds in Kafka Streams

In Kafka Streams, application state lives in local state stores backed by changelog topics on the broker. A rebuild means resetting consumer offsets and re-materializing those stores from scratch.

Because state and offsets are tightly coupled, any mismatch between the two forces a full replay.

2.1. Rebuild Triggers

We’ll typically need a rebuild when:

• Logic changes — we alter how records are aggregated, joined, or windowed
• Bug fixes — previously materialized results are no longer correct
• Topology changes — we add or remove operators, or shift repartition boundaries
• Schema evolution — record semantics or serialization format changes
• Data corrections — continuing from current offsets would preserve wrong outputs

2.2. Backpressure and Lag

Once a rebuild starts, the application replays historical data far faster than it would process a live stream. That burst of throughput puts pressure on several parts of the system at once:

• CPU, memory, and network utilization spike as records are replayed at full speed
• Stateful operators (aggregations, joins, windowed computations) become bottlenecks due to intensive state store writes and RocksDB compactions
• Downstream consumers and output topics may throttle, increasing end-to-end lag
• Internal repartition and changelog topics add broker and network load on top of the replay itself

3. Rebuilds in Apache Flink

Flink manages state centrally through periodic checkpoints and on-demand savepoints. The same triggers apply here: logic changes, bug fixes, schema evolution, and upstream data corrections can all invalidate current state.

However, Flink also requires a rebuild in situations Kafka Streams doesn’t encounter in the same way: state corruption, incompatible state serializers, or lost source offsets that prevent a clean restore from a checkpoint or savepoint.

3.1. Point-in-Time Rebuilds

One advantage Flink offers is the ability to restore a job to a specific point in time using a savepoint. Instead of replaying from the beginning of history, we restart the job from a savepoint that captures the full operator and keyed state at that moment. Events after the savepoint are then re-evaluated under the new logic.

This is especially useful when we need to:

• Roll out an application upgrade safely
• Apply a hotfix without reprocessing the entire history
• Backfill only a recent window of data after a correction

3.2. Backpressure During Rebuilds

Flink rebuilds share the same core problem as Kafka Streams: replaying historical data removes the natural rate limit of live ingestion. Throughput spikes, stateful operators and sinks become bottlenecks, and backpressure propagates upstream.

Where Flink differs is checkpointing. Backpressure increases barrier alignment time and overall checkpoint duration, raising the risk of timeouts or outright failures. In practice, keeping a rebuild stable means throttling source consumption, extending checkpoint intervals, and monitoring backpressure metrics closely.

We can also speed things up by temporarily scaling out. Increasing job parallelism spreads the replay workload across more task slots and shortens the rebuild window. Once the job catches up, we scale back down.

3.3. Avoiding Rebuilds With the State Processor API

In some cases, we can skip the rebuild altogether. The Flink State Processor API lets us read and modify state directly inside a savepoint. Instead of replaying events, we fix or migrate the state offline and restart the job from the patched savepoint.

This works well for:

• Removing or correcting corrupted state for specific keys
• Migrating state schemas when upgrading a job

The trade-off is that we’re editing state outside the normal processing path, so thorough validation is essential. Still, when source replay is expensive or the source data is no longer available, this approach can save significant time and infrastructure cost.

4. Conclusion

In this article, we looked at how Kafka Streams and Apache Flink handle state rebuilds. We covered what triggers them, the backpressure and lag they introduce, and the strategies each framework offers to manage or avoid them.

Kafka Streams ties state to local stores and changelog topics, making rebuilds a full-replay operation. Flink’s checkpoint and savepoint model opens the door to point-in-time rebuilds and offline state patching through the State Processor API, giving us more flexibility when a rebuild becomes necessary.