Introduction to Stream Processing & Flink

We start from scratch — understanding what stream processing is, why it matters, and writing your first Flink job before lunch. Every concept is immediately practiced in code.

Focus:

• What makes streaming different from batch? Event streams, bounded vs. unbounded data, and writing your first Flink SQL query against a live Kafka topic

• How do we handle time and windows? Fixed, sliding, and session windows — building windowed aggregations that compute real-time metrics

• What happens when data arrives late? Watermarks, triggers, and late data handling — configuring watermarks and observing how they affect your results

We connect Flink to real systems and explore the full range of APIs. By the end of the day, you’ll have a complete end-to-end streaming pipeline running.

Focus:

• How do we connect Flink to Kafka and databases? Kafka source/sink, Schema Registry, CDC with Debezium — building a complete Kafka → Flink → sink pipeline

• SQL, DataStream, or PyFlink for AI pipelines? Choosing the right API for your use case — SQL for fast development, DataStream for custom logic, and PyFlink for integrating with Python ML libraries and AI pipelines

• How do we deploy and monitor Flink in production? Kubernetes deployment, metrics, Grafana dashboards, and operational patterns

We go deeper into Flink’s internals — how it manages state, handles failures, and scales. You’ll build stateful applications and learn to recover from failures gracefully.

Focus:

• How does Flink work under the hood? Cluster architecture, JobManager, TaskManagers, deployment modes, and resource management

• How do we build stateful applications? Keyed state, state backends, exactly-once guarantees, and tracking running totals across partitions

• What happens when things fail? Checkpointing, savepoints, recovery mechanisms, and elastic scaling