Kafka Streams Part 1: Stream Processing Made Simple

Imagine a bakery. Every morning it collects all the orders from yesterday, bakes everything at once, and delivers the whole batch at night. That works — but what if a customer wants a croissant right now?

That is the difference between batch processing and stream processing.

Most tutorials about Kafka Streams drown you in jargon: topologies, serdes, rebalancing, exactly-once semantics... This series takes a different path. We go step by step, in plain words, with code you can actually run. By the end, you will be able to build and run real stream processing applications with confidence.

In this first part, we cover the fundamentals and build our first app together.

---

What Is Kafka Streams, Really?

Kafka Streams (also called Kafka Streams DSL) is a Java library for processing data that flows through Apache Kafka.

That sentence has three important words. Let's unpack them:

1. Java library — It is just a dependency you add to your project. Like Jackson or Guava. You write normal Java code.
2. Processing data — It transforms, filters, counts, joins, and enriches records as they arrive.
3. Through Kafka — The input and output of your processing are Kafka topics.

Here is the part that surprises most people:

Why Not Just Use Plain Kafka Consumers?

Fair question. You can already read from a topic with a plain consumer. So why a library?

A plain consumer hands you raw bytes, one record at a time. Everything else is your problem:

• Want to count events per user? Build your own state store, and somehow recover it after a crash.
• Want to join two topics? Write all that plumbing yourself.
• Want to scale from 1 machine to 6? Good luck splitting partitions and rebalancing safely.
• Want to process every record exactly once, even after failures? That's a research paper worth of work.

Kafka Streams gives you all of that out of the box:

In short: plain consumers give you the raw ingredients. Kafka Streams gives you the whole kitchen.

---

Kafka in 5 Minutes (The Parts That Matter)

You don't need to be a Kafka expert for this series. But four ideas come up constantly, so let's learn them now.

1. Events

An event is a fact that happened: "Order 123 was placed", "User 42 clicked the buy button". In Kafka, an event is called a record, and it has three parts we care about:

• Key (optional): what the event is about, e.g. user-42
• Value: the event data itself, e.g. {"url": "/checkout", "amount": 30}
• Timestamp: when it happened

2. Topics

A topic is like a named conveyor belt for one kind of event: orders, clicks, payments. Producers put records on the belt. Consumers read them off.

3. Partitions

Here is where it gets interesting. A topic is split into partitions. Each partition is an ordered, append-only log:

Two golden rules of partitions:

• Records with the same key always land in the same partition. All events for user-42 go to exactly one partition, in order. This is how Kafka keeps per-key order.
• A partition is assigned to exactly one consumer in a group. Two consumers never read the same partition. This is how Kafka parallelizes work without you writing a single line of coordination code.

Keep these two rules in your pocket. Almost everything in Kafka Streams — grouping, joining, scaling — is built on them.

4. Consumer Groups

Consumers with the same group ID share the work: partitions are divided among them. Add a consumer, and Kafka hands it some partitions. Remove one, and Kafka redistributes. This rebalancing happens automatically.

Your Kafka Streams app joins a consumer group under the hood — you just never see it, because the library manages it. The group ID comes from a config called application.id, which brings us to...

---

Your First App: Word Count

Time to write real code. We'll build the "hello world" of stream processing: count how many times each word appears, live, as sentences arrive.

The plan: read sentences from a sentences topic, split them into words, keep a running count per word, and write each updated count to a word-counts topic.

Here is the whole journey of one sentence:

That picture has a name: a topology. It is a pipeline of processing steps. You describe it once, and Kafka Streams runs it forever, on any number of machines.

Step 1: The Project

Create a plain Java project (Java 17 or newer) with this dependency:

That's it. One dependency.

Step 2: The Code

Not bad, right? That is a complete, fault-tolerant, scalable, real-time application. Let's walk through the interesting parts.

The Config, Explained

The Pipeline, Line by Line

builder.stream("sentences") — This is a source: it reads from a topic and gives you a KStream, which is a stream of key-value records. Think of it as an endless list of (null, "Kafka is fast") entries — no key, just sentences as values.

flatMapValues(...) — Turns each value into many values. One sentence in, several words out. "Kafka is fast" becomes kafka, is, fast. This is your everyday flatMap from Java streams — it just runs on an infinite stream instead of a list.

groupBy((sentenceKey, word) -> word) — Re-keys every record so the word is the new key. Why? Because of our golden rule: same key → same partition. Grouping by word guarantees all kafka records end up together, so one counter can count them all.

count() — Keeps a running count per key. This needs memory of the past (state), and Kafka Streams handles it silently: it stores the counts in a fast local database and backs it up to a hidden Kafka topic so nothing is lost on a crash. This returns a KTable — a constantly updated view of "current count per word". We dig into KTables in Part 2.

counts.toStream().to("word-counts", ...) — KTables update in place, but topics only grow — so we convert the table back to a stream of updates and write each one out. Every time a word's count changes, a fresh record like (kafka, 4) lands in the output topic.

Step 3: Run It

Start a local Kafka with Docker (single command, no config files):

Create the two topics:

Run the app from your IDE (or mvn compile exec:java). Then open a console producer and type some sentences:

Finally, watch the output:

You'll see something like:

Notice kafka appears twice — first with 1, then with 2. That's the update behavior of count(): every new occurrence emits the new total. The output topic contains the history of updates, and the latest record per key always holds the current count.

Type another sentence in the producer and watch the counts change instantly. Congratulations — that's stream processing. 🎉

---

How Scaling Works (A Sneak Peek)

Here's a beautiful property of what we just built. Start one instance of the app, and it handles all 3 partitions. Start three instances with the same application.id, and Kafka assigns one partition to each — triple the throughput, zero code changes:

Each partition-sized piece of work is called a stream task. Tasks are the unit of parallelism: 3 partitions means at most 3 tasks, whether they run as 3 threads on one machine or 3 machines entirely. Kill an instance, and its tasks are handed to survivors automatically.

We go deep on scaling (threads vs instances vs standby replicas) in Part 3.

---

What's Next?

You now understand what Kafka Streams is, how Kafka topics work under it, and you've built and run a real streaming app.

But we casually used two mysterious types — KStream and KTable — and waved our hands at "state". In Part 2, we make them crystal clear, and they unlock the truly powerful stuff:

• KStream vs KTable: the diary vs the whiteboard (the single most important concept in Kafka Streams)
• Stateful operations: count, aggregate, reduce
• Joining streams together — like SQL, but live
• Windowing: how to count "per minute" on a stream that never ends

Continue to Part 2: KStreams, KTables, Joins and Windows →

---

Key Takeaways from Part 1:

• Stream processing handles events one at a time, immediately — batch's opposite
• Kafka Streams is a library inside your app, not a separate cluster to run
• Same key → same partition → per-key order; one partition → one consumer → parallelism without coordination code
• You describe a topology (a pipeline of steps); the library runs it forever, anywhere
• application.id names your app, its consumer group, and its internal topics — don't change it casually
• count() outputs Long — pass Produced.with(..., Serdes.Long()) or you'll meet a ClassCastException
• Partitions of the input topic cap your parallelism: more partitions, more possible tasks