Kafka Streams Part 2: KStreams, KTables, Joins and Windows

In Part 1 we built a word counter and quietly used two types without really explaining them: KStream and KTable. Time to fix that, because this is the most important concept in all of Kafka Streams. Get these two right, and everything else — joins, windows, state — falls into place naturally.

Here's the one-line version:

A KStream is a diary. A KTable is a whiteboard.

Let's make that concrete.

---

KStream: The Diary

A diary records events as they happen. You never go back and erase an old page — you just write a new line. Every entry matters on its own.

A KStream is exactly that: a sequence of records where every record is a fact that happened.

Two clicks on /shoes from user-1 are not a mistake. They are two separate events, and both belong in the stream. Nothing is overwritten, ever. Use a KStream when the event itself is the information: clicks, payments, sensor readings, log lines.

KTable: The Whiteboard

A whiteboard holds current status. When the status changes, you erase the old value and write the new one. Nobody cares what the whiteboard said yesterday.

A KTable is a table where each record updates the row for its key: only the latest value per key survives.

Reading the same key twice leaves the KTable unchanged — the second write just replaces the first. Use a KTable for current state: a user's profile, today's product price, a device's last known status.

The Trick That Makes KTables Possible

How can a table "erase" values when Kafka topics only ever grow? Two Kafka features do the magic:

Log compaction. A normal topic keeps records for N days and deletes them by age. A compacted topic instead promises: for every key, keep at least the latest value, forever. Old records for the same key get cleaned up in the background. So the whole topic is effectively the table.

Tombstones. A record with a null value means "delete this key". Compaction keeps tombstones around long enough for everyone to see them, then drops the key entirely.

So a KTable is really just a live view over a compacted topic. Beautiful.

Bonus: GlobalKTable

One variation worth knowing: a GlobalKTable gives every instance of your app a full copy of the table, instead of splitting it into partitions. It costs more memory, but it lets you join against data with any key (not just the record's key) — perfect for small reference data like country codes or product catalogs.

---

Stateless Operations: One Record In, Records Out

These transform records without any memory of the past. If you know Java streams, you already know most of them.

Say we have a stream of payment events, keyed by user:

filter — keep only what matches. Every record is judged alone.

mapValues — transform the value, leave the key alone. (Prefer this over map when you can — keeping the key means keeping the partitioning.)

flatMapValues — one value in, many values out (we used this in Part 1 to split sentences into words).

selectKey — assign a new key (careful: this breaks "same key → same partition", which matters below).

branch — split one stream into several by predicate, like sorting mail into boxes:

merge — the opposite: combine two streams of the same type into one.

peek — look at records without changing them. Perfect for logging and debugging:

---

Stateful Operations: Where Memory Comes In

Now the fun part. To count, sum, or find the max, you need to remember the past. Kafka Streams calls this state, and it powers the operations below.

But state has a precondition: records must be grouped by key — because remembering something "per key" only works when all records for a key land in the same place (golden rule from Part 1: same key → same partition).

groupByKey vs groupBy

count — How Many?

You met this in Part 1:

aggregate — The Swiss Army Knife

count is just a special case of aggregate. You give it a starting value and a function that folds each new record into the running result:

Walking through it with three payments of 10, 20, and 5 from user-1:

Every step emits the new current value — KTable behavior. That's why consumers of this topic can always just read the latest record per key.

reduce — Combine New with Old

reduce is aggregate simplified: no separate starting value, just "how do I merge the new record into what I have?"

---

Joins: SQL Power, Streaming Speed

Real apps combine data from multiple topics. Kafka Streams joins work like SQL joins — the same key on both sides is the matchmaker — but with streaming semantics. Three combinations matter.

Stream + Table (the classic: enrich events with state)

Click events stream by, but a click alone is thin. Join it with the user profile table to fatten it up:

How it behaves: each arriving click looks up users at that moment. If user-42's profile changes at 10:05, clicks before that see the old profile, clicks after see the new one. The stream is history; the table is "now" — the join combines them exactly the way you'd expect.

Table + Table (materialized SQL join)

Two tables join like two database tables, and the result updates whenever either side changes:

Change someone's plan, and the joined table re-emits userId → new description automatically.

Stream + Stream (needs a time limit!)

Two infinite streams have no defined "end", so a join between them must be windowed — otherwise Kafka Streams would have to remember one side forever:

Impression at 12:00:00, click at 12:00:07 → match, one joined record out. Click at 12:01:00 → no match, the impression's window is long closed.

---

Windows: Slicing an Endless Stream

"How many clicks per minute?" sounds simple, but a stream never ends — so per minute of what? We need to slice time into windows and compute per slice. Four kinds to know:

Tumbling Windows — Fixed, Non-Overlapping Chunks

The clock cuts time into equal back-to-back blocks: 12:00–12:01, 12:01–12:02, ... Each event falls into exactly one block.

Note the key type: Windowed<String> — it's the key plus its window (like user-42 @[12:01, 12:02)), because the same user now has a count in every window.

Hopping Windows — Fixed Size, Slides Forward

Same block size, but the window hops forward by a smaller step, so blocks overlap and each event lands in several windows:

Use it for smoothed metrics — "a moving 5-minute count, updated every minute":

Sliding Windows — Sized Around Each Event

Instead of fixed grid positions, the window snaps around each event, covering [eventTime − 5s, eventTime + 5s]. It's mostly used inside stream-stream joins (JoinWindows from our ad example works exactly this way) — "did a matching event happen within 5 seconds of this one?"

Session Windows — Closed by Inactivity

No fixed size at all. A session grows while events keep arriving, and closes after a gap of silence. Perfect for user activity bursts:

With a 30-minute gap setting: 3 clicks, a 40-minute pause, then 2 more clicks → one session of 3 and one session of 2 (if more clicks had arrived at minute 29, they'd have merged into the first session instead).

Emitting Final Results Only: suppress

One annoyance with windows: by default you get an update for every record (count is 1... 2... 3...) and the final value only when the window closes. For things like "alerts per minute, emitted once at the end", suppress the intermediate updates:

Now each window emits exactly one record: the final count, once the window plus its grace period has passed.

---

Where Does the State Actually Live?

We've been casually accumulating state all part — counts, sums, sessions. Where is it? Answer: right next to your code, with a backup in Kafka.

The design is a clever double-win:

• Fast: every lookup and update hits RocksDB on local disk — no network round trip to any database.
• Durable: every state change also goes to a compacted changelog topic in Kafka. If an instance dies, a new one rebuilds its store by replaying the changelog. Kafka itself is your state's backup system.

You never manage any of this. Name your stores well, and it just works.

---

What's Next?

You now hold the full mental model: streams as events, tables as state, joins to combine them, and windows to slice time. Part 3 turns that knowledge into something you'd bet production on:

• Custom Serdes for your own JSON classes (and the trap that crashes apps at 3 AM)
• Event time vs processing time, and why it decides which window a record belongs to
• Exactly-once processing with one config line
• Error handling: poison pills and dead-letter queues
• Scaling with threads, instances, and standby replicas
• Testing topologies without a Kafka broker

Continue to Part 3: Making It Production-Ready →

---

Key Takeaways from Part 2:

• KStream = diary (every record is a fact), KTable = whiteboard (latest value per key)
• KTables ride on log compaction + tombstones — the topic is the table
• groupBy with a new key creates a hidden repartition topic — use groupByKey when keys already match
• aggregate is the general stateful tool; count and reduce are its convenient shortcuts
• Always name state stores with Materialized.as("...") — auto-generated names break across code changes
• Stream-Table joins enrich events; Table-Table joins mirror SQL; Stream-Stream joins must be windowed
• Window types: tumbling (fixed grid), hopping (overlapping slides), sliding (around events), session (closed by inactivity)
• State lives in local RocksDB, backed up to changelog topics in Kafka — fast and durable