Data & AI

Kafka Tiered Storage Architecture

Data & AI Pipelines diagram · made in Skemio

Kafka Tiered Storage architecture diagram illustrating local NVMe broker storage for hot data and asynchronous offloading to remote S3 object storage.
Kafka Tiered Storage architecture diagram illustrating local NVMe broker storage for hot data and asynchronous offloading to remote S3 object storage.
Open it in SkemioMake your own

What it shows

  • Active log segments remain on local broker NVMe drives for low latency.
  • Sealed log segments upload asynchronously to object storage like S3.
  • Brokers fetch historical segments transparently for lagging consumers.

The takeaway

Tiered storage decouples compute from storage, turning Kafka into a long-term replay log.

More data & ai pipelines diagrams