System Design

Centralized Log Aggregation with Kafka & OpenSearch

System Design diagram · made in Skemio

Architecture pipeline showing Kubernetes pods outputting logs to Fluent Bit, buffered by Apache Kafka, processed by Logstash, and indexed in OpenSearch.
Architecture pipeline showing Kubernetes pods outputting logs to Fluent Bit, buffered by Apache Kafka, processed by Logstash, and indexed in OpenSearch.
Open it in SkemioMake your own

What it shows

  • Backpressure buffering: Kafka absorbs burst write spikes without dropped logs.
  • Lightweight collection: DaemonSets read container stdout using zero app memory.
  • Batched ingestion: Consumer workers transform and bulk-index into OpenSearch.

The takeaway

Placing an event buffer between log shippers and search clusters decouples peak throughput from index limits.

More system design diagrams