Centralized Log Aggregation with Fluent Bit & Kafka

DevOps & CI/CD diagram · made in Skemio

Architecture diagram illustrating Fluent Bit collecting logs on Kubernetes nodes, buffering them into Apache Kafka, and Logstash writing to OpenSearch.

Open it in Skemio Make your own

What it shows

Fluent Bit DaemonSets consume minimal CPU/RAM per node compared to Logstash.
Kafka acts as a durable backpressure buffer during unexpected log spikes.
Logstash consumers parse, enrich, and batch-index without dropping data.

The takeaway

Decouple log shipping from log indexing with a streaming queue to survive ingestion spikes.

#DevOps
#Observability
#ApacheKafka
#Kubernetes
#DataEngineering
#CloudArchitecture

More devops & ci/cd diagrams