What is log aggregation and how do you implement it with the ELK stack?
Log aggregation centralizes logs from all services into one searchable system. The ELK Stack:
- Elasticsearch: Distributed search and analytics engine that indexes and stores logs.
- Logstash: Data processing pipeline that ingests, transforms, and forwards logs.
- Kibana: Web UI for searching, visualizing, and creating dashboards from Elasticsearch data.
Modern replacement: The EFK Stack uses Fluent Bit (lightweight, lower memory than Logstash) as a DaemonSet in Kubernetes to collect container logs and forward to Elasticsearch. Or use Loki (from Grafana Labs) for a simpler, cost-effective log aggregation layer.