Kubernetes
Kubernetes Logging: Centralized Log Collection with ELK and Fluentd
Containers are ephemeral, and their logs vanish when pods are terminated or rescheduled. Centralized logging collects logs from every container in your cluster, parses and indexes them, and provides a searchable interface. The EFK stack (Elasticsearch, Fluentd, Kibana) is the most common logging solution for Kubernetes.
Need this done for your project?
We implement, you ship. Async, documented, done in days.
Log Collection with Fluentd DaemonSet
Deploy Fluentd as a DaemonSet so one instance runs on every node. Fluentd reads container logs from `/var/log/containers/*.log` on the host filesystem (mounted via hostPath), parses the JSON log format, enriches entries with Kubernetes metadata (pod name, namespace, labels) using the `fluent-plugin-kubernetes_metadata_filter`, and forwards them to Elasticsearch. Use Fluent Bit as a lighter alternative if resource constraints are tight: it handles log collection and forwarding with a fraction of Fluentd's memory footprint, then routes to Fluentd for complex transformations if needed.
Elasticsearch Deployment and Index Management
Deploy Elasticsearch on Kubernetes using the Elastic Cloud on Kubernetes (ECK) operator, which manages cluster topology, rolling upgrades, and TLS. Start with a 3-node cluster for high availability. Configure index lifecycle management (ILM) to roll over indices daily and delete logs older than 30 days to control storage costs. Use index templates to define mappings for your log fields. For cost-sensitive environments, consider running hot-warm-cold architectures where recent logs live on SSDs and older logs move to cheaper HDD-backed nodes.
Structured Logging and Application Best Practices
Applications should emit logs as structured JSON to stdout. This eliminates the need for complex Fluentd parsing rules and makes logs immediately queryable in Kibana. Include consistent fields: timestamp, log level, service name, request ID, and user ID. Use correlation IDs that propagate across microservices to trace a request through multiple services. Avoid logging sensitive data (passwords, tokens, PII). Configure log levels per environment: DEBUG in development, INFO in staging, WARN in production, adjustable at runtime via ConfigMaps without redeploying.
Alternatives: Loki and Managed Solutions
Grafana Loki is a cost-effective alternative to Elasticsearch that indexes only labels (not full text), making it significantly cheaper to operate. It integrates natively with Grafana for log exploration. Install it via the Loki Helm chart and deploy Promtail as the log collector. For teams that do not want to manage logging infrastructure, cloud-native solutions like AWS CloudWatch Container Insights, Google Cloud Logging, or Datadog provide turn-key log collection with built-in dashboards and alerting, at the cost of per-GB pricing.
Why Anubiz Engineering
100% async — no calls, no meetings
Delivered in days, not weeks
Full documentation included
Production-grade from day one
Security-first approach
Post-delivery support included
Ready to get started?
Skip the research. Tell us what you need, and we'll scope it, implement it, and hand it back — fully documented and production-ready.