Log Management: ELK Stack, Loki, and Centralized Logging

Applications generate logs — access logs, error traces, debug output, audit trails — but raw logs on disk are nearly useless at scale. When you have dozens of servers and hundreds of services, finding a specific error means searching across machines, correlating timestamps, and parsing inconsistent formats. Centralized log management solves this by collecting, parsing, indexing, and visualizing logs in one place.

This guide covers three major log management stacks: the ELK Stack (Elasticsearch, Logstash, Kibana), Grafana Loki, and the Fluentd ecosystem.

The Log Pipeline

Every log management system follows the same pipeline:

1. Ship — Collect logs from each source (files, stdout, syslog)
2. Parse — Extract structured fields (timestamp, level, service, message)
3. Index — Store logs for fast search
4. Visualize — Query and display logs in dashboards

ELK Stack

The ELK Stack consists of Elasticsearch (storage and search), Logstash (parsing and transformation), and Kibana (visualization). Filebeat handles log shipping.

Architecture

App → Filebeat → Logstash → Elasticsearch → Kibana
       Filebeat → Logstash → Elasticsearch → Kibana

Filebeat Configuration

Filebeat tails log files and ships them to Logstash:

# filebeat.yml
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/nginx/access.log
      - /var/log/nginx/error.log
    fields:
      service: nginx
      environment: production

output.logstash:
  hosts: ["logstash:5044"]

logging.level: info

Filebeat uses a registry file to track which lines have been sent. If the connection drops, it resumes from the last acknowledged position.

Logstash Parsing

Logstash receives logs, parses them, and sends structured data to Elasticsearch:

# logstash.conf
input {
  beats {
    port => 5044
  }
---

filter {
  grok {
    match => {
      "message" => "%{COMBINEDAPACHELOG}"
    }
  }
  date {
    match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
  }
  mutate {
    remove_field => ["message", "path"]
  }
---

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    index => "nginx-logs-%{+YYYY.MM.dd}"
  }
---

The grok filter uses regex patterns to parse unstructured log lines into fields like clientip, response, bytes, and referrer. Custom grok patterns handle application-specific formats.

Elasticsearch Indexing

Elasticsearch indexes parsed logs for near-real-time search. Index lifecycle management (ILM) rolls old indexes:

PUT _ilm/policy/logs
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "50GB",
            "max_age": "30d"
          }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
---

This policy rolls indexes at 50 GB or 30 days, then deletes them after 90 days. Hot nodes handle recent data; warm or cold nodes store older data at lower cost.

Kibana Visualization

In Kibana, create data views matching your index patterns (e.g., nginx-logs-*). Build visualizations:

• Discover — Search logs with Lucene or KQL syntax
• Dashboards — Combine pie charts of response codes, time-series of request rates, and tables of slowest endpoints
• Alerts — Trigger on error rate thresholds or missing log sources

response: >= 500 AND service: nginx

Grafana Loki

Loki is a log aggregation system inspired by Prometheus. Unlike Elasticsearch, Loki indexes only metadata labels and stores raw log content compressed — making it cheaper for high-volume logging.

Architecture

App → Promtail → Loki → Grafana
       Promtail → Loki → Grafana

Promtail Configuration

Promtail ships logs and attaches labels:

# promtail-config.yml
scrape_configs:
  - job_name: nginx
    static_configs:
      - targets: [localhost]
        labels:
          job: nginx
          environment: production
          __path__: /var/log/nginx/*.log
    pipeline_stages:
      - regex:
          expression: '^(?P<remote_ip>\S+) \S+ \S+ \[(?P<timestamp>[^\]]+)\] "(?P<method>\S+) (?P<path>\S+) \S+" (?P<status>\d+)'
      - labels:
          status:
      - timestamp:
          source: timestamp
          format: "02/Jan/2006:15:04:05 -0700"

LogQL Queries

Grafana uses LogQL — similar to PromQL — to query Loki:

{job="nginx", environment="production"} |= "error"
| json
| status >= 500
| rate($__interval)

LogQL supports label matchers, content filters (|=, !=, |~), and aggregation. The query above returns the rate of 5xx errors per interval.

Loki vs Elasticsearch

| Feature | Loki | Elasticsearch | |

Fluentd

Fluentd is an open-source data collector that unifies log shipping with a plugin architecture. It treats logs as JSON events and routes them through pipelines.

Configuration

# fluentd.conf
<source>
  @type tail
  path /var/log/nginx/access.log
  format apache2
  tag nginx.access
  pos_file /var/log/fluentd/nginx.access.pos
</source>

<source>
  @type forward
  port 24224
</source>

<filter nginx.access>
  @type record_transformer
  <record>
    hostname ${hostname}
    service nginx
  </record>
</filter>

<match nginx.access>
  @type copy
  <store>
    @type elasticsearch
    host elasticsearch
    port 9200
    logstash_format true
  </store>
  <store>
    @type file
    path /var/log/fluentd/backup
  </store>
</match>

Buffer and Retry

<match **>
  @type elasticsearch
  buffer_type file
  buffer_path /var/log/fluentd/buffer
  flush_interval 5s
  retry_limit 17
  retry_wait 1s
  max_retry_wait 300s
</match>

Fluentd buffers logs to disk before sending. If Elasticsearch is down, logs accumulate in the buffer and are retried with exponential backoff. This prevents data loss during outages.

Structured Logging Practices

The best log management starts in application code:

// Good: Structured JSON
logger.info({
  event: "user.login",
  userId: user.id,
  ip: req.ip,
  duration_ms: Date.now() - start,
---);

// Bad: Unstructured text
logger.info(`User ${user.id} logged in from ${req.ip}`);

Structured logs eliminate parsing guesswork. Follow these conventions:

• Use consistent field names across services (event, duration_ms, error_code)
• Include correlation IDs for request tracing
• Log at appropriate levels: DEBUG for development, INFO for normal operations, WARN for anomalies, ERROR for failures
• Never log sensitive data — PII, passwords, tokens

Log Rotation and Retention

Prevent disks from filling:

# /etc/logrotate.d/nginx
/var/log/nginx/*.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    sharedscripts
    postrotate
        nginx -s reopen
    endscript
---

Set retention policies at the aggregation layer. A common strategy:

• Hot storage (7 days): Full-text searchable, high performance
• Warm storage (30 days): Searchable, compressed
• Cold storage (90+ days): Archived to S3 or GCS, searchable with delays

Best Practices

• Tag everything — Add service name, environment, version, and hostname as labels or fields
• Use structured logging — JSON logs with consistent schemas eliminate parsing errors
• Monitor the log pipeline — Alerts on log shipping delays or dropped events
• Sample high-volume logs — For debug-level or access logs, sample 1% instead of dropping everything
• Correlate with metrics — Link log queries to Prometheus metrics for full observability

Summary

Centralized log management turns noisy, distributed log files into searchable, actionable data. The ELK Stack provides full-text search and rich visualizations. Grafana Loki offers cheaper label-based indexing ideal for Kubernetes. Fluentd bridges sources and destinations with reliable buffering. Regardless of the stack, structured logging and consistent field naming make the difference between a useful log system and an expensive archive.

Log Management Architecture

Log Collection Strategies

Log collection begins at the source — each application and system component produces log output through stdout/stderr, files, or syslog. The sidecar pattern deploys a log shipper container (Filebeat, Fluentd, Vector) alongside each application container, reading log files from a shared volume. The node-level shipper deploys a single shipper per host that collects logs from all containers on that node. Node-level collection is simpler but loses per-pod metadata that sidecar collection preserves. For Kubernetes, the DaemonSet pattern deploys a log collector pod on every node. Cloud-native applications should write structured logs to stdout rather than files — container runtimes capture stdout automatically, and orchestration platforms handle log rotation and storage. The collector adds metadata (pod name, namespace, service name) and sends logs to a central aggregation point over a reliable transport protocol.

Log Processing Pipeline

The log processing pipeline transforms raw log lines into searchable, analyzable records. The parser stage extracts structured fields from log formats — JSON parsing automatically extracts fields, while regex parsing handles legacy formats. The enrichment stage adds context: environment, version, data center, and geo-location from IP addresses. The normalization stage maps fields from different services to a common schema, enabling cross-service searches. Logs are then buffered in a message queue (Kafka, Redis) for durability and backpressure handling during traffic spikes. The indexing stage writes to the search backend (Elasticsearch, Loki, ClickHouse). The pipeline must handle schema evolution — new fields should be added without breaking existing queries. Field type conflicts — one service writes duration as a string, another as a number — must be resolved through schema mapping.

Retention and Storage Cost

Log storage costs grow linearly with volume and retention period. A service generating 100 GB of logs daily costs approximately $3,000-5,000 per month for 30-day retention in Elasticsearch. Cost optimization strategies include tiered retention — hot storage for 7-14 days with full-text search, warm storage for 30-90 days with reduced replica count, and cold storage (S3, GCS) for archival with query-on-demand. Sampling high-volume debug logs reduces storage while maintaining diagnostic capability. Aggregation pre-computes metrics from logs and stores only the aggregations (request counts, error rates, p50/p99 latencies) rather than individual log entries. Log lifecycle policies automatically delete or archive logs based on age and importance. Security and compliance requirements may mandate minimum retention periods (typically 1-7 years) that must be balanced against storage costs through archival strategies.

Frequently Asked Questions

What is the difference between structured and unstructured logging? Structured logging outputs machine-parseable formats (JSON) with named fields — {"timestamp": "...", "level": "error", "service": "api", "duration_ms": 42}. Unstructured logging outputs human-readable strings — “API request failed in 42ms”. Structured logs are searchable, filterable, and analyzable without regex parsing. All production systems should adopt structured logging.

How do I prevent log storage from consuming all disk space? Implement log rotation with size and time policies, configure retention limits in your log aggregation system, use log level filtering to avoid storing debug logs in production, and set disk quotas. Use a separate partition for logs to prevent application crashes from log-related disk exhaustion. Monitor log storage usage and alert on trends.

What is the ELK stack and should I use it? ELK (Elasticsearch, Logstash, Kibana) is the traditional open-source log management stack. Elasticsearch indexes and searches logs, Logstash processes and transforms them, Kibana visualizes and explores them. Modern alternatives include Loki (Grafana’s log system designed for Kubernetes, cheaper than Elasticsearch), SigNoz, and managed services like Datadog and Splunk. The choice depends on scale, budget, and existing infrastructure.

For a comprehensive overview, read our article on Blue Green Deployments.

For a comprehensive overview, read our article on Ci Cd Pipeline Guide.