Skip to content
Skills Library
DevOps & cloud · Automation

elk-stack

/elk-stack

Deploy and manage the ELK Stack (Elasticsearch, Logstash, Kibana) for log aggregation and analysis. Configure log pipelines, create visualizations, and implement log-based monitoring. Use when centralizing logs, implementing search functionality, or building log analytics platforms.

Licence
MIT
Last repository change
22.05.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
2
Scripts
0
HTML comments
0
URLs in SKILL.md
1
Deprecation

No signals detected. Repository changed on 22.05.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Deploy and manage the ELK Stack (Elasticsearch, Logstash, Kibana) for log aggregation and analysis. Configure log pipelines, create visualizations, and implement log-based monitoring. Use when centralizing logs, implementing search functionality, or building log analytics platforms.

Original author description

Deploy and manage the ELK Stack (Elasticsearch, Logstash, Kibana) for log aggregation and analysis. Configure log pipelines, create visualizations, and implement log-based monitoring. Use when centralizing logs, implementing search functionality, or building log analytics platforms.

How it works

  1. When to Use This Skill
  2. Prerequisites
  3. Docker Deployment
  4. Elasticsearch Configuration
  5. Logstash Pipeline
  6. Filebeat Configuration
  7. Elasticsearch Queries
  8. Kibana Setup
  9. Alerting
  10. Common Issues
Original SKILL.md headings · 455 lines, 1,052 words
SKILL.md preview
---
name: elk-stack
description: Deploy and manage the ELK Stack (Elasticsearch, Logstash, Kibana) for log aggregation and analysis. Configure log pipelines, create visualizations, and implement log-based monitoring. Use when centralizing logs, implementing search functionality, or building log analytics platforms.
license: MIT
metadata:
  author: devops-skills
  version: "1.0"
---

# ELK Stack

Centralize and analyze logs with Elasticsearch, Logstash, and Kibana.

## When to Use This Skill

Use this skill when:
- Centralizing logs from multiple sources
- Building log search and analytics platforms
- Creating log-based dashboards and alerts
- Implementing full-text search for logs
- Processing and transforming log data

## Prerequisites

- Docker or server infrastructure
- Sufficient disk space for log storage
- Network access from log sources

## Docker Deployment

```yaml
# docker-compose.yml
version: '3.8'

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
    ports:
      - "9200:9200"
    volumes:
      - elasticsearch-data:/usr/share/elasticsearch/data

  logstash:
    image: docker.elastic.co/logstash/logstash:8.11.0
    volumes:
      - ./logstash/pipeline:/usr/share/logstash/pipeline
      - ./logstash/config:/usr/share/logstash/config
    ports:
      - "5044:5044"
      - "5000:5000"
    depends_on:
      - elasticsearch

  kibana:
    image: docker.elastic.co/kibana/kibana:8.11.0
    ports:
      - "5601:5601"
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    depends_on:
      - elasticsearch

  filebeat:
    image: docker.elastic.co/beats/filebeat:8.11.0
    user: root
    volumes:
      - ./filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    depends_on:
      - logstash

volumes:
  elasticsearch-data:
```

## Elasticsearch Configuration

### Index Templates

```json
PUT _index_template/logs-template
{
  "index_patterns": ["logs-*"],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "index.lifecycle.name": "logs-policy"
    },
    "mappings": {
      "properties": {
        "@timestamp": { "type": "date" },
        "message": { "type": "text" },
        "level": { "type": "keyword" },
        "service": { "type": "keyword" },
        "host": { "type": "keyword" },
        "trace_id": { "type": "keyword" }
      }
    }
  }
}
```

### Index Lifecycle Management

```json
PUT _ilm/policy/logs-policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "50GB",
            "max_age": "1d"
          }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "shrink": { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "freeze": {}
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}
```

## Logstash Pipeline

### Basic Pipeline

```ruby
# logstash/pipeline/main.conf
input {
  beats {
    port => 5044
  }
  
  tcp {
    port => 5000
    codec => json_lines
  }
}

filter {
  # Parse JSON logs
  if [message] =~ /^\{/ {
    json {
      source => "message"
    }
  }
  
  # Parse timestamp
  date {
    match => ["timestamp", "ISO8601", "yyyy-MM-dd HH:mm:ss"]
    target => "@timestamp"
  }
  
  # Add environment tag
  mutate {
    add_field => { "environment" => "production" }
  }
  
  # Grok pattern for nginx logs
  if [type] == "nginx" {
    grok {
      match => {
        "message" => '%{IPORHOST:client_ip} - %{USER:user} \[%{HTTPDATE:timestamp}\] "%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:http_version}" %{NUMBER:status} %{NUMBER:bytes}'
      }
    }
  }
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    index => "logs-%{+YYYY.MM.dd}"
  }
}
```

### Advanced Filtering

```ruby
filter {
  # Parse application logs
  grok {
    match => {
      "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} \[%{DATA:service}\] %{GREEDYDATA:log_message}"
    }
  }
  
  # Extract trace ID from message
  if [log_message] =~ /trace_id=/ {
    grok {
      match => { "log_message" => "trace_id=%{UUID:trace_id}" }
    }
  }
  
  # GeoIP lookup
  if [client_ip] {
    geoip {
      source => "client_ip"
      target => "geoip"
    }
  }
  
  # Drop debug logs in production
  if [level] == "DEBUG" and [environment] == "production" {
    drop {}
  }
  
  # Enrich with lookup
  translate {
    field => "status"
    destination => "status_description"
    dictionary => {
      "200" => "OK"
      "404" => "Not Found"
      "500" => "Internal Server Error"
    }
  }
}
```

## Filebeat Configuration

```yaml
# filebeat/filebeat.yml
filebeat.inputs:
  - type: container
    paths:
      - '/var/lib/docker/containers/*/*.log'
    processors:
      - add_docker_metadata:
          host: "unix:///var/run/docker.sock"

  - type: log
    enabled: true
    paths:
      - /var/log/nginx/*.log
    tags: ["nginx"]
    fields:
      type: nginx

output.logstash:
  hosts: ["logstash:5044"]

logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
```

## Elasticsearch Queries

### Basic Queries

```json
// Search all logs
GET logs-*/_search
{
  "query": {
    "match_all": {}
  }
}

// Search by keyword
GET logs-*/_search
{
  "query": {
    "match": {
      "message": "error"
    }
  }
}

// Filter by field
GET logs-*/_search
{
  "query": {
    "bool": {
      "must": [
        { "match": { "level": "ERROR" } },
        { "range": { "@timestamp": { "gte": "now-1h" } } }
      ],
      "filter": [
        { "term": { "service": "api-gateway" } }
      ]
    }
  }
}
```

### Aggregations

```json
// Count by log level
GET logs-*/_search
{
  "size": 0,
  "aggs": {
    "log_levels": {
      "terms": { "field": "level" }
    }
  }
}

// Error rate over time
GET logs-*/_search
{
  "size": 0,
  "aggs": {
    "errors_over_time": {
      "date_histogram": {
        "field": "@timestamp",
        "fixed_interval": "5m"
      },
      "aggs": {
        "error_count": {
          "filter": { "term": { "level": "ERROR" } }
        }
      }
    }
  }
}
```

## Kibana Setup

### Index Patterns

1. Go to Stack Management → Index Patterns
2. Create pattern: `logs-*`
3. Set time field: `@timestamp`

### Saved Searches

Create saved searches for common queries:
- `level:ERROR` - All errors
- `service:api-gateway AND level:ERROR` - API gateway errors
- `response_time:>1000` - Slow requests

### Visualizations

Common visualization types:
- **Line Chart**: Error rate over time
- **Pie Chart**: Distribution by log level
- **Data Table**: Top error messages
- **Metric**: Total error count

### Dashboard Example

Create dashboard with:
1. Total log count (Metric)
2. Error rate trend (Line chart)
3. Logs by service (Pie chart)
4. Recent errors (Data table)
5. Log stream (Discover panel)

## Alerting

### Watcher (X-Pack)

```json
PUT _watcher/watch/error_alert
{
  "trigger": {
    "schedule": { "interval": "5m" }
  },
  "input": {
    "search": {
      "request": {
        "indices": ["logs-*"],
        "body": {
          "query": {
            "bool": {
              "must": [
                { "match": { "level": "ERROR" } },
                { "range": { "@timestamp": { "gte": "now-5m" } } }
              ]
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": { "ctx.payload.hits.total.value": { "gt": 100 } }
  },
  "actions": {
    "notify_slack": {
      "webhook": {
        "scheme": "https",
        "host": "hooks.slack.com",
        "port": 443,
        "method": "post",
        "path": "/services/xxx",
        "body": "{\"text\": \"High error rate detected: {{ctx.payload.hits.total.value}} errors in last 5 minutes\"}"
      }
    }
  }
}
```

## Common Issues

### Issue: High Disk Usage
**Problem**: Elasticsearch consuming too much disk
**Solution**: Implement ILM policies, reduce retention

### Issue: Slow Searches
**Problem**: Queries taking too long
**Solution**: Optimize index settings, add more shards, use filters

### Issue: Log Parsing Failures
**Problem**: Logs not parsed correctly
**Solution**: Test grok patterns, check for log format changes

### Issue: Memory Pressure
**Problem**: Elasticsearch OOM errors
**Solution**: Increase heap size (max 50% of RAM), limit field data

## Best Practices

- Implement index lifecycle management
- Use index templates for consistent mappings
- Parse logs at ingestion time
- Limit stored fields to reduce storage
- Use data streams for time-series data
- Monitor cluster health
- Implement proper security (X-Pack)
- Regular index maintenance

## Related Skills

- [loki-logging](../loki-logging/) - Alternative logging stack
- [prometheus-grafana](../prometheus-grafana/) - Metrics monitoring
- [audit-logging](../../../compliance/auditing/audit-logging/) - Compliance logging
Files in this package (2)
  • SKILL.md 9 KB
  • references/elasticsearch-queries.md 2 KB

Versions

The database has no other skills named elk-stack.

Database snapshot: 13.11.2026 · package version e6e3645b3f05 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100elasticsearch-expert
    Expert-level Elasticsearch, search, ELK stack, and full-text search. Use when the user mentions search, ELK, Logstash, Kibana, or full text search.
  • 100elk
    Covers the Elastic Stack (ELK) 8.x/9.x for observability use cases: Elasticsearch cluster management, Elastic Agent/Fleet, Kibana, ingest pipelines, KQL/ES|QL, Index Lifecycle Management, APM agents, distributed tracing, log management, data streams, and diagnostics. Use for…
  • 100logstash
    Configure Logstash for log ingestion, parsing, transformation, and output to Elasticsearch and other destinations. Use when a user needs to build log processing pipelines, write Grok patterns, parse unstructured logs, enrich events, or set up multi-pipeline Logstash deployments.
  • 91ubuntu-monitoring
    Use when setting up monitoring and logging on Ubuntu 24.04 LTS — Prometheus and node_exporter, Grafana dashboards, Loki/Alloy log aggregation (Promtail legacy), ELK stack (Elasticsearch, Logstash, Kibana), alerting with Alertmanager, systemd journal management, rsyslog, and…
  • 100setting-up-log-aggregation
    Execute use when setting up log aggregation solutions using ELK, Loki, or Splunk. Trigger with phrases like "setup log aggregation", "deploy ELK stack", "configure Loki", or "install Splunk". Generates production-ready configurations for data ingestion, processing, storage, and…
  • 100setting-up-log-aggregation
    Execute use when setting up log aggregation solutions using ELK, Loki, or Splunk. Trigger with phrases like "setup log aggregation", "deploy ELK stack", "configure Loki", or "install Splunk". Generates production-ready configurations for data ingestion, processing, storage, and…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.