elk-stack
/elk-stack
Deploy and manage the ELK Stack (Elasticsearch, Logstash, Kibana) for log aggregation and analysis. Configure log pipelines, create visualizations, and implement log-based monitoring. Use when centralizing logs, implementing search functionality, or building log analytics platforms.
- Author
- BagelHole ↗
- Repository
- DevOps-Security-Agent-Skills ↗
- Licence
- MIT
- Last repository change
- 22.05.2026
Assessment and security
- Files in this package
- 2
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 1
No signals detected. Repository changed on 22.05.2026, without references to retired models or APIs.
MIT, open: use and modify while retaining author attribution.
What it does and when to use it
Deploy and manage the ELK Stack (Elasticsearch, Logstash, Kibana) for log aggregation and analysis. Configure log pipelines, create visualizations, and implement log-based monitoring. Use when centralizing logs, implementing search functionality, or building log analytics platforms.
Original author description
Deploy and manage the ELK Stack (Elasticsearch, Logstash, Kibana) for log aggregation and analysis. Configure log pipelines, create visualizations, and implement log-based monitoring. Use when centralizing logs, implementing search functionality, or building log analytics platforms.
How it works
- When to Use This Skill
- Prerequisites
- Docker Deployment
- Elasticsearch Configuration
- Logstash Pipeline
- Filebeat Configuration
- Elasticsearch Queries
- Kibana Setup
- Alerting
- Common Issues
SKILL.md preview
---
name: elk-stack
description: Deploy and manage the ELK Stack (Elasticsearch, Logstash, Kibana) for log aggregation and analysis. Configure log pipelines, create visualizations, and implement log-based monitoring. Use when centralizing logs, implementing search functionality, or building log analytics platforms.
license: MIT
metadata:
author: devops-skills
version: "1.0"
---
# ELK Stack
Centralize and analyze logs with Elasticsearch, Logstash, and Kibana.
## When to Use This Skill
Use this skill when:
- Centralizing logs from multiple sources
- Building log search and analytics platforms
- Creating log-based dashboards and alerts
- Implementing full-text search for logs
- Processing and transforming log data
## Prerequisites
- Docker or server infrastructure
- Sufficient disk space for log storage
- Network access from log sources
## Docker Deployment
```yaml
# docker-compose.yml
version: '3.8'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
environment:
- discovery.type=single-node
- xpack.security.enabled=false
- "ES_JAVA_OPTS=-Xms1g -Xmx1g"
ports:
- "9200:9200"
volumes:
- elasticsearch-data:/usr/share/elasticsearch/data
logstash:
image: docker.elastic.co/logstash/logstash:8.11.0
volumes:
- ./logstash/pipeline:/usr/share/logstash/pipeline
- ./logstash/config:/usr/share/logstash/config
ports:
- "5044:5044"
- "5000:5000"
depends_on:
- elasticsearch
kibana:
image: docker.elastic.co/kibana/kibana:8.11.0
ports:
- "5601:5601"
environment:
- ELASTICSEARCH_HOSTS=http://elasticsearch:9200
depends_on:
- elasticsearch
filebeat:
image: docker.elastic.co/beats/filebeat:8.11.0
user: root
volumes:
- ./filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
- /var/lib/docker/containers:/var/lib/docker/containers:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
depends_on:
- logstash
volumes:
elasticsearch-data:
```
## Elasticsearch Configuration
### Index Templates
```json
PUT _index_template/logs-template
{
"index_patterns": ["logs-*"],
"template": {
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1,
"index.lifecycle.name": "logs-policy"
},
"mappings": {
"properties": {
"@timestamp": { "type": "date" },
"message": { "type": "text" },
"level": { "type": "keyword" },
"service": { "type": "keyword" },
"host": { "type": "keyword" },
"trace_id": { "type": "keyword" }
}
}
}
}
```
### Index Lifecycle Management
```json
PUT _ilm/policy/logs-policy
{
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_size": "50GB",
"max_age": "1d"
}
}
},
"warm": {
"min_age": "7d",
"actions": {
"shrink": { "number_of_shards": 1 },
"forcemerge": { "max_num_segments": 1 }
}
},
"cold": {
"min_age": "30d",
"actions": {
"freeze": {}
}
},
"delete": {
"min_age": "90d",
"actions": {
"delete": {}
}
}
}
}
}
```
## Logstash Pipeline
### Basic Pipeline
```ruby
# logstash/pipeline/main.conf
input {
beats {
port => 5044
}
tcp {
port => 5000
codec => json_lines
}
}
filter {
# Parse JSON logs
if [message] =~ /^\{/ {
json {
source => "message"
}
}
# Parse timestamp
date {
match => ["timestamp", "ISO8601", "yyyy-MM-dd HH:mm:ss"]
target => "@timestamp"
}
# Add environment tag
mutate {
add_field => { "environment" => "production" }
}
# Grok pattern for nginx logs
if [type] == "nginx" {
grok {
match => {
"message" => '%{IPORHOST:client_ip} - %{USER:user} \[%{HTTPDATE:timestamp}\] "%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:http_version}" %{NUMBER:status} %{NUMBER:bytes}'
}
}
}
}
output {
elasticsearch {
hosts => ["elasticsearch:9200"]
index => "logs-%{+YYYY.MM.dd}"
}
}
```
### Advanced Filtering
```ruby
filter {
# Parse application logs
grok {
match => {
"message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} \[%{DATA:service}\] %{GREEDYDATA:log_message}"
}
}
# Extract trace ID from message
if [log_message] =~ /trace_id=/ {
grok {
match => { "log_message" => "trace_id=%{UUID:trace_id}" }
}
}
# GeoIP lookup
if [client_ip] {
geoip {
source => "client_ip"
target => "geoip"
}
}
# Drop debug logs in production
if [level] == "DEBUG" and [environment] == "production" {
drop {}
}
# Enrich with lookup
translate {
field => "status"
destination => "status_description"
dictionary => {
"200" => "OK"
"404" => "Not Found"
"500" => "Internal Server Error"
}
}
}
```
## Filebeat Configuration
```yaml
# filebeat/filebeat.yml
filebeat.inputs:
- type: container
paths:
- '/var/lib/docker/containers/*/*.log'
processors:
- add_docker_metadata:
host: "unix:///var/run/docker.sock"
- type: log
enabled: true
paths:
- /var/log/nginx/*.log
tags: ["nginx"]
fields:
type: nginx
output.logstash:
hosts: ["logstash:5044"]
logging.level: info
logging.to_files: true
logging.files:
path: /var/log/filebeat
name: filebeat
keepfiles: 7
```
## Elasticsearch Queries
### Basic Queries
```json
// Search all logs
GET logs-*/_search
{
"query": {
"match_all": {}
}
}
// Search by keyword
GET logs-*/_search
{
"query": {
"match": {
"message": "error"
}
}
}
// Filter by field
GET logs-*/_search
{
"query": {
"bool": {
"must": [
{ "match": { "level": "ERROR" } },
{ "range": { "@timestamp": { "gte": "now-1h" } } }
],
"filter": [
{ "term": { "service": "api-gateway" } }
]
}
}
}
```
### Aggregations
```json
// Count by log level
GET logs-*/_search
{
"size": 0,
"aggs": {
"log_levels": {
"terms": { "field": "level" }
}
}
}
// Error rate over time
GET logs-*/_search
{
"size": 0,
"aggs": {
"errors_over_time": {
"date_histogram": {
"field": "@timestamp",
"fixed_interval": "5m"
},
"aggs": {
"error_count": {
"filter": { "term": { "level": "ERROR" } }
}
}
}
}
}
```
## Kibana Setup
### Index Patterns
1. Go to Stack Management → Index Patterns
2. Create pattern: `logs-*`
3. Set time field: `@timestamp`
### Saved Searches
Create saved searches for common queries:
- `level:ERROR` - All errors
- `service:api-gateway AND level:ERROR` - API gateway errors
- `response_time:>1000` - Slow requests
### Visualizations
Common visualization types:
- **Line Chart**: Error rate over time
- **Pie Chart**: Distribution by log level
- **Data Table**: Top error messages
- **Metric**: Total error count
### Dashboard Example
Create dashboard with:
1. Total log count (Metric)
2. Error rate trend (Line chart)
3. Logs by service (Pie chart)
4. Recent errors (Data table)
5. Log stream (Discover panel)
## Alerting
### Watcher (X-Pack)
```json
PUT _watcher/watch/error_alert
{
"trigger": {
"schedule": { "interval": "5m" }
},
"input": {
"search": {
"request": {
"indices": ["logs-*"],
"body": {
"query": {
"bool": {
"must": [
{ "match": { "level": "ERROR" } },
{ "range": { "@timestamp": { "gte": "now-5m" } } }
]
}
}
}
}
}
},
"condition": {
"compare": { "ctx.payload.hits.total.value": { "gt": 100 } }
},
"actions": {
"notify_slack": {
"webhook": {
"scheme": "https",
"host": "hooks.slack.com",
"port": 443,
"method": "post",
"path": "/services/xxx",
"body": "{\"text\": \"High error rate detected: {{ctx.payload.hits.total.value}} errors in last 5 minutes\"}"
}
}
}
}
```
## Common Issues
### Issue: High Disk Usage
**Problem**: Elasticsearch consuming too much disk
**Solution**: Implement ILM policies, reduce retention
### Issue: Slow Searches
**Problem**: Queries taking too long
**Solution**: Optimize index settings, add more shards, use filters
### Issue: Log Parsing Failures
**Problem**: Logs not parsed correctly
**Solution**: Test grok patterns, check for log format changes
### Issue: Memory Pressure
**Problem**: Elasticsearch OOM errors
**Solution**: Increase heap size (max 50% of RAM), limit field data
## Best Practices
- Implement index lifecycle management
- Use index templates for consistent mappings
- Parse logs at ingestion time
- Limit stored fields to reduce storage
- Use data streams for time-series data
- Monitor cluster health
- Implement proper security (X-Pack)
- Regular index maintenance
## Related Skills
- [loki-logging](../loki-logging/) - Alternative logging stack
- [prometheus-grafana](../prometheus-grafana/) - Metrics monitoring
- [audit-logging](../../../compliance/auditing/audit-logging/) - Compliance logging
Files in this package (2)
- SKILL.md 9 KB
- references/elasticsearch-queries.md 2 KB
Versions
The database has no other skills named elk-stack.
Database snapshot: 13.11.2026 · package version e6e3645b3f05 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100elasticsearch-expert
Expert-level Elasticsearch, search, ELK stack, and full-text search. Use when the user mentions search, ELK, Logstash, Kibana, or full text search. - 100elk
Covers the Elastic Stack (ELK) 8.x/9.x for observability use cases: Elasticsearch cluster management, Elastic Agent/Fleet, Kibana, ingest pipelines, KQL/ES|QL, Index Lifecycle Management, APM agents, distributed tracing, log management, data streams, and diagnostics. Use for… - 100logstash
Configure Logstash for log ingestion, parsing, transformation, and output to Elasticsearch and other destinations. Use when a user needs to build log processing pipelines, write Grok patterns, parse unstructured logs, enrich events, or set up multi-pipeline Logstash deployments. - 91ubuntu-monitoring
Use when setting up monitoring and logging on Ubuntu 24.04 LTS — Prometheus and node_exporter, Grafana dashboards, Loki/Alloy log aggregation (Promtail legacy), ELK stack (Elasticsearch, Logstash, Kibana), alerting with Alertmanager, systemd journal management, rsyslog, and… - 100setting-up-log-aggregation
Execute use when setting up log aggregation solutions using ELK, Loki, or Splunk. Trigger with phrases like "setup log aggregation", "deploy ELK stack", "configure Loki", or "install Splunk". Generates production-ready configurations for data ingestion, processing, storage, and… - 100setting-up-log-aggregation
Execute use when setting up log aggregation solutions using ELK, Loki, or Splunk. Trigger with phrases like "setup log aggregation", "deploy ELK stack", "configure Loki", or "install Splunk". Generates production-ready configurations for data ingestion, processing, storage, and…