Elastic Stack (ELK)

ELK: enough to start indexing; not yet enough for production retention strategy.

Step 1: Run ES + Kibana

The official Elastic compose file boots Elasticsearch, Kibana, and the security plumbing in one command. Use it; do not assemble by hand.

• Compose. docker compose up with the upstream docker-compose.yml from the Elastic docs.
• Memory floor. ES needs 4GB+ heap to be useful; bump vm.max_map_count on Linux or it refuses to start.
• Kibana on 5601. First load takes 60 to 90 seconds while ES initialises; the UI redirects on its own.
• Auth. The elastic superuser password is printed on first boot; copy it, do not lose it.

Step 2: Ship logs

• Install Filebeat; configure to read /var/log/syslog.
• Start Filebeat; ES receives entries.

Step 3: Index template

Without an index template, ES guesses field types on first ingest and you live with whatever it picks. Define the mapping up front.

• Where. Kibana, Stack Management, Index Templates; the UI walks through pattern, mappings, and lifecycle.
• Field mappings. Type each log field explicitly: keyword for IDs, date for timestamps, ip for addresses.
• Lifecycle policy. Attach an ILM policy to roll over and delete by age; otherwise the index grows forever.
• Apply on indexing. Templates only affect future indices; existing indices keep their original mapping until rolled over.

Step 4: Search

Discover is where the operational value lands. Field-based queries are fast; free-text queries are convenient.

• Discover. Kibana, Discover; pick the index pattern; query by field name like level: ERROR.
• Free-text. Quoted strings match against the default message field; useful for known error strings.
• Visualise. Bar chart of error count over time; the simplest dashboard surface in the stack.
• Saved searches. Save the queries the on-call uses; turns ad-hoc spelunking into a runbook step.

Antipatterns

• No index lifecycle policy. Disk fills; ES dies.
• Default replicas=1 with 1 node. No HA.
• Free-text search instead of structured. Slow at scale.

What to do this week

Three moves. (1) Run the tutorial end-to-end on your own laptop / sandbox. (2) Apply the pattern to one production workload. (3) Document the variations you needed; share with the team.