Bear's Log

I’ve been working on a centralized log gathering + monitoring environment for work, which I’ll blog and outline in a longer post soon, but I just had to post about the most recent logstash and Kibana synergy.

Kibana is basically the web UI that Logstash needs - so much so I believe that they are just going to remove the built-in web UI from Logstash. That is good news considering the error I’ll show below :)

The latest version of Kibana is a conversion from a PHP app to a Ruby+Sinatra app which, in my opinion, is absolutely fantastic. It installed cleanly and was up and running in less than 10 minutes.

Logstash itself hadn’t changed all that much between v1.1.1 and v1.1.4 but I feel it has become much more stable (or maybe I’m just using it properly now ;) and with the addition of some new input methods - getting log data into Logstash has become super simple.

But getting data into Logstash is not the chewy goodness at the center of this treat – the structured data stored in Elastic Search and the UI that Kibana offers is the real nugget.

Later I’ll be working on some tools to do pre and post filtering and getting events to be triggered, because after all, what good is all this data if you can’t do something with it!

Configurations

The Logstash cookbook had a good example upstart script that I’ve made few changes to (editor’s note: \o/ pull request merged!)

# logstash - indexer instance
#
description "logstash indexer instance"
start on virtual-filesystems
stop on runlevel [06]

respawn
respawn limit 5 30

# set HOME to point to where you want the embedded elasticsearch
# data directory to be created and ensure /opt/logstash is owned
# by logstash:adm

#env HOME=/opt/logstash
#env JAVA_OPTS='-Xms512m -Xmx512m'

chdir /opt/logstash
setuid logstash
setgid adm
console log

# for versions 1.1.1 - 1.1.4 the internal web service crashes when touched
# and the current workaround is to just not run it and run Kibana instead
script
    exec java -jar logstash.jar agent -f /etc/indexer.conf --log /var/log/logstash-indexer.out
end script

Kibana upstart script that runs it from /opt

# kibana
#
description "Kibana - logstash web search"
start on virtual-filesystems
stop on runlevel [06]
respawn
respawn limit 5 30

chdir /opt/kibana
setuid www-data
setgid www-data
console log

script
    exec ruby kibana.rb
end script

Versions:

• Kibana - git cloned at the time this post was created (github hash: 0715dc4af7df4365130769265275c52b045d49b8 )
• Logstash - 1.1.4