Last week brought the release of Elasticsearch 2.0, Logstash 2.0 and Kibana 4.2. Please visit elastic.co to obtain a copy. There have been multiple improvements and changes within the three environments which has also required me to do some heavy changes to the ELK shell script which I was working on a post for. In its current state, the script is functional however some of the methods for mapping data have been deprecated which is now a slight blocker for some of the automation. It’s a busy time but I am hoping to get the post and video of installation up and available in the next two weeks. For the moment, below is some information regarding the Elastic products:
- Elasticsearch – Data Storage “for search”
- APIs to easily interact with JSON formatted data.
- Uses Apache Lucene for text based searching.
- Simple scalability. Too much data on one node? Spin up a fresh elasticsearch instance, point the new node to the cluster and BAM! One extra line of code in a yml file is all you need.
- Data replication across current and new nodes is automatic! As new nodes get introduced into the environment, sharding occurs without any required interactions.
- Logstash – Monitoring and Parsing
- Monitor files or entire directories for changes.
- As new data is introduced, logstash workers will use your custom filters to extract field names; src ip, dest ip, mac address.
- Use pattern matching to create your custom parsers!
- 2.0 introduces better out of the box defaults such as changes to workers which improves throughput.
- Kibana – Find
- Your WebGUI for all things Elastic. Create dashboards for infrastructure monitoring or security event identification.
In the post, I will be mentioning some of the more important files to modify for a speedy deployment. If you don’t want to use the shell script, the goal will be for anyone to use it as a tutorial to spin up their own instances and begin indexing data from whatever log source they choose. We are a more security focused group so the post will primarily be about security related uses; however, we are planning on using it in our own environment for netflow and general infrastructure administration. Got any tips for us? Please leave us a comment as we’d love to hear from you!