Impressive Performance With Elastic 6.5.1 & Search Guard

After Implementing Search Guard ten days ago I was finally pushed into using Elasticsearch 6. Having noticed that 6.5.0 was out I decided to wait until Search Guard, which seems to lag about a week behind, managed to get their update done.

The 6.5.0 release proved terribly buggy, but now here we are with 6.5.1, running tests in A Small Development Environment, and the results are impressive. The combination of this code and an upgrade from Ubuntu 16.04 to 18.04 has made the little test machine, which we refer to as ‘hotpot‘, as speedy as our three node VPS based cluster.

Perflog Dashboard
Perflog Dashboard

This is a solid long term average of fully collecting over eleven accounts per minute, but the curious thing is that it’s not obvious what resource is limiting throughput. Ram utilization eventually ratcheted up to 80% but the CPU load average has been not more than 20% the whole time.

Utilization Dashboard
Utilization Dashboard

There is still a long learning curve ahead, but what I think I see here is that an elderly four core i7, if it has a properly tuned zpool disk subsystem, will be able to support a group of eight users in constant collection mode.

Kimsufi Servers
Kimsufi Servers

And that makes this page of Kimsufi Servers intriguing.  The KS-9 looks to be the sweet spot, due to the presence of SSDs instead of spindles. If our monthly hardware is $21 that puts us in a place where maybe a $99/month small team setup makes sense to offer.

There is much to be done with Search Guard before this can happen, but hopefully we’ll be ready at the start of 2019.

Implementing Search Guard

Our conversion to Elasticsearch began almost a year ago. Aided by their marvelous O’Reilly book, Elasticsearch: The Definitive Guide, we grew comfortable with the system, exploring Timesketch and implementing Wazuh for our internal monitoring. Our concerns here were the same issue we faced during prior Splunk adventures – how do we fund the annual cost of an enterprise license?

Search Guard solves the explicit cost question and it does a good job on the implicit barrier to entry problem. What you see below is the contents of the Search Guard tab on our prototype system, which more or less installed itself with single command.

Search Guard
Search Guard

The initial experience was so smooth we decided to implement Search Guard on our cluster, which has been a learning experience. The system requires Elasticsearch 6.x, but we have clung to the familiar environment of the 5.6 version of the system. The switch required a solid day of fiddling with Bash scripts and Python code in order to make everything work with the newest Elasticsearch, and then the cluster upgrade was not nearly so straightforward.

The self-installing demo reuses a PKI setup. That’s great for lowering the barrier to entry for initial experiments, but there is no way that can be used on a publicly accessible system. Having done a bit of PKI here and there, the instructions and scripts they offer are fairly smooth.

The troubles began when we moved from Elasticsearch 5.6.13 to 6.4.3. A stumble on our part during Search Guard install left us with a system that was stuck tight. Their install procedure could not continue from the state we had put the system into, while our command line tools and system knowledge were insufficient to back out of the partially completed process.

Resolving that took the better part of a day, but it proved beneficial in the end, as their voluminous documentation did not address the specific problem, but it did offer many pointers. Think: six months troubleshooting experience in an afternoon, and questions posted to their Google Group yielded authoritative answers within hours.

There are six weeks left in 2018 and during this time we are going to accomplish the following:

  • Finish converting botnetsu.press to Search Guard Enterprise
  • Index Twitter, RSS data, and one chat service for the team
  • Explore roles and permissions against real world considerations
  • Create a public facing dashboard for botnetsu.press
  • Implement Search Guard for a Wazuh system

Best of all, Search Guard offers a gratis Enterprise license to non-profits. We have applied for this for both botnetsu.press here in the U.S., as well as a similar effort in the U.K. Given just a bit of luck, we’ll have two teams active in the field by the end of first quarter, and maybe some of the commercial opportunities we are pursuing will come to fruition as well.