Installing Search Guard

One of the barriers to making the Netwar System more broadly available has been taking the step of making a system publicly available. We used an Apache reverse proxy and passwords at first, but that was clumsy and limited. We also wanted to make select Visualizations available via iframe embedding, and that is a forbidding transit without a proper map.

Search Guard provides a demo install and it’s great for what it is, providing a working base configuration and most importantly furnishing an operational Certificate Authority. If you took a look around their site, you know they make their money with enterprise licensing, often with a compliance angle. The demo’s top level config file gives equal time to Active Directory, Kerberos, LDAP, and JSON Web Tokens. An enterprise solution architect is going to give a pleased nod to this – there’s something for everyone in there.

But our position is a bit different. Certificate Authority creation is a niche that anyone who has handled an ISP or hosting operation understands, but there seems to be an implicit assumption in the Search Guard documents – that the reader is already familiar with Elasticsearch in an enterprise context. None of us had ever used it in an enterprise setting, so it’s been a steep learning curve.

This post can be seen as a follow on to Installing Netwar System, which covers how to commission a system without securing it for a team. We currently use Elasticsearch 6.5.4 in production, this article is going to cover using 6.7.1, which we need to start exploring.

The instructions for Installing Netwar System cover not just Elasticsearch, they also address tuning that has to be done in sysctl.conf, limits.conf, which you need to do if you plan on putting any sort of load on the system.

The first step is installing the Searchguard Demo, starting from /usr/share/elasticsearch:

bin/elasticsearch-plugin install -b com.floragunn:search-guard-6:6.7.1-24.3

This will complete the install, but with the following warnings:

@ WARNING: plugin requires additional permissions @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ /proc/sys/net/core/somaxconn read
java.lang.RuntimePermission accessClassInPackage.sun.misc
java.lang.RuntimePermission accessDeclaredMembers
java.lang.RuntimePermission accessUserInformation
java.lang.RuntimePermission createClassLoader
java.lang.RuntimePermission getClassLoader
java.lang.RuntimePermission setContextClassLoader
java.lang.RuntimePermission shutdownHooks
java.lang.reflect.ReflectPermission suppressAccessChecks getNetworkInformation getProxySelector * connect,accept,resolve getProperty.ssl.KeyManagerFactory.algorithm insertProvider.BC putProviderProperty.BC setProperty.ocsp.enable
java.util.PropertyPermission * read,write
java.util.PropertyPermission write doAs modifyPrivateCredentials * accept
for descriptions of what these permissions allow and the associated risks.
-> Installed search-guard-6

This is fine, no action required. The next step is to execute the actual demo installation script.

cd /usr/share/elasticsearch/plugins/search-guard-6/tools/
chmod 755

This will install the demo’s files in /etc/elasticsearch and it will provide a tip on how to update its configuration, using the following command. I grew weary of hunting for this, so I turned it into a shell script called /usr/local/bin/updatesg.

/usr/share/elasticsearch/plugins/search-guard-6/tools/ -cd "/usr/share/elasticsearch/plugins/search-guard-6/sgconfig" -icl -key "/etc/elasticsearch/kirk-key.pem" -cert "/etc/elasticsearch/kirk.pem" -cacert "/etc/elasticsearch/root-ca.pem" -nhnv

What this is doing, briefly, is picking up the files in the sgconfig directory and applying them to the running instance, using a client certificate called ‘kirk’, which was signed by the local root Certificate Authority.

There are five files that are part of the configuration and a sixth that was merged with your /etc/elasticsearch/elasticsearch.yml file.

  • sg_config.yml
  • sg_internal_users.yml
  • sg_action_groups.yml
  • sg_roles.yml
  • sg_roles_mapping.yml
  • elasticsearch.yml.example

The sg_config.yml file defines where the system learns how to authenticate users. I have created a minimal file that only works for the management cert and locally defined users.

The sg_internal_users.yml file is the equivalent of the /etc/passwd and /etc/group files of Unix. You add a username, you hash their password, and they need at least one item in their roles: subsection. The roles: are how authorization for various activities are conferred on the user.

Defining roles is more complex than it needs to be for a standalone system, a result of the broad support for different enterprise systems. There are three files involved in this. The first are the action groups. Here’s an example:

An action group contains one or more permissions, and optionally it can reference one or more other action groups.

The next thing to look at is the sg_roles.yml file. This where those action groups get assigned to a particular Search Guard role. You’ll see the action groups referenced in ALLCAPS, and there are additional very specific authorizations that some types of users need. These are applied to the cluster: and to groups of indices:. They also apply to tenants:, which is an enterprise feature that won’t appear in the demo install. A tenant is somewhat analogous to a Unix group, it’s a container of indices and related things (like Visualizations).

The first role is the internal stuff for the Kibana server itself. The second is a modification I made in order to permit read only access to one specific index (usertest) and a cluster of indices with similar names (usersby*). The blep tenant does do anything yet, we’re just starting to explore that feature.

Finally, all of this stuff has to be turned into effects within Elasticsearch itself. A user with a Search Guard role that has permissions, either directly, or via an action group bundle, has to be connected to an Elasticsearch role. The sg_roles_mapping.yml file does this.

One of the biggest challenges in this has been the lack of visual aides. Search Guard has no diagrams to go with their documentation and Elasticsearch is nearly as bare. I happened to find this one while working on this article. It would have been an immense help to have seen it six months ago.

You will also need to install Kibana, using these instructions. The following are the essential lines from /etc/kibana/kibana.yml. false
xpack.spaces.enabled: false
elasticsearch.username: "kibanaserver"
elasticsearch.password: "kibanaserver"
elasticsearch.requestHeadersWhitelist: [ "Authorization", "sgtenant" ]
elasticsearch.ssl.verificationMode: none
elasticsearch.url: "https://localhost:9200"

Possible Pitfalls

Our first production Elasticsearch cluster was 5.6.12, then we upgraded to 6.5.1, and finally settled on 6.5.4. We’re on our third set of hardware for the cluster, and along the way there have been a number of problems. The following are things to always check with a Search Guard install:

  • Which user owns /usr/share/elasticsearch?
  • Which user owns /etc/elasticsearch?
  • Which user owns /usr/share/kibana?
  • Which user owns /etc/kibana?
  • Where is the Elasticsearch data stored?
  • Which user owns the data directory?
  • Are the perms in /etc directories all set to 600?

Just in case it wasn’t abundantly clear, DO NOT TRY THIS WITH PRODUCTION DATA. You will certainly hit some sort of snag your first time out and it’s quite possible to leave yourself with a system you can not access. Make a virtual machine, use zfs for the spaces Elasticsearch uses, and take a snapshot at each milestone, so you aren’t rebuilding from scratch. Not doing these things was the price we paid for any fluency we might display with the platform now.

Having come this far, it would appear the next natural step would be doing the work required to build a self-signed Certificate Authority suitable for use with a small cluster of machines. That will have to wait until next week, in the meantime this post provides plenty of guidance for your experiments.