A Small Development Environment

The early version of the Netwar System ran with a handful of Twitter accounts and a flat file system. Today we use a 64 gig Xeon with 48 Twitter accounts for internal studies and a trio of 16 gig VPSes for botnetsu.press, our semi-public service. The requirements for an R&D system exceed a virtual machine, unless you’ve got a Xeon grade desktop.

We happen to have a Dell m4600 laptop and eight unallocated Twitter accounts, so this has been built out as an R&D environment. The system has a four core i7, 16 gig of ram, and in addition to the system volume there is a 60 gig msata SSD and a 500 gig spindle in the disk carrier that fits in the CD/DVD bay. This is essentially a miniature of our larger Xeon system.

Disk performance has always been our problem with Elasticsearch, so the msata drive was split into cache and log space for a 465G ZFS partition.

Disk /dev/sdb: 55.9 G

/dev/sdb1 28G

/dev/sdb2 27.9G

Disk /dev/sdc: 465.8 G

/dev/sdc1465.8G

 

The final configuration looks like this:

 

               capacity     operations    bandwidth

poolalloc free readwrite readwrite

—————————————-

zorp 612K 464G .   0 0 3.44K 5.37K

sdc1 612K 464G .   0 0 1.89K 4.32K

logs——

sdb2 4K .  27.7G .      0 0 1.55K 1.05K

cache ——

sdb1 40.5 . K28.       0G 0 0150 96

—————————————-

The following software is needed:

Once you’ve got them all installed you’ll see the following ports in use.

Elasticsearch

127.0.0.1:9200

127.0.0.1:9300

Kibana

127.0.0.1:5601

Neo4j

127.0.0.1:7687

127.0.0.1:7473

127.0.0.1:7474

Netdata

127.0.0.1:8125

0.0.0.0:19999

Redis

127.0.0.1:6379

A few caveats, first be sure these are the final lines in /etc/security/limits.conf or you will quickly learn to hate Elasticsearch.

elasticsearch – nofile 300000

root – nofile 300000

Next, examine the configurations for Elasticsearch and Kibana in /etc. You’ll want to ensure there is more than the default 2 gig for the JVM and modify the Kibana config so you can reach port 5601 from elsewhere.

 

We have come to the point where we must release configuration advice and some Python code in order for others to learn to use the system. We’re going to trust that the requisite system integration capabilities, analytical tradecraft, and team management skills are going to limit the number of players who can actually do this. There isn’t a specific Github repository for this just yet, but there will be in the coming days.