The early version of the Netwar System ran with a handful of Twitter accounts and a flat file system. Today we use a 64 gig Xeon with 48 Twitter accounts for internal studies and a trio of 16 gig VPSes for botnetsu.press, our semi-public service. The requirements for an R&D system exceed a virtual machine, unless you’ve got a Xeon grade desktop.
We happen to have a Dell m4600 laptop and eight unallocated Twitter accounts, so this has been built out as an R&D environment. The system has a four core i7, 16 gig of ram, and in addition to the system volume there is a 60 gig msata SSD and a 500 gig spindle in the disk carrier that fits in the CD/DVD bay. This is essentially a miniature of our larger Xeon system.
Disk performance has always been our problem with Elasticsearch, so the msata drive was split into cache and log space for a 465G ZFS partition.
Disk /dev/sdb: 55.9 G
Disk /dev/sdc: 465.8 G
The final configuration looks like this:
capacity operations bandwidth
poolalloc free readwrite readwrite
zorp 612K 464G . 0 0 3.44K 5.37K
sdc1 612K 464G . 0 0 1.89K 4.32K
sdb2 4K . 27.7G . 0 0 1.55K 1.05K
sdb1 40.5 . K28. 0G 0 0150 96
The following software is needed:
- Elasticsearch/Kibana (use Wazuh instructions, but stick to 5.6.12)
- Redis (just apt install redis-server)
Once you’ve got them all installed you’ll see the following ports in use.
A few caveats, first be sure these are the final lines in /etc/security/limits.conf or you will quickly learn to hate Elasticsearch.
elasticsearch – nofile 300000
root – nofile 300000
Next, examine the configurations for Elasticsearch and Kibana in /etc. You’ll want to ensure there is more than the default 2 gig for the JVM and modify the Kibana config so you can reach port 5601 from elsewhere.
We have come to the point where we must release configuration advice and some Python code in order for others to learn to use the system. We’re going to trust that the requisite system integration capabilities, analytical tradecraft, and team management skills are going to limit the number of players who can actually do this. There isn’t a specific Github repository for this just yet, but there will be in the coming days.