Articles here are written by a single author (thus far) but represent the collective views of a loose group of two dozen collaborators, hence the use of the first person plural ‘we’. We take on civil investigations, criminal defense, penetration testing, and geopolitical/cybersecurity threat assessments.
Group members have native fluency in English, French, German, Spanish, Romanian, and we do a fair job with Arabic when it is required. Several of us have corporate or IPS infrastructure backgrounds, and our tools, both chosen and created, reflect this internal integration capability.
This is an inventory of the major systems we currently employ.
The Gephi data visualization package is a piece of free software which permits the handling of networks with tens of thousands of nodes and hundreds of thousands of links. We use this for macro scale examinations of Twitter and some types of financial data, coding import procedures to express complex metrics, when required. When you see colorful network maps, this is likely the source.
The Maltego OSINT link analysis system began life as a penetration tester’s toolkit. It offers a rich set of entities, integration of many free and paid services, and local transform creation. There is a team collaboration feature for paid subscribers and the free Community Edition can read any graph we produce. This is used internally in the same way a financial audit firm would employ a spreadsheet – it is a de facto standard for recording and sharing investigation information.
Sentinel Visualizer is a law enforcement/intel grade link analysis package that supports both geospatial and temporal analysis. This only comes out in the face of paying engagements with large volumes of data, as it has a somewhat intimidating learning curve.
Hunch.ly is a Google Chrome extension that preserves the trail of web sites one visits, applying a standing list of selectors to each page and permitting the addition of investigator’s notes. This tool supports the notion of multiple named investigations, preserves content statically, and can export in a variety of formats. Users are free to follow their noses without the burden of bookmarking and making screen shots while investigating, then later attempting to share their findings in a coherent fashion. The system recently began supporting local Maltego transforms.
The RiskIQ service is an aggregator of a dozen passive threat data repositories in addition to it’s own native tracking of domain registrations, DNS, SSL certificates, and other threat assessment data. The service is delivered as a web based search engine and a companion set of Maltego transforms. This system is a panopticon for bad actor infrastructure which we use daily.
The Elasticsearch platform is used for many things, but for us it is a full text search engine with temporal analysis capabilities that will easily handle tens of thousands of Twitter accounts that have produced tens of millions of tweets. This is a construction kit for us, the right way to collate and correlate the work of teams of Actors, Collectors, and Directors. We currently curate 25 million tweets from ISIS accounts that were collected by TRAC, we support Liberty STRATCOM with collection and analysis, and the botnetsu.press system is in use by activists who track violent right wing groups in the west.
What not to do is just as important as the right stuff. Here are some things we avoided, that we tested but did not implement, or that we have used but later abandoned.
Analyst’s Notebook – nonstarter, 2x the cost of Sentinel Visualizer, and not nearly as open.
Windows – with the exception of Sentinel Visualizer, we don’t have anything that is Windows dependent. Generally speaking, things have to behave for Linux and OSX, with Windows support being nice, but not required.
Splunk – we tried to love it, truly we did. It just didn’t work out.
OSSIM – largely abandonware from what we hear. AlienVault’s Open Threat Exchange is doing fine though, and it all turns up in RiskIQ.
Aeon, Timeline, etc – we always jump at collaborative timeline tools, then later end up sitting back and being annoyed. SaaS solutions are out there, but we have confidentiality concerns that hold us back from using them.
TimeSketch – very cool, an Elastic based tool, but more incident response focused than intel oriented.
SpiderFoot – very cool, but we settled on RiskIQ/Maltego installed on a remotely accessible workstation. This is one we should put back up and use enough to advise others.
There have been many more digressions over the years, these are some of the more formative ones.