Skip to content
Lesson 7 of 8

OSINT Tools and Automation

5 min read

From Manual Search to Automation

The techniques covered so far — WHOIS, DNS, dorking, username correlation — are perfectly executable by hand, but at scale they become unfeasible. Investigating a single target can involve dozens of queries to different sources, and repeating that process for multiple targets manually consumes enormous time and is error-prone. This is where automation tools and frameworks come in, orchestrating many sources into reproducible workflows.

These tools do not replace the analyst's judgment; they amplify it. They automate collection and correlation, but interpretation, verification, and decisions remain human. A common beginner mistake is to blindly trust a tool's output without validating it. Automation is a powerful starting point, not a conclusion.

A caveat about scope is also in order: many of these tools can generate active traffic toward the target (DNS queries, subdomain resolution). In passive mode they work only with third-party sources, but in active mode they touch the target's infrastructure, which must be done exclusively within an authorized engagement.

theHarvester, Amass, and Command-Line Tools

theHarvester is a classic and lightweight tool focused on gathering emails, subdomains, hosts, and employee names from public sources. It is ideal for a quick initial reconnaissance phase:

# Broad passive reconnaissance of an organization
theHarvester -d example.com -b bing,crtsh,duckduckgo,otx

# Dump the result to a file for later analysis
theHarvester -d example.com -b all -f result_example

Amass is the standard for subdomain and attack-surface mapping, with clearly separated passive and active modes:

# PASSIVE enumeration (third-party sources only, does not touch the target)
amass enum -passive -d example.com

# ACTIVE enumeration with resolution (generates traffic: only with authorization)
amass enum -active -d example.com -brute

Their simplicity and speed make these tools a common starting point. The difference between -passive and -active is not cosmetic: the former is quiet and low-risk; the latter leaves a trace and is only appropriate within the permitted scope.

SpiderFoot and recon-ng

SpiderFoot takes automation much further. It is a framework with over a hundred modules that integrate with dozens of sources (passive DNS, Shodan, Have I Been Pwned, certificate logs, and many more). It runs as a web application or via CLI, launches a scan on a target, and automatically correlates the findings:

# CLI scan limited to passive modules (does not touch the target)
sf.py -s example.com -t DOMAIN_NAME -m sfp_dnsresolve,sfp_crt,sfp_hunter

It allows configuring passive mode so as not to touch the target, and it is one of the most complete tools for automating an investigation end to end.

recon-ng takes a different philosophy: it offers a modular console interface inspired by Metasploit, with modules for each type of task and a built-in database that stores results:

# Typical flow inside the recon-ng console
[recon-ng][default] > marketplace install all
[recon-ng][default] > modules load recon/domains-hosts/hackertarget
[recon-ng][default] > options set SOURCE example.com
[recon-ng][default] > run

Its design makes it easy to chain modules and build reproducible workflows, ideal for those who prefer granular control. Alongside these, specialized tools like Amass (subdomains), Sherlock (usernames), or Photon (web crawling) round out the investigator's arsenal. And Maltego remains the reference for visualization: its model of "entities" and "transforms" represents relationships as an interactive graph, revealing patterns — an email linking three domains, a person connecting two organizations — that would go unnoticed in a flat list.

Building a Workflow

No tool does everything, so the real art lies in combining them into a coherent flow. A common pattern:

1. Broad passive reconnaissance   → theHarvester / SpiderFoot (passive mode)
2. Dig in by area                 → Amass (subdomains), crt.sh (certificates),
                                     Sherlock (usernames), ExifTool (metadata)
3. Correlate and visualize        → Maltego / entity graph
4. Verify and document            → manual corroboration + knowledge base

You start broad to get an overview, dig in with specialized tools on specific areas, and culminate by visualizing and verifying. Every jump from one phase to the next must record what was queried, when, and with what result.

API Key Management and OPSEC

API key management is an important practical detail. Many sources (Shodan, Censys, Hunter.io, VirusTotal) offer programmatic access via API, often with limited free quotas. Configuring these keys multiplies the tools' reach, but requires caring for their security:

# Load keys from environment variables, NEVER written in code
export SHODAN_API_KEY="..."
export HUNTER_API_KEY="..."

# Make sure your config files do not end up in a repository
echo ".env" >> .gitignore
echo "api_keys.yaml" >> .gitignore

Keys must never be uploaded to public repositories or shared: a leaked key not only consumes your quota, it identifies you. As for operational OPSEC: run the tools from a machine or environment dedicated to the investigation, control network attribution when using active modules, and keep API accounts separate from your personal identity. These principles are developed in Lesson 8.

Documentation and Reproducibility

Above the choice of tools comes documentation. Every automated workflow must record what was queried, when, and with what result, to guarantee the traceability and reproducibility of the investigation. A good practice is to standardize the output format (JSON, for example) and centralize findings in a single knowledge base, so that subsequent analysis relies on clean data, dated and attributed to its source.

Automation Checklist

  • [ ] Scope and authorization confirmed; passive mode by default.
  • [ ] Broad reconnaissance before digging in with specialized tools.
  • [ ] Active mode (Amass -active, resolution, brute force) only within the engagement.
  • [ ] API keys in environment variables; never in the repository.
  • [ ] Tool output verified, not accepted blindly.
  • [ ] Findings centralized with source, date, and confidence for reproducibility.