rockfish hunt

Run graph-based behavioral threat detection on Parquet flow data.

Overview

The hunt engine builds a communication graph from network flow data and applies configurable detection algorithms to identify threats beyond signature matching — C2 beaconing, lateral movement, data exfiltration, and more.

Findings are scored with anomaly detection models (HBOS or Isolation Forest), assigned severity levels, and mapped to MITRE ATT&CK tactics.

Usage

rockfish hunt [OPTIONS]

Detection Types

Select Specific Detections

rockfish hunt -d /data --sensor my-sensor --hive \
  --detections beaconing,lateral,fanout

AI Egress Detection (opt-in)

Flags connections to AI/LLM services by matching a configurable domain list against already-ingested TLS SNI, DNS query names, and HTTP hostnames/URLs (e.g. /v1/chat/completions). It’s a pure query over data you already store — retroactive, no new capture — and is off by default. Enable it in the hunt: section of your config:

hunt:
  # Detect egress to AI/LLM services (Shadow AI visibility). Off by default.
  ai_egress_enabled: true
  # Optional: override the built-in provider list (SQL LIKE patterns).
  ai_egress_domains:
    - "%api.openai.com"
    - "%anthropic.com"
    - "%generativelanguage.googleapis.com"
    - "%openai.azure.com"
    - "%bedrock-runtime.%amazonaws.com"
    # ... add your own

When enabled, matches surface as AI Egress findings (MITRE Exfiltration), one per host + service, alongside the other detections. This gives visibility into known AI SaaS usage; self-hosted-over-TLS and unknown providers are not covered by the domain-match layer.

AI Stream Detection (opt-in, plugin-gated)

Catches what the domain list can’t — self-hosted, unknown-provider, and even ECH/DoH LLM traffic — by scoring the shape of token streaming rather than the destination. It’s provider- and hostname-independent, keyed on the payload_entropy plugin’s SPLT signal (Sequence of Packet Lengths and Times, with per-packet direction). Each flow gets a 0–1 score from four signals — high entropy_toclient (encrypted), server-heavy direction, small median chunk length (token-sized), and a “typing”-band median inter-arrival time — and flows at or above ai_stream_min_score become AI Stream findings.

Data-strict: requires the payload_entropy toolkit plugin (the same one behind Encrypted Traffic Analytics). Without it the detection simply logs and skips. Off by default:

hunt:
  ai_stream_enabled: true
  # Tunables (defaults shown):
  ai_stream_min_packets: 8            # min SPLT packets sampled per flow
  ai_stream_min_entropy: 7.0          # entropy_toclient floor (encrypted)
  ai_stream_min_toclient_ratio: 0.6   # server-heavy direction
  ai_stream_max_chunk_bytes: 400      # median toclient chunk (token-sized)
  ai_stream_min_iat_us: 1000          # "typing"-cadence IAT band (µs)
  ai_stream_max_iat_us: 2000000
  ai_stream_min_score: 0.75           # 0..1 score threshold
  ai_stream_ports: [443]              # empty = any port

Pair it with ai_egress for both identity (which named provider) and shape (unnamed but streaming like an LLM).

AI Stream — ML (opt-in, self-supervised)

A machine-learned refinement of ai_stream that needs no labeled data from you. It uses ai_egress as an automatic labeler: flows whose community_id matches a configured AI provider domain are confirmed positive LLM-stream examples. From those it builds a per-sensor one-class profile (per-feature mean/std over the SPLT feature vector) and flags any flow without a domain match whose standardized distance to that profile is within the ai_stream_ml_inlier_pctl percentile of the confirmed streams’ own distances — i.e. self-hosted / unknown / ECH LLM egress that looks like your real AI traffic. Because it’s calibrated to your network, it’s more precise than the fixed-threshold ai_stream.

Requires ai_egress’s domain list, the payload_entropy plugin, and enough observed AI usage to learn from. On cold-start (fewer than ai_stream_ml_min_train, default 30, confirmed streams) it automatically falls back to the ai_stream heuristic — so it’s never worse than v1.

ai_stream and ai_stream_ml are mutually exclusive — enable at most one. Setting both ai_stream_enabled: true and ai_stream_ml_enabled: true is a fatal config error at startup. Use ai_stream for the fixed-threshold shape score, or ai_stream_ml for the self-supervised model (recommended once the payload_entropy plugin and some AI traffic are present).

hunt:
  ai_stream_ml_enabled: true
  ai_stream_ml_min_train: 30      # min confirmed AI streams before ML kicks in
  ai_stream_ml_inlier_pctl: 0.9   # flag candidates within this percentile of positives

Output

Parquet (default)

Findings are written to {data-dir}/{sensor}/hunt/*.parquet for ingestion into the report.

Stdout

# JSON output
rockfish hunt -d /data --sensor my-sensor --stdout

# Pretty-printed JSON
rockfish hunt -d /data --sensor my-sensor --stdout --pretty

# Table format
rockfish hunt -d /data --sensor my-sensor --stdout --format table

Severity Filtering

# Only high and critical findings
rockfish hunt -d /data --sensor my-sensor --min-severity high

Tuning

Anomaly Scoring Algorithms

All detections produce findings that are scored using one of two anomaly detection methods. When a detection has 5 or more candidate groups, statistical scoring is applied; otherwise, fixed severity thresholds are used.

HBOS (Histogram-Based Outlier Scoring)

The default scoring method. HBOS builds equal-width histograms for each feature dimension, then scores each observation based on how rare its bin is.

How it works:

1. For each feature (e.g., coefficient of variation, connection count, byte ratio), divide the observed range into 10 equal-width bins
2. Count the number of observations in each bin to compute density: density = count_in_bin / total_observations
3. Apply a floor to prevent log(0): density = max(density, 0.5 / total)
4. Compute per-feature score: score = -log10(density) — rarer bins produce higher scores
5. Sum all feature scores for the final anomaly score

Properties:

• Assumes feature independence (no cross-feature correlations)
• O(n) time complexity — fast, single-pass histogram construction
• Supports feature inversion for metrics where lower values are more suspicious (e.g., beacon CV where 0.01 is more suspicious than 0.5)

Isolation Forest (iForest)

An ensemble method that isolates anomalies using random partitioning trees.

How it works:

1. Build 100 random isolation trees, each sampling 256 data points
2. Each tree recursively partitions data by randomly selecting a feature and split value until each point is isolated (or max depth is reached)
3. For each observation, compute the average path length across all trees — anomalies are isolated in fewer splits
4. Convert to anomaly score: score = 2^(-avg_path_length / c(n)) where c(n) is the expected path length for a balanced BST
5. Transform to match HBOS scale: final_score = -log10(1 - raw_score)

Properties:

• Captures cross-feature interactions (unlike HBOS)
• More robust to feature scaling
• Higher computational cost than HBOS
• Deterministic (seed = 42 for reproducibility)

Select scoring method:

rockfish hunt -d /data --sensor my-sensor --scoring-method iforest

Severity Mapping

Anomaly scores are mapped to severity levels using percentile-based thresholds across the entire finding population:

If the maximum score across all findings is below 2.0, severity is capped at Medium to suppress false positives in benign environments.

Detection Algorithm Details

Beaconing

Detects C2 callbacks by measuring the regularity of connection intervals.

1. Group connections by (src_ip, dest_ip, dest_port)
2. Compute inter-arrival time intervals between consecutive connections
3. Calculate the coefficient of variation: CV = stddev / mean
4. A perfect beacon has CV ≈ 0; random traffic has CV ≈ 1.0

Scoring features: CV (inverted), connection count, mean interval, byte consistency (CV of payload sizes)

Lateral Movement

Detects multi-hop attack chains where internal hosts are progressively compromised.

1. Build a temporal adjacency graph: src → [(dest, timestamp)]
2. Identify pivot hosts (both source and destination)
3. For each pivot, look for inbound → outbound sequences within a 1-hour window
4. Extend chains recursively (up to 10 hops)

C2 Fanout

Detects a single external IP receiving connections from many internal hosts (botnet controller pattern).

Port Scan

Detects hosts probing many ports on a target.

1. Count distinct destination ports per (src_ip, dest_ip) pair
2. Detect sequential port runs (e.g., 80-84) and compute sequential_ratio
3. Compute scan rate (ports per second)

Scoring features: unique ports, flow count, sequential ratio, scan rate

Community Detection

Identifies botnet-like clusters using Kosaraju’s Strongly Connected Components algorithm.

1. Build a directed graph from flow data
2. Find SCCs where every node can reach every other node
3. Compute density: edges / (n × (n-1))

DNS Tunneling

Detects data exfiltration encoded in DNS subdomain labels.

1. Pre-filter: average subdomain length must exceed 15 characters
2. Analyze unique subdomain count, TXT record ratio, and query rate per base domain

Scoring features: unique subdomains, avg label length, TXT ratio, query rate

Data Exfiltration

Detects internal hosts uploading disproportionate data volumes to external hosts.

1. Compute byte ratio: bytes_out / (bytes_out + bytes_in) — ratio ≥ 0.8 is suspicious
2. Filter: minimum 10 MB outbound

Scoring features: total bytes out, byte ratio, flow count

New Connection Pair

Detects (src_ip, dest_ip, dest_port) tuples never seen in the 7-day baseline window. Particularly important for OT/IoT networks where traffic is highly deterministic.

Known OT ports (Modbus 502, DNP3 20000, MQTT 1883/8883, BACnet 47808, EtherNet/IP 44818, S7comm 102, OPC UA 4840, IEC 104 2404) trigger elevated severity.

Polling Disruption

Detects when previously periodic communication becomes irregular or stops entirely. Designed for SCADA/OT environments.

1. Identify connections periodic in baseline (CV ≤ 0.3)
2. Detect disruption: either stopped (0 recent flows) or irregular (recent CV > 0.8)

Baseline Deviation

Detects significant deviations from historical traffic patterns.

1. Compare recent (1 hour) vs baseline (7 days) for same connection tuples
2. Compute ratios: flow_ratio = recent / baseline, bytes_ratio = recent / baseline
3. Flag new protocols not seen in baseline

Scoring features: flow count ratio, bytes ratio, new protocol count

Continuous Mode

# Run every hour (default)
rockfish hunt -d /data --sensor my-sensor --hive --continuous

# Run every 15 minutes
rockfish hunt -d /data --sensor my-sensor --hive \
  --continuous --interval-minutes 15

Time Window

rockfish hunt -d /data --sensor my-sensor -t "24 hours"  # default
rockfish hunt -d /data --sensor my-sensor -t "7 days"
rockfish hunt -d /data --sensor my-sensor -t "1 hour"

Examples

# Standard 24-hour threat hunt
rockfish hunt -d /data/rockfish --sensor prod-01 --hive -t "24 hours"

# Continuous with high severity filter
rockfish hunt -d /data --sensor prod-01 --hive \
  --continuous --interval-minutes 30 --min-severity high

# Beaconing with custom thresholds
rockfish hunt -d /data --sensor my-sensor \
  --detections beaconing --min-beacon-connections 50 --max-beacon-cv 0.15