rockfish ingest

Ingest Suricata EVE JSON logs and write columnar Parquet files.

Overview

The ingest command reads Suricata’s EVE JSON log output — from files, stdin, or a Unix socket — partitions events by type, and writes compressed Parquet files. It supports continuous ingestion via follow mode, automatic S3 upload, and hive-style date partitioning.

Usage

rockfish ingest [OPTIONS]

Input Sources

File Input

# From a file
rockfish ingest -i /var/log/suricata/eve.json

# From stdin
cat eve.json | rockfish ingest -i -

Unix Socket

Connect directly to Suricata’s Unix socket output:

# Stream socket (default)
rockfish ingest --socket /var/run/suricata/eve.sock

# Datagram socket
rockfish ingest --socket /var/run/suricata/eve.sock --socket-type dgram

Follow Mode

Tail a live log file like tail -F, handling log rotation and truncation:

rockfish ingest -i /var/log/suricata/eve.json --follow

Follow mode saves state to a .state file so ingestion can resume after restart. Use --from-beginning to ignore saved state.

Output

Directory Layout

Flat layout (default):

output/my-sensor/
  alert/alert_2026-02-16T14-00-00.parquet
  flow/flow_2026-02-16T14-00-00.parquet
  dns/dns_2026-02-16T14-00-00.parquet

Hive-partitioned layout (--hive):

output/my-sensor/
  alert/year=2026/month=02/day=16/alert_2026-02-16T14-00-00.parquet
  flow/year=2026/month=02/day=16/flow_2026-02-16T14-00-00.parquet

Options

Event Type Filtering

# Only process alerts and flows
rockfish ingest -i eve.json --include alert,flow

# Process everything except stats
rockfish ingest -i eve.json --exclude stats

Supported types: alert, flow, dns, http, tls, fileinfo, anomaly, smtp, ssh, stats, dhcp, mqtt, modbus, dnp3, and more.

S3 Upload

Enable automatic S3 upload after each Parquet flush:

rockfish ingest -i eve.json --s3

Credentials are read from the environment file (/opt/rockfish/etc/rockfish.env):

ROCKFISH_S3_BUCKET=rockfish-data
ROCKFISH_S3_REGION=us-east-1
AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...

The S3 bucket layout mirrors the local directory:

s3://bucket/{sensor}/{event_type}/year=YYYY/month=MM/day=DD/*.parquet

Examples

# Basic file ingestion with hive partitioning
rockfish ingest -i eve.json -o /data/rockfish --sensor prod-01 --hive

# Continuous socket ingestion with S3 upload
rockfish ingest --socket /var/run/suricata/eve.sock \
  -o /data/rockfish --sensor prod-01 --hive --s3

# Follow mode with custom flush interval
rockfish ingest -i /var/log/suricata/eve.json \
  --follow --flush-interval 30 --sensor edge-01

# Ingest only alerts and flows
rockfish ingest -i eve.json --include alert,flow -vv