SIEM Log Noise Analysis Skill

log-noise.md

argument-hint	[index=<index>] [hours=24] [top=5]
allowed-tools	Bash Read Write

Log Noise Analysis — Find and Tune the Noisiest Events

Drill into your SIEM to find the highest-volume events, identify what's generating the noise, and produce tuning recommendations with concrete events/day and events/sec savings.

Input

$ARGUMENTS optionally contains:

• index=<name> to scope to a specific Splunk index (default: *)
• hours=N for the analysis window (default: 24)
• top=N how many noisy event IDs to drill into per sourcetype (default: 5)

Configuration

SPLUNK_URL  -- from $SPLUNK_URL
SPLUNK_TOKEN -- from $SPLUNK_TOKEN
VAULT=Obsidian Vault
HOURS=24  (from args or default)
TOP=5     (from args or default)

Verify $SPLUNK_URL and $SPLUNK_TOKEN exist before proceeding.

Workflow

This is an adaptive multi-step process — each step's queries depend on the previous step's results.

Step 1 — Sourcetype volume

Get total event counts per sourcetype for the analysis window:

curl -s -k \
  -H "Authorization: Bearer $SPLUNK_TOKEN" \
  -d "search=search index=<INDEX> earliest=-<HOURS>h latest=now | stats count by sourcetype source | sort -count" \
  -d "output_mode=json" \
  -d "exec_mode=oneshot" \
  -d "earliest_time=-<HOURS>h" \
  -d "latest_time=now" \
  -d "count=0" \
  "$SPLUNK_URL/services/search/jobs/export" > /tmp/noise_sourcetypes.json

Parse NDJSON (take final result set — last occurrence per unique key due to streaming). Calculate:

• Events per hour = count / HOURS
• Events per second = count / (HOURS * 3600)
• % of total

Note the top sourcetypes by volume. Proceed with all sourcetypes that have meaningful volume (>100 events).

Step 2 — Event ID breakdown per sourcetype

For each significant sourcetype from Step 1, run an event ID query. Use the right field based on the sourcetype:

Sourcetype Pattern	Event ID Field	Query Pattern
XmlWinEventLog	EventCode + source	stats count by source EventCode
WinEventLog	EventCode + source	stats count by source EventCode
aws:cloudtrail	eventName	stats count by eventName
mscs:azure:* / azure:*	operationName	stats count by operationName
audit*	type	stats count by type
Other	Try EventCode, fall back to action, signature, event_type	stats count by EventCode

Example for Windows:

curl -s -k \
  -H "Authorization: Bearer $SPLUNK_TOKEN" \
  -d "search=search index=<INDEX> sourcetype=XmlWinEventLog earliest=-<HOURS>h latest=now | stats count by source EventCode | sort -count" \
  -d "output_mode=json" \
  -d "exec_mode=oneshot" \
  -d "earliest_time=-<HOURS>h" \
  -d "latest_time=now" \
  -d "count=0" \
  "$SPLUNK_URL/services/search/jobs/export" > /tmp/noise_winevt_ids.json

Run applicable queries in parallel for all significant sourcetypes.

Parse results. Identify the top N noisiest event IDs per sourcetype.

Step 3 — Drill into noisy event IDs

For each of the top noisy event IDs, run a drill-down query using the appropriate fields. This identifies what is generating the noise.

Use this field mapping for drill-downs:

Sysmon (source contains "Sysmon")

EventCode	Name	Drill-down SPL
1	Process Creation	stats count by Image CommandLine User | sort -count | head 20
3	Network Connection	stats count by Image DestinationIp DestinationPort | sort -count | head 20
7	Image Load	stats count by Image ImageLoaded | sort -count | head 20
10	Process Access	stats count by SourceImage TargetImage | sort -count | head 20
11	File Create	stats count by Image TargetFilename | sort -count | head 20
12,13,14	Registry	stats count by Image EventCode TargetObject | sort -count | head 20
22	DNS Query	stats count by Image QueryName | sort -count | head 20
Other	Generic	stats count by Image | sort -count | head 20

Windows Security (source contains "Security")

EventCode	Name	Drill-down SPL
4624	Logon Success	stats count by TargetUserName LogonType IpAddress | sort -count | head 20
4625	Logon Failure	stats count by TargetUserName LogonType IpAddress | sort -count | head 20
4672	Special Privileges	stats count by SubjectUserName | sort -count | head 20
4688	Process Creation	stats count by NewProcessName CommandLine SubjectUserName | sort -count | head 20
5145	Share Access	stats count by ShareName RelativeTargetName SubjectUserName | sort -count | head 20
5379	Credential Manager	stats count by SubjectUserName TargetName | sort -count | head 20
Other	Generic	stats count by SubjectUserName | sort -count | head 20

CloudTrail (sourcetype aws:cloudtrail)

All eventNames use the same drill-down:

stats count by userIdentity.arn sourceIPAddress userAgent | sort -count | head 20

Azure (sourcetype mscs:azure:* or azure:*)

stats count by identity callerIpAddress | sort -count | head 20

Linux Audit (sourcetype audit*)

stats count by exe key | sort -count | head 20

Build the full drill-down query like:

curl -s -k \
  -H "Authorization: Bearer $SPLUNK_TOKEN" \
  -d "search=search index=<INDEX> sourcetype=XmlWinEventLog source=\"*Sysmon*\" EventCode=1 earliest=-<HOURS>h latest=now | stats count by Image CommandLine User | sort -count | head 20" \
  -d "output_mode=json" \
  -d "exec_mode=oneshot" \
  -d "earliest_time=-<HOURS>h" \
  -d "latest_time=now" \
  -d "count=0" \
  "$SPLUNK_URL/services/search/jobs/export" > /tmp/noise_drill_sysmon_1.json

Run drill-down queries in parallel where possible (they're independent).

Step 4 — Calculate tuning recommendations

For each drill-down result, calculate:

• Events/day = count * (24 / HOURS)
• Events/sec = count / (HOURS * 3600)
• % of parent = count / parent_event_id_total * 100
• % of total = count / total_events * 100

A tuning candidate is any single (Image + CommandLine) or (user + event) combination that represents >5% of its parent event ID's volume.

For each candidate, produce a recommendation:

• Filter description: what to exclude (e.g., Image="svchost.exe" AND CommandLine="*netsvcs*")
• Projected savings: events/day and events/sec removed
• Risk note: flag if the exclusion could hide real threats. For example:
  • Excluding svchost.exe from Sysmon EC 1 = HIGH RISK (common LOLBin)
  • Excluding SYSTEM logons from 4624 = LOW RISK (machine noise)
  • Excluding scheduled task runs = MEDIUM RISK (persistence vector but noisy)
  • Excluding Sysmon EC 10 for specific SourceImage→TargetImage = LOW RISK if it's a known security tool

Step 5 — Write Obsidian note

Note name: Log Noise — <YYYY-MM-DD>

obsidian create vault="{VAULT}" name="Log Noise — $(date +%Y-%m-%d)" overwrite content="..."

Note structure:

---
scan_date: <YYYY-MM-DD>
analysis_window: <N>h
splunk_index: <INDEX or "all">
total_events: <N>
events_per_second: <N>
sourcetypes_analyzed: <N>
tuning_candidates: <N>
potential_savings_pct: <N>%
---

# Log Noise Analysis — <YYYY-MM-DD>

> [!info] Analysis Summary
> **Index:** `<INDEX>`  |  **Window:** <N> hours  |  **Total events:** <N>
> **Event rate:** <N>/sec  |  **Sourcetypes:** <N>
> **Tuning candidates found:** <N>  |  **Potential savings:** <N>% of volume

## Volume by Sourcetype

(TUI bar chart — same style as attack-coverage)

​```
Sourcetype Volume (<N>h)
─────────────────────────────────────────────────────────────────────
XmlWinEventLog    ████████████████████  68,712  (47%)  2,863/hr   0.8/sec
aws:cloudtrail    ███████████░░░░░░░░░  55,579  (38%)  2,316/hr   0.6/sec
mscs:azure:evhub  ██░░░░░░░░░░░░░░░░░░   4,626   (3%)    193/hr   0.1/sec
audit-5           ░░░░░░░░░░░░░░░░░░░░     910   (1%)     38/hr   0.0/sec
​```

## Noisiest Events

(TUI bar chart per sourcetype, then drill-down)

### <Source Name> (e.g., Sysmon)

​```
Event ID Breakdown
─────────────────────────────────────────────────────
EC 1  Process Creation     ████████████████████  57,553  (84%)  2,398/hr
EC 10 Process Access       ███░░░░░░░░░░░░░░░░░   4,255   (6%)    177/hr
EC 13 Registry Value Set   ██░░░░░░░░░░░░░░░░░░   2,113   (3%)     88/hr
EC 7  Image Loaded         █░░░░░░░░░░░░░░░░░░░   1,466   (2%)     61/hr
...
​```

#### EC 1 — Process Creation (57,553 events)

Top noise generators:

| # | Image | CommandLine (truncated) | User | Count | %  | /day | /sec |
|---|-------|------------------------|------|-------|----|------|------|
| 1 | svchost.exe | -k netsvcs -p -s ... | SYSTEM | 12,345 | 21% | 12,345 | 0.14 |
| 2 | SearchProtocolHost.exe | /... | SYSTEM | 8,901 | 15% | 8,901 | 0.10 |
| 3 | ... | ... | ... | ... | ... | ... | ... |

(Repeat for each drilled-into event ID)

## Tuning Recommendations

Sorted by savings (highest first). Only includes candidates >5% of parent volume.

| # | Source | Event | Suggested Filter | Risk | Events/Day Saved | Events/Sec Saved | % Reduction |
|---|--------|-------|-----------------|------|-----------------|-----------------|-------------|
| 1 | Sysmon | EC 1 | Image="SearchProtocolHost.exe" | LOW | -8,901 | -0.10 | 13% |
| 2 | Sysmon | EC 10 | SourceImage="*MsMpEng*" AND TargetImage="*lsass*" | LOW | -3,200 | -0.04 | 5% |
| 3 | Security | 4672 | SubjectUserName="SYSTEM" | LOW | -5,500 | -0.06 | 8% |
| 4 | Sysmon | EC 1 | Image="svchost.exe" | ⚠ HIGH | -12,345 | -0.14 | 18% |

> [!warning] High-Risk Exclusions
> Items marked ⚠ HIGH involve processes commonly abused by attackers (LOLBins, admin tools).
> Consider allowlisting specific CommandLine patterns instead of excluding the Image entirely.

## Cumulative Savings

​```
If all LOW-risk tuning applied:
  Before: 145,827 events/day  (1.7/sec)
  After:  102,345 events/day  (1.2/sec)
  Saved:   43,482 events/day  (0.5/sec)  — 30% reduction

If all tuning applied (including HIGH-risk):
  Before: 145,827 events/day  (1.7/sec)
  After:   78,123 events/day  (0.9/sec)
  Saved:   67,704 events/day  (0.8/sec)  — 46% reduction
​```

Step 6 — Summary

Report to the user:

• Total events in window and event rate
• Top 3 noisiest event IDs with their generators
• Number of tuning candidates found
• Projected savings if low-risk tuning applied
• Confirm note written to Obsidian