Antonlovesdnb/siem-check.md Secret

## siem-check.md

      
    Raw
  

              siem-check.md
            
          
    argument-hint
    [hours=24]
  
  
    allowed-tools
    
  
  Bash
  Read
  Write
  
  
SIEM Health Check — Adaptive Diagnostic with Cross-Signal Correlation

Run a multi-stage health assessment of your Splunk environment: discover silent hosts, ingestion delays, data model gaps, volume anomalies, and internal errors — then correlate findings into a prioritized risk briefing written to Obsidian.
Input

$ARGUMENTS optionally contains:

hours=N for the analysis window (default: 24)

Configuration

SPLUNK_URL  -- from $SPLUNK_URL
SPLUNK_TOKEN -- from $SPLUNK_TOKEN
VAULT=Obsidian Vault
HOURS=24  (from args or default)
INDEX=*    (always — health check scans all indexes)

Verify $SPLUNK_URL and $SPLUNK_TOKEN exist before proceeding.
Workflow

This is an adaptive multi-step process — early results determine what to investigate deeper. Claude correlates findings across all checks to produce a unified risk assessment.
Step 1 — Parallel baseline queries

Run all 5 queries in parallel (they are independent):
1a. Host inventory with volume stats

Get every host that has sent data, with event counts and timing:
curl -s -k \
  -H "Authorization: Bearer $SPLUNK_TOKEN" \
  -d "search=search index=* earliest=-<HOURS>h latest=now | stats count latest(_time) as last_event earliest(_time) as first_event by host | eval hours_since_last=round((now()-last_event)/3600,1) | sort -count" \
  -d "output_mode=json" \
  -d "exec_mode=oneshot" \
  -d "earliest_time=-<HOURS>h" \
  -d "latest_time=now" \
  -d "count=0" \
  "$SPLUNK_URL/services/search/jobs/export" > /tmp/health_hosts.json
1b. Host volume compared to previous period (anomaly detection)

Compare current period vs the prior period of equal length to find volume deviations:
curl -s -k \
  -H "Authorization: Bearer $SPLUNK_TOKEN" \
  -d "search=search index=* earliest=-<HOURS*2>h latest=now | eval period=if(_time>relative_time(now(),\"-<HOURS>h\"),\"current\",\"previous\") | stats count by host period | xyseries host period count | eval change_pct=round((current-previous)/previous*100,1) | where isnotnull(previous) | sort change_pct" \
  -d "output_mode=json" \
  -d "exec_mode=oneshot" \
  -d "earliest_time=-<HOURS*2>h" \
  -d "latest_time=now" \
  -d "count=0" \
  "$SPLUNK_URL/services/search/jobs/export" > /tmp/health_volume_change.json
Replace <HOURS*2> with double the HOURS value (e.g., if HOURS=24, use 48h).
1c. Ingestion delay — index time vs event time

Calculate the lag between when events occurred and when Splunk indexed them:
curl -s -k \
  -H "Authorization: Bearer $SPLUNK_TOKEN" \
  -d "search=search index=* earliest=-<HOURS>h latest=now | eval delay=(_indextime-_time) | stats avg(delay) as avg_delay max(delay) as max_delay perc95(delay) as p95_delay median(delay) as median_delay count by sourcetype | eval avg_delay=round(avg_delay,0) | eval max_delay=round(max_delay,0) | eval p95_delay=round(p95_delay,0) | eval median_delay=round(median_delay,0) | sort -avg_delay" \
  -d "output_mode=json" \
  -d "exec_mode=oneshot" \
  -d "earliest_time=-<HOURS>h" \
  -d "latest_time=now" \
  -d "count=0" \
  "$SPLUNK_URL/services/search/jobs/export" > /tmp/health_delay.json
1d. Data model acceleration status

Check which data models exist and their acceleration state:
curl -s -k \
  -H "Authorization: Bearer $SPLUNK_TOKEN" \
  -d "search=| rest /services/data/models | fields title acceleration acceleration.earliest_time eai:acl.app | rename acceleration as is_accelerated acceleration.earliest_time as accel_window eai:acl.app as app" \
  -d "output_mode=json" \
  -d "exec_mode=oneshot" \
  -d "count=0" \
  "$SPLUNK_URL/services/search/jobs/export" > /tmp/health_datamodels.json
1e. Splunk internal errors and warnings

Pull recent internal log errors — forwarder issues, parsing failures, queue warnings:
curl -s -k \
  -H "Authorization: Bearer $SPLUNK_TOKEN" \
  -d "search=search index=_internal earliest=-<HOURS>h latest=now (log_level=ERROR OR log_level=WARN) | stats count by component log_level message | sort -count | head 50" \
  -d "output_mode=json" \
  -d "exec_mode=oneshot" \
  -d "earliest_time=-<HOURS>h" \
  -d "latest_time=now" \
  -d "count=0" \
  "$SPLUNK_URL/services/search/jobs/export" > /tmp/health_errors.json
Step 2 — Parse and analyze results

Read all 5 NDJSON files. Parse each — take the final result set per unique key.
From host inventory (1a):

Identify hosts where hours_since_last > HOURS * 0.5 → potential silent hosts
Identify hosts where hours_since_last >= HOURS → fully silent (no data in entire window)
Note total host count and total event volume

From volume comparison (1b):

Flag hosts where change_pct < -50% → significant volume drop
Flag hosts where change_pct > 200% → unusual volume spike
Calculate the average deviation across all hosts

From ingestion delay (1c):

Flag sourcetypes where p95_delay > 300 seconds (5 min) → delayed ingestion
Flag sourcetypes where max_delay > 3600 seconds (1 hour) → severely delayed
Identify cloud sourcetypes (cloudtrail, azure, gcp) — higher delay thresholds are expected
Calculate overall average delay

From data models (1d):

Identify CIM-relevant data models that are NOT accelerated (key ones: Network_Traffic, Endpoint, Authentication, Change, Web, Email, Intrusion_Detection, Malware)
Note the acceleration window for models that ARE accelerated

From internal errors (1e):

Group errors by category:

Forwarder issues: TcpInputProc, TailingProcessor, AggregatorMiningProcessor
Parsing/indexing: LineBreakingProcessor, DateParserVerbose, AggregatorMiningProcessor
Queue pressure: blocked queues, queue full, throttling
License: LicenseUsage, license_usage
App errors: any app-specific errors


Rank by frequency and severity

Step 3 — Adaptive deep-dive (Claude decides what to investigate)

Based on Step 2 findings, run targeted follow-up queries for the most concerning issues. This is where Claude adds value — deciding what matters and what to drill into.
If silent hosts found: Query their last known sourcetypes to understand what stopped:
curl -s -k \
  -H "Authorization: Bearer $SPLUNK_TOKEN" \
  -d "search=search index=* host=\"<SILENT_HOST>\" earliest=-7d latest=now | stats count latest(_time) as last_event by sourcetype | eval hours_ago=round((now()-last_event)/3600,1) | sort -last_event" \
  -d "output_mode=json" \
  -d "exec_mode=oneshot" \
  -d "count=0" \
  "$SPLUNK_URL/services/search/jobs/export" > /tmp/health_silent_<HOST>.json
If severe ingestion delays found: Check if delay is consistent or spiking:
curl -s -k \
  -H "Authorization: Bearer $SPLUNK_TOKEN" \
  -d "search=search index=* sourcetype=<DELAYED_ST> earliest=-<HOURS>h latest=now | eval delay=(_indextime-_time) | timechart span=1h avg(delay) as avg_delay max(delay) as max_delay" \
  -d "output_mode=json" \
  -d "exec_mode=oneshot" \
  -d "earliest_time=-<HOURS>h" \
  -d "latest_time=now" \
  -d "count=0" \
  "$SPLUNK_URL/services/search/jobs/export" > /tmp/health_delay_trend.json
If queue errors found: Check current queue depth:
curl -s -k \
  -H "Authorization: Bearer $SPLUNK_TOKEN" \
  -d "search=search index=_internal earliest=-1h latest=now source=*metrics.log group=queue | stats latest(current_size_kb) as current_kb latest(max_size_kb) as max_kb by name | eval pct_full=round(current_kb/max_kb*100,1)" \
  -d "output_mode=json" \
  -d "exec_mode=oneshot" \
  -d "count=0" \
  "$SPLUNK_URL/services/search/jobs/export" > /tmp/health_queues.json
Run applicable follow-up queries in parallel.
Step 4 — Cross-signal correlation

This is the key LLM step. Correlate findings across all checks to identify compound risks that no single query would surface:
Correlation patterns to look for:

Silent host + volume anomaly → host may have stopped forwarding
Silent host + forwarder error in _internal → confirms forwarding issue, include the error
Delayed ingestion + queue pressure → ingestion pipeline is backed up
Un-accelerated data model + high volume sourcetype → CIM searches will be slow or fail
Volume spike + new sourcetype → possible misconfiguration or new data onboarding
Multiple hosts with similar volume drops → systemic issue (network, forwarder deployment, etc.)

For each correlated finding, assess:

Impact: What detections or visibility are affected?
Urgency: Is this active data loss or a degraded state?
Root cause hypothesis: What's the most likely explanation?
Remediation: Specific action to fix it

Risk levels:

CRITICAL: Active data loss, silent critical assets (DCs, security appliances), or complete pipeline failure
HIGH: Significant ingestion delays (>15 min for security data), multiple hosts silent, CIM data models unaccelerated for key security data
MEDIUM: Volume anomalies >50%, non-critical data model gaps, intermittent errors
LOW: Minor delays, informational warnings, cosmetic issues

Step 5 — Write Obsidian note

Note name: SIEM Health — <YYYY-MM-DD>
obsidian create vault="{VAULT}" name="SIEM Health — $(date +%Y-%m-%d)" overwrite content="..."
Note structure:
---
scan_date: <YYYY-MM-DD>
analysis_window: <N>h
splunk_index: "all"
total_hosts: <N>
total_events: <N>
events_per_second: <N>
critical_findings: <N>
high_findings: <N>
medium_findings: <N>
low_findings: <N>
---

# SIEM Health Check — <YYYY-MM-DD>

> [!<callout>] Health Summary
> **Index:** `*`  |  **Window:** <N>h  |  **Hosts:** <N>  |  **Events:** <N>
> **Event rate:** <N>/sec  |  **Avg ingestion delay:** <N>s
> **Findings:** <N> critical, <N> high, <N> medium, <N> low

## Findings

Sorted by severity (critical first), then by impact.

### CRITICAL

> [!failure] <Finding title>
> **Impact:** <what's affected>
> **Evidence:** <specific data from queries>
> **Correlation:** <how multiple signals connect — this is the LLM value-add>
> **Remediation:** <specific fix>

(Repeat for each critical finding. Use `> [!failure]` for critical, `> [!warning]` for high, `> [!caution]` for medium, `> [!info]` for low.)

### HIGH

> [!warning] <Finding title>
> ...

### MEDIUM / LOW

(Same pattern, use appropriate callouts)

## Host Status

Host Status (h)
─────────────────────────────────────────────────────────────────────
     ████████████████████  145,827  (32%)   Active    0.1h ago
     ███████████░░░░░░░░░   55,579  (12%)   ⚠ -62%   0.3h ago
     ░░░░░░░░░░░░░░░░░░░░        0   (0%)   ✕ SILENT  24h+ ago

## Volume Anomalies

| Host | Current Period | Previous Period | Change | Status |
|------|---------------|----------------|--------|--------|
| host1 | 12,345 | 45,678 | -73% | ⚠ DROP |
| host2 | 89,012 | 23,456 | +279% | ⚠ SPIKE |

## Ingestion Delay


Ingestion Delay by Sourcetype
─────────────────────────────────────────────────────────────────────
aws:cloudtrail     ████████████████████  avg 847s   p95 1,203s   max 3,601s   ⚠ DELAYED
XmlWinEventLog     ██░░░░░░░░░░░░░░░░░░  avg  12s   p95    45s   max   120s   ✓ OK

(Flag sourcetypes where p95 > 300s. Use different thresholds for cloud vs on-prem — cloud sources commonly have 5-15 min delay which is normal.)

## Data Model Status

| Data Model | Accelerated | Window | App | Impact |
|-----------|------------|--------|-----|--------|
| Network_Traffic | ✕ No | — | SA-NetworkProtection | CIM network searches will be slow |
| Endpoint | ✓ Yes | -7d | SA-EndpointProtection | — |
| Authentication | ✓ Yes | -30d | SA-AccessProtection | — |

(Only show CIM-relevant security data models. Flag unaccelerated ones with impact notes.)

## Splunk Internal Errors

| Category | Component | Level | Count | Sample Message |
|----------|----------|-------|-------|---------------|
| Forwarder | TcpInputProc | ERROR | 234 | Connection from 10.1.2.3 lost |
| Parsing | DateParserVerbose | WARN | 1,203 | Failed to parse timestamp... |

(Group by category, show top errors with truncated sample messages.)

## Recommendations

Prioritized action items based on all findings:

1. **[CRITICAL]** <action> — <reason from correlation>
2. **[HIGH]** <action> — <reason>
3. **[MEDIUM]** <action> — <reason>
...

Use > [!failure] callout if critical findings exist, > [!warning] if only high, > [!success] if no high or critical.
Step 6 — Summary

Report to the user:

Total hosts and event volume
Finding counts by severity (critical/high/medium/low)
Top 3 most important findings with one-line descriptions
Cross-correlations discovered (the LLM insight)
Confirm note written to Obsidian
No results found