Important
The 2/3 meta notable event searches below are mainly meant to prevent duplicate analyst work. If they come back without results for a given input, that doesn't mean the value you searched for is "clean", since it doesn't have associated events (e.g. when filtering for a hash/E-Mail address/company asset/external host/user etc). It just means that there aren't any investigation steps that you might consider skipping to avoid duplicated effort because there already exists an associated verdict history. There might still have been other, possibly malicious events in the notable index, that have not (yet) been reviewed/triaged for actual investigation by an analyst. Such events wouldn't show up here since we're filtering for output that is useful for our investigation by providing additional context via past analyst comments. Instead, no events showing up perhaps should be a flag for you to perform your investigation with extra diligence, since your working with an occurence not observed/analyzed before and further verdicts might be derived based on the outcome of your investigation.
Tip
I like to use this one to conclude the quick, initial assessment (just going by the basic, automatically aggregated attributes) I usually run through upfront to help me exclude FPs quickly and avoid wasting time by not diving into deeper analysis straight away.
If an occurence does not look like a TP at first glance but I don't have the evidence to back an FP verdict either, the next step before diving into deeper analysis AND/OR initiating response actions is often reaching out to the user to inquire whether the alerted event was just a benign occurence known to the user or if it truly is a suspicious event without known intention/cause; (only applicable if it's something simple enough for the end-user to understand e.g. a suspicious download, presumably via the browser). Reaching out like that is often a hinderance to fast investigations though, since they might not be available right away (e.g. different timezone/in a meeting etc.). This is where the following search comes in. It allows you to avoid redundant interactions with the user or potentially even entirely unwarranted response actions by discovering the history of related verdicts in the past based on a given attribute/keyword.
Warning
Please make sure to either search for a very specific keyword AND/OR have a closer look at the underlying events via the corresponding IR dashboard link(s) provided and verify they actually match what you're actually looking for (e.g. when investigating a suspicious download, you might initiate a search based on the file name and end up finding a bunch of FP entries; you should then make sure the domain matches as well) before hinging an FP verdict on a previous comment.
| inputlookup incident_review_lookup
| eval analyst_name=user
| append [
| search index=notable earliest=-200d "INSERT_ATTR_KEYWORD_TO_SEARCH_HERE"
| `get_event_id` | eval rule_id=event_id | eval rule_name=search_name
| eval mark=1
]
| stats latest(rule_name) as rule_name, values(analyst_name) as analyst, values(comment) as comment, values(_time) as _time, max(mark) as mark by rule_id
| search mark=1
| where mvjoin(comment,"")!=""
| eval earliest=min(_time)-3600
| eval latest=min(_time)+3600
| eval ir_deeplink="https://your-sub.splunkcloud.com/en-US/app/SplunkEnterpriseSecuritySuite/incident_review?earliest=".earliest."&latest=".latest."&search=event_id%3D%22".rule_id."%22"
| fields comment rule_name analyst ir_deeplink
| sort 0 -_time
credit: me
to be used in similar situations as the attribute/kw based search above
index=notable earliest=-200d
| `get_event_id` | eval rule_id=event_id | eval rule_name=search_name
| eval time_constr_mark=1
| append [
| inputlookup incident_review_lookup | stats values(*) as * by rule_id
| search comment="*INSERT_COMMENT_KEYWORD_TO_SEACH_HERE*"
| eval analyst_name=user
| eval comment_mark=1
]
| stats latest(rule_name) as rule_name, values(analyst_name) as analyst, values(comment) as comment, values(_time) as _time, max(time_constr_mark) as time_constr_mark, max(comment_mark) as comment_mark by rule_id
| search comment_mark=1 time_constr_mark=1
| eval earliest=min(_time)-3600
| eval latest=min(_time)+3600
| eval ir_deeplink="https://your-sub.splunkcloud.com/en-US/app/SplunkEnterpriseSecuritySuite/incident_review?earliest=".earliest."&latest=".latest."&search=event_id%3D%22".rule_id."%22"
| fields comment rule_name analyst ir_deeplink
| sort 0 -_time
credit: me
especially useful for getting up to speed if your org has a rotating role of "lead-reviewer" who concerns themselves with incoming alerts for a certain period (involves being the first point of contact and primarily taking care of resolving/routing alerts to appropriate responders/secondary reviewers). Might also be useful in other circumstances, such as when returning from vacation and needing to quickly grasp current state of affairs (preferrably based on a higher-level aggregation than having to directly dig through the ES IR dashboard).
| inputlookup incident_review_lookup ``` use ES-internal lookup to retrieve IR history ```
| search status=4 OR status=5 ``` only include comments with status Closed/Resolved ```
| stats values(*) as * by rule_id ``` group by what is effectively an event ID (unintuitively named in the lookup ..) ```
``` | search NOT disposition="disposition:9" # optional: I use this to filter out events that were closed fullauto, based on a dedicated disposition we have configured @ my current org, omit this or adjust accordingly ```
| where mvjoin(comment,"")!="" ``` throw out events which have never received a comment ```
| search comment!="FP" AND comment!="duplicate" ``` filter out events with at least one comment that fully matches (case insensitive) "FP" OR "duplicate" ```
| eval analyst_name=user
| eval comment_mark=1 ``` mark these events as relevant, processed later after grouping with results of other search ```
| append [ ``` append events themselves (which are referenced in the lookup of IR activity above) from notable index, careful: result size limited based on your limits.conf, usually 10k/50k - this is the narrower search compared to above, 7 days of events should hopefully be under maxout ```
| search index=notable earliest=-7d@d latest=@d ``` retrieve all events over a period of seven full days ending with the start of the current day ```
| `get_event_id` ``` use ES macro to assemble event ID equivalent to the one coming from the lookup above ```
| eval rule_id=event_id | eval rule_name=search_name ``` rename fields to match the keys coming from upstream results ```
| eval time_constr_mark=1 ``` mark these events as falling in the desired time range ```
]
| stats latest(rule_name) as rule_name, values(analyst_name) as analyst, latest(comment) as closing_comment, values(_time) as _time, max(time_constr_mark) as time_constr_mark, max(comment_mark) as comment_mark by rule_id ``` group the time restricted events from the index with lookup rows ```
| search comment_mark=1 time_constr_mark=1 ``` intersection of events that fall in the time range and also match the comment criteria ```
| eval earliest=min(_time)-3600 | eval latest=min(_time)+3600 ``` prep time for scoped IR dashboard deeplink ```
| eval ir_deeplink="https://your-sub.splunkcloud.com/en-US/app/SplunkEnterpriseSecuritySuite/incident_review?earliest=".earliest."&latest=".latest."&status=4&status=5&search=event_id%3D%22".rule_id."%22"
``` the following segment is based on a semi-auto comment scheme we have established at our org sprinkled with some natural language "heuristics" ^^ - might be somewhat applicable in general but if you get weird truncations, just leave this out ```
| eval comment_fragment=split(closing_comment," - ")
| eval comment_fragment=mvindex(comment_fragment,-1)
| eval closing_comment=if(isnull(comment_fragment),closing_comment,comment_fragment)
| eval comment_fragment=split(closing_comment,". ")
| eval comment_fragment=mvindex(comment_fragment,0)
| eval closing_comment=if(isnull(comment_fragment),closing_comment,comment_fragment)
| eval closing_comment=lower(closing_comment) ``` this does not matter for search but it does affect stats output aggregation```
| stats count, last(_time) as last_time, values(*) as * by rule_name ``` group by rule name for high level overview ```
| eval last_time=strftime(last_time, "%Y-%m-%d %T")
| eval example_events=mvindex(ir_deeplink,-3,-1) ``` limit to last 3 events based on search order (as not to blow up the resulting aggregated view) ```
``` perform random sampling when it comes to choosing a window of comment examples - avoids depending on search result order, this additional randomness should aid us in picking candidates that might be somewhat representative based on prevalence ```
| eval sample_size=5
| eval total_count=mvcount(closing_comment)
| eval sampling_seed=random() % total_count-sample_size+1 ``` sampling starting index, but we prefer less randomness over having a reduced count of examples ```
| eval sampling_seed=if(sampling_seed<0,0,sampling_seed) ``` if we pushed it too far in attempting to recover example size, we adjust until we're non-negative again ```
| eval sampled_comments=mvindex(closing_comment,sampling_seed,sampling_seed+sample_size-1) ``` index end is inclusive, so minus one ```
| fields count rule_name last_time sampled_comments example_events | sort 0 -count
credit: the idea stems from a colleague of mine, but I ended up taking care of the implementation
essential snippet that runs lightning-fast and returns all index names; I use it when I'm looking for that one index I need to reach a verdict, but can't quite remember the name of off the top of my head
| rest /services/data/indexes
``` remove internal indexes from the list (if needed) ```
| search isInternal=0
| fields title id isInternal isReady
| rename isReady as enabled
| stats count by title, enabled
| fields - count
credit: akew on https://community.splunk.com/t5/Splunk-Search/Is-it-possible-to-get-a-list-of-available-indices/m-p/58945
runs through your notable index and collects all URLs present accross any notable events matching a given attribute (e.g. username). Finally, an aggregated overview suitable for kicking off further subinvestigations is presented.
Tip
You can also specifically use this snippet to "cast a net" around a given input URL and find URLs that are related via having appeared in the same notable event previously. (you could also perform that action iteratively, expanding the collection of connected "links" until it eventually stabilizes in size .. not possible to automatically do that currently though ..)
index=notable earliest=-7d "TODO_INSERT_KEYWORD"
| streamstats count as idx
| stats count as unused_for_expansion by url, idx
| eval url_frags=split(url,"/") | eval domain=mvindex(url_frags,0)
| stats values(url) as urls, dc(url) as cardinality, count by domain
| eval samples=mvindex(urls,0,5)
| fields count cardinality domain samples | sort 0 -count
credit: me
a simple graph showing occurences over time. You would expect a low volume compared against overall (browsing) baseline. Peaks and valleys should align. If not, there might be something lurking ...
index=TODO_your_fw_logs src_ip="TODO" user="TODO" earliest=-72h
| eval type=if(like(url, "%TODO_IOC_TO_CHECK%"),"TARGET","BASELINE")
| timechart count by type
credit: me
Tip
If you get alerted to a hit for a previously unencountered, suspicious network resource, a good point to start off your investigation would be to dig out the first contact to that resource in the firewall logs. Given that occurence, you can then draw a time window around it and trace the surrounding logs for the user (≈ src_ip). This search provides a sane default for the time window and automatically returns statistics about the matching traffic.
index=TODO_your_FW_logs [
search index=TODO_your_FW_logs "TODO_IOC" user=TODO_optional_user_target earliest=-10d latest=now
| stats earliest(_time) as first_contact, earliest(src_ip) as src_ip
| eval earliest=first_contact-15 | eval latest=first_contact+45 | fields earliest, latest, src_ip
] | stats count by app | sort 0 -count
credit: me
This search operates on the same, automatic time window as the search above and returns the logs, filtered by src_ip and _time, which spares you from having to carry out the repetitive task of manually snooping around the logs and setting the filters yourself.
The output might let you determine whether the resource was called "directly" and if not, discover likely candidates in terms of referrers/associated resources. E.g. the user might have either visited the site completely by hand or on the contrary, clicked on an URL that was embedded in an E-Mail or which was served as an ad instead.
The results might also allow you to draw conclusions towards a verdict based on how far the overall volume of traffic in the time window deviates from the baseline for that user/IP.
index=TODO_your_FW_logs
[
search index=TODO_your_FW_logs "TODO_IOC" user=TODO_optional_user_target earliest=-10d latest=now
| stats earliest(_time) as first_contact, earliest(src_ip) as src_ip
| eval earliest=first_contact-15 | eval latest=first_contact+45 | fields earliest, latest, src_ip
]
| fields _time, url, action, src_ip, dest_ip, bytes, http_referrer
| eval referrer = if(len(http_referrer)<=100,http_referrer,substr(http_referrer,1,100)." ... [!TRUNCATED]")
| eval url=if(isnull(url),"from ".src_ip." to ".dest_ip." : ".round((bytes/1024),0)."KB",url)
| eval marker=if(like(url,"%TODO_IOC_domain%"),"X","") | table marker, _time, action, referrer, url | sort 0 +_time
credit: me
When investigating an account whose credentials might have been compromised, this is a pragmatic search to kick off the investigation by returning any conspicuous source IP addresses. It's based on an initial baseline period, presumably before the potential breach first occurred. You might have to adjust the time windows for your purposes/adjust them on a case by case basis. It might also make sense to run through multiple scenarios.
``` start off by searching for all events related to the given user in the time period under scrutiny ```
index=YOUR_IDP_LOG user=TODO_INSERT earliest=-30d
| append [
| search index=YOUR_IDP_LOG user=TODO_INSERT earliest=-90d latest=-30d
| eval baseline=1 ``` we assume this period to be a secure baseline ```]
| stats count, max(baseline) as baseline by src_ip ``` group by source IP ```
| search NOT baseline=1 | sort 0 -count ``` gives all IPs that are potential anomalies, sorted by descending count ```
credit: me
Note
This one entirely depends on your firewall logs having a reliable field that details which account fired off a given request.
Tip
When having to reconstruct a timeline in order to correlate events on a multi-user machine, in my experience the firewall log can be a really simple yet fairly reliable starting off point (as opposed to jumping into the trenches and dealing with Windows event logs or digging through different kinds of access logs right away).
I heavily considered leaving this one out since it's so simple while also being very situational with a heavy prerequisite .. here it is anyway:
index=TODO_your_FW_logs src_asset=somehostXXXXX | timechart count by user
search that groups "dynamic" email addresses into generalized schemes
| address-scheme | example address | digest |
|---|---|---|
| "simple address" | admin@contoso.com | admin@contoso.com |
| "simple address" | no-reply@news.contoso.com | no-reply@news.contoso.com |
| "simple address" | tom.scott@visa.go.kr | tom.scott@visa.go.kr |
| address with dynamic subdomain | test@069-dyn-59.k-eta.go.kr | test@*.k-eta.go.kr |
| address with dynamic subdomain | some.user@sub.aaa-06.k-eta.go.kr | some.user@*.k-eta.go.kr |
| address with dynamic prefix | msprvs1=31719wTef8Rm=bounces-542468-2@bounces-us.cisco.com | *@bounces-us.cisco.com |
| address with dynamic prefix | HSBC.753826820.6542019.2568375204@notification.hsbc.com.cn | *@notification.hsbc.com.cn |
| address with dynamic prefix and dynamic postfix | b0604f71.AW4AAANw4jpAAAAAAAAAAR7oQ6oPAA-tntIAAAAAAAUMewBjgMjb@a875033.bnc3.mailjet.com | *@*.mailjet.com |
| makeresults format=csv data="email_address
admin@contoso.com
info@eps.go.kr
tom.scott@visa.go.kr
no-reply@news.contoso.com
noreply@k-eta.go.kr
admin@design-system.service.gov.uk
noreply@coronavirus.data.gov.uk
noreply@overseas.mofa.go.kr
test@069-dyn-59.k-eta.go.kr
admin@5f2a46e.k-eta.go.kr
no-reply@aaa-06.k-eta.go.kr
some.user@sub.aaa-06.k-eta.go.kr
msprvs1=31719wTef8Rm=bounces-542468-2@bounces-us.cisco.com
HSBC.753826820.6542019.2568375204@notification.hsbc.com.cn
bounces+2015189-qq91-accountspayable=contoso.com@sendgrid.net
3bL5mYmIKAKULTLTQJUFQd-STWJUdQLTLTJQ.HTR@scoutcamp.bounces.google.com
b0604f71.AW4AAANw4jpAAAAAAAAAAR7oQ6oPAA-tntIAAAAAAAUMewBjgMjb@a875033.bnc3.mailjet.com
alice.addison=contoso.com__974ti99ihe5e1lr2.qud4knvtdvn29lof@z1raabytcyqi0flq.dtvay.0y-biauw1ey.eu17.bnc.salesforce.com
"
| eval email_address_raw = lower(email_address)
| rex field=email_address_raw "^(?<email_address_prefix_simple>(([a-zA-Z]+\.)?[a-zA-Z-]+))@.*"
| rex field=email_address_raw ".*?(?<email_address_prefix_digest>([=.][a-zA-Z-]+)(\.[a-zA-Z-]+)*)@.*"
| eval email_address_prefix_digest = if(isnull(email_address_prefix_digest), null(), "*".email_address_prefix_digest)
| eval email_address_prefix_digest = coalesce(email_address_prefix_simple, email_address_prefix_digest, "*")
| rex field=email_address_raw ".*@(?<email_address_postfix_raw>.*)"
| rex field=email_address_raw ".*@.*?(?<email_address_postfix_digest>([a-zA-Z-]+\.)+([a-zA-Z]+))$"
| eval email_address_postfix_digest = if(email_address_postfix_raw == email_address_postfix_digest, email_address_postfix_raw, "*.".email_address_postfix_digest)
| eval email_address_digest = email_address_prefix_digest."@".email_address_postfix_digest
| fields - email_address_prefix_simple, email_address_prefix_digest, email_address_postfix_raw, email_address_postfix_digest, email_address_raw
credit: me