Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Parsing User-Agent info out of Azure App Service http Logs and getting friendly names
# in the example,they use numbers for the keys, but state they can be other things.
# i just left them as numbers to match their setup. i'm guessing you could use
# (new-guid).guid instead and avoid the ln var all together if that's your thing.
# note that i am not using the 'count' value from up above for this as it's not needed
# i just wanted that to look at when i was originally examining the $agentinfo results
# i usually don't end powershell lines in semicolons, but i actually ran this as a
# oneliner and spread it out here for readability.
$ln=1;
$hash=[ordered]@{};
$agentinfo | %{ $hash.add("$ln",$_.Name);$ln++};
# now we have a list of our agents to work with, let's call the service
# put the api key you got from registering here
$apikey = "your api key goes in here"
# this is the uri for the api thatt i am calling
$apiuri = "https://api.whatismybrowser.com/api/v2/user_agent_parse"
# put the apikey in the header
$header = @{"X-API-KEY"=$apikey}
# for each agent, create a json body to send, i thought i could
# just send the @{} itself as a body, but that did not seem to
# work when i tried it, posting it as json worked fine, however
$results = $agentinfo | %{
iwr $apiuri -method post -headers $header -body ( @{"user_agent"=$_.name} | convertto-json )
}
# and now to view the results, you can dig down into the $results object
# yourself, or, in my case, just print the unique browsers like so...
($results.content | convertfrom-json).parse | select -unique simple_software_string
# also probably a good idea to save the parse to a file at least so you
# can review it later if you need to
($results.content | convertfrom-json).parse | convertto-json > results.json
# list of the log files full names
$logfile = (gci "C:\path\to\my\log\files\*.log").fullname
# the header we'll use for our csv conversion. you could probably use other methods,
# but this is how I usually do it and it works fine. the headers you use may vary,
# you should be able to find it in your file and just copy/paste it like
# this and split it by space like below
$header = "date time s-sitename cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Cookie) cs(Referer) cs-host sc-status sc-substatus sc-win32-status sc-bytes cs-bytes time-taken" -split " "
# now i want to import the data as csv using a space delimiter and the header above,
# in the case of merged files (which i have had in the past where i just echoed a
# bunch of files together), and to avoid the header rows unless you want to
# skip them, i add this where clause
$data = ipcsv $logfile -Header $header -Delimiter " " | where date -ne "#Fields:"
# now that we have a variable with our data in it, let's filter out our user agents
$agentinfo = $data | # filter our agent info out of our log data
where sc-status -eq '200' | # filter out successful calls
where cs-username -ne '-' | # make sure there is a user, since those are the browswers we care about
group "cs(User-Agent)" | # group by the user agent so we can see counts for these browswers
select Count,Name | # get a count and a group name so we can see which are most impoortant
sort count -desc | # sort it so they are ranked by use
select Count,@{n="Name";e={$_.Name.Replace("+"," ")}} # get rid of the + signs in the useragent
# the last step isn't really needed, but the + makes it hard for me to eyeball it, so i replace it
# you can also send it without the plus to the service below, so that's how i do it
# since we only want to see some simple tables of data, this will give us
# a hashtable with the agent strings as keys so we can lookup the info
# we want from the results from the web service. you could add
# other fields in if you wanted, but I was mainly looking for browser
# software/version and operating system so this worked for me.
$useragents = ($results.content | convertfrom-json).parse |
%{
@{
$_.user_agent = @{
software = $_.software
operating_system = $_.operating_system
simple_software_string = $_.simple_software_string
}
}
}
# show a full merge of the original counts with this data, but
# worth noting the key thing we are looking at in this is the
# count and the simple string combo of browser and os
$agentinfo |
select *,@{n="ua";e={ $useragents.($_.Name) } } |
select `
Count `
, @{n="simple";e={$_.ua.simple_software_string}} `
, @{n="os";e={$_.ua.operating_system}} `
, @{n="browser";e={$_.ua.software}}
# this isn't really all i wanted to see, however. so while
# i can see the combo and the breakouts, i am also curious
# about the totals just by browser and just by OS. to see
# that i can just append some grouping and specific
# selects to the above. i could also dump what's above into it's
# own variable to shorten the code line up, but i think
# it's easier to see when i just add the group lines
# and selects to the end.
$agentinfo |
select *,@{n="ua";e={ $useragents.($_.Name) } } |
select `
Count `
, @{n="simple";e={$_.ua.simple_software_string}} `
, @{n="os";e={$_.ua.operating_system}} `
, @{n="browser";e={$_.ua.software}} |
group os | # change this to 'browser' to see the browser data
select `
Name `
, @{
n = "Count"
e = { ($_.Group|select count|measure -sum count).sum }
}
# and, lastly, i'd like to add a pct to this calculation. so for
# that i'll add a total var from the $agentinfo data with our
# original counts by agent string, and use that and a little
# math and rounding to pretty it up
$total = ($agentinfo | select count|measure -sum count).sum
$agentinfo |
select *,@{n="ua";e={ $useragents.($_.Name) } } |
select `
Count `
, @{n="simple";e={$_.ua.simple_software_string}} `
, @{n="os";e={$_.ua.operating_system}} `
, @{n="browser";e={$_.ua.software}} |
group os | # change this to 'browser' to see the browser data
select `
Name `
, @{
n = "Total"
e = { ($_.Group|select count|measure -sum count).sum }
} `
, @{
n = "Pct"
e = {
[math]::Round(
100 *
( ($_.Group|select count|measure -sum count).sum / $total )
, 2)
}
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.