Basic linux commands

AWK script to get standard deviation

AWK script to get min, max, sum, average, standard deviation for numbers:

cat data.txt | awk -F ":" 'BEGIN {max = 0; min = 999999; sum = 0; sumsq = 0} { x= ($NF + 0); sum += x; sumsq += x*x; if(x > max)max = x; if(x < min) min = x}  END { print "Min: ", min,  " Max: ",max, " Sum: ", sum, " Average: ", sum/NR , " Std Dev: ", sqrt(sumsq/NR - (sum/NR)^2) }'  

Assume you have data.txt that has following data set,
2
3
6
8
11

Result:
========
Min:  2  Max:  11  Sum:  30  Average:  6  Std Dev:  3.28634


Refer: http://azimuth.biz/2010/02/25/calculate-standard-deviation-using-awk/

Basic linux commands

Grep commands:
1) Search files with content  text 'build', exclude .xml files
  grep -r  --include=\*.xml  "build"

To count numbers in a file
cat a.txt | awk '{s+=$1} END {print s}'

Get unique text in file

cat -n | sort --key=2.1 -b -u | sort -n


get unique list of HTTP call for jboss access logs.

cat  access.log  | awk -F " " '{print $7}' | cat -n | sort --key=2.1 -b -u | sort -n