Tips & tricks for analyzing and manipulating data in bash

data_tips_and_tricks.md

About

This gist contains a few simple commands for manipulating and analyzing various types of data from the command line. This might be useful, for example, when trying to audit a web applicaton for security defects.

Table of Contents

Encoding and Decoding

1. Hex
2. URL
3. Base64

Hashing

1. MD5
2. SHA-1
3. Others

Replacing

1. Replacing characters in stdin
2. Converting upper to lower case or vice versa in stdin
3. Replacing regex in stdin
4. Replacing regex in file and save changes

Analyzing

1. Using jq to reformat JSON

Encoding

Hex

Hex encoding will return a string with all characters converted to their hex equivalent, with no formatting. Hex decoding will return a string of the ASCII/binary characters represented by each hex byte in the input string.

xxd

Encoding:

echo -n "STRING" | xxd -p

Ex:

$ echo -n "test" | xxd -p
74657374

Decoding:

echo "STRING" | xxd -r -p

Ex:

$ echo -n "74657374" | xxd -r -p
test

URL encoding

URL encoding is useful for inspecting web requests, or manipulating cookie data, for example.

Python (Reference)

Encoding:

python -c "import urllib, sys; print urllib.quote(sys.argv[1])" "STRING"

Ex:

$ python -c "import urllib, sys; print urllib.quote(sys.argv[1])" "'test'"
%27test%27

Decoding:

python -c "import urllib, sys; print urllib.unquote(sys.argv[1])" "STRING"

Ex:

$ python -c "import urllib, sys; print urllib.unquote(sys.argv[1])" "%27test%27"
'test'

Base64

Base64 encoding is a means of representing binary data with ASCII strings, and it is common to see it used on the web for various things.

base64 (part of coreutils)

Encoding:

echo -n "STRING" | base64

Ex:

$ echo -n "test" | base64
dGVzdA==

Decoding:

echo "STRING" | base64 --decode

Ex:

$ echo "dGVzdA==" | base64 --decode
test

openssl

Encoding:

echo -n "STRING" | openssl enc -base64

Ex:

$ echo -n "test" | openssl enc -base64
dGVzdA==

Decoding:

echo "STRING" | openssl enc -d -base64

Ex:

$ echo "dGVzdA==" | openssl enc -d -base64
test

Hashing

MD5

openssl

echo -n "STRING" | openssl dgst -md5

Ex:

$ echo -n "test" | openssl dgst -md5
(stdin)= 098f6bcd4621d373cade4e832627b4f6

md5sum

echo -n "STRING" | md5sum

Ex:

$ echo -n "test" | md5sum
098f6bcd4621d373cade4e832627b4f6

SHA-1

openssl

echo -n "STRING" | openssl dgst -sha1

Ex:

$ echo -n "test" | openssl dgst -sha1
(stdin)= a94a8fe5ccb19ba61c4c0873d391e987982fbbd3

shasum

echo -n "STRING" | shasum

Ex:

$ echo -n "test" | shasum
a94a8fe5ccb19ba61c4c0873d391e987982fbbd3

Others

OpenSSL supports many different cryptographic hashing algorithms, depending on the version of OpenSSL you have installed. You can find what algorithms are supported on your system by typing man openssl and finding the "MESSAGE DIGEST COMMANDS" section.

Some common ones:

• MD4
• MD5
• RIPEMD160
• SHA1
• SHA256
• SHA512

Ex:

$ echo -n "test" | openssl dgst -sha256
(stdin)= 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08


$ echo -n "test" | openssl dgst -ripemd160
(stdin)= 5e52fee47e6b070565f74372468cdc699de89107

Replacing

Replace characters in stdin

tr

echo "STRING" | tr "SEARCH SET" "REPLACE SET"

Ex:

$ echo "this is a test" | tr "s" "z"
thiz iz a tezt


$ echo "this is a test" | tr "ts" "sz"
shiz iz a sezs

sed

echo "STRING" | sed "y/SEARCH SET/REPLACE SET"

Ex:

$ echo "this is a test" | sed "y/s/z/"
thiz iz a tezt


$ echo "this is a test" | sed "y/ts/sz"
shiz iz a sezs

Converting cases in stdin

tr

Lowercase

tr "[:upper:]" "[:lower:]"

Ex:

$ echo "This is a TEST." | tr "[:upper:]" "[:lower:]"
this is a test.

Uppercase

tr "[:lower:]" "[:upper:]"

Ex:

$ echo "This is a TEST." | tr "[:lower:]" "[:upper:]"
THIS IS A TEST.

Replace regex in stdin

sed

echo "STRING" | sed [-r] "s/MATCHING REGEX/REPLACEMENT/[gi]"

Ex:

# replace first instance
$ echo "this is a test" | sed "s/s/z/" 
thiz is a test


# replace all instances
$ echo "this is a test" | sed "s/s/z/g" 
thiz iz a tezt


# replace all instances, ignoring case
$ echo "This Is A Test" | sed "s/t/z/gi" 
zhis Is A zesz


# replace all instances using extended regular expressions
$ echo "Roses are red, violets are blue, Linux is bar, Unix is foo" | sed -r "s/ (is|are) /=/g" 
Roses=red, violets=blue, Linux=bar, Unix=foo

Replace regex in file and save changes

sed

sed -i[SUFFIX OF BACKUP] "s/MATCHING REGEX/REPLACEMENT/[g]"

Ex:

$ cat test
this is a test
$ sed -iold "s/s/z/g" test
$ cat test
thiz iz a tezt
$ cat testold
this is a test

Analyzing

Using jq to reformat JSON

Working with APIs, you'll often get a bunch of JSON that you only want one little piece of. jq is the answer!

Let's look at an example API response:

{
   "networks" : [
      {
         "shared" : true,
         "tenant_id" : "customer1",
         "admin_state_up" : true,
         "status" : "ACTIVE",
         "name" : "public",
         "id" : "blahblah1",
         "subnets" : []
      },
      {
         "shared" : false,
         "tenant_id" : "customer1",
         "admin_state_up" : true,
         "status" : "ACTIVE",
         "name" : "private",
         "id" : "blahblah",
         "subnets" : [
            "blahblah2"
         ]
      }
   ]
}

This isn't the easiest to work with using grep/sed/awk, especially without json_pp, so that's where jq comes in. Say I want just the "id" from each of these ports. I can do:

curl http://site/v1/networks | jq ".networks[].id"

And I will get back:

"blahblah1"
"blahblah"

If I wanted a JSON structured list back, I'd simply do:

curl http://site/v1/networks | jq "[.networks[].id]"

Which returns:

[
  "blahblah1",
  "blahblah"
]

Or a list of JSON objects with just the pieces I want defined:

curl http://site/v1/networks | jq "[{id:.ports[].id}]"

Which gets me:

[
  {
    "id": "blahblah1"
  },
  {
    "id": "blahblah"
  }
]

jq has lots of other handy features, so definitely check out its manual if you want to learn more.