Using jq to get+filter aws data

example.md

I've been playing with jq, and I've been having a hard time finding examples of how it works with output from a service like AWS (which I use a lot).

Here is one I use a lot with vagrant-ec2.

When we're launching and killing a lot of instances, the AWS API is the only way to track down which instances are live, ready, dead, etc.

To find instances that are tagged with e.g. {"Key" = "Name", "Value" = "Web-00'} in the middle of a vagrant dev cycle, or a prod launch/replace cycle, you can do something like this:

aws ec2 describe-instances | jq '.Reservations[].Instances | select(.[].Tags[].Value | startswith("Web") ) | 
  select(.[].Tags[].Key == "Name") | 
  {InstanceId: .[].InstanceId, PublicDnsName: .[].PublicDnsName, State: .[].State, LaunchTime: .[].LaunchTime, Tags: .[].Tags}
  | [.]'

This is not as good as a process that does this for you as part of its feedback, but it provides a nice ad-hoc report, that is also a parsable json list.

find_security_group_info_fast.md

Security groups in VPC need to be specified by ID. Getting that from the json output is a bit annoying. All the same, I should learn this JMESpath syntax that the aws cli filter uses. In the mean time:

saws> aws ec2 describe-security-groups | jq '.SecurityGroups[] | select(.GroupName | contains("zk")) | .GroupId'
"sg-.......8"
"sg-.......c"
"sg-.......f"

saws> aws ec2 describe-security-groups | jq '.SecurityGroups[] | select(.GroupName | contains("zknode")) | .GroupName'
"zk"
"sg_zk1"
"sg_zk2"

find_vpc_networks_and_elbs.md

aws ec2 describe-subnets| jq '.[][] | select(.Tags[].Value | contains("foo"))'

Or if you're just doing what I'm doing, and creating ELBs, etc. then you only want the subnet IDs:

aws ec2 describe-subnets| jq '[.[][] | select(.Tags[].Value | contains("foo")) | .SubnetId]'

To find the names of the ELBs in your account:

aws elb describe-load-balancers | jq '.LoadBalancerDescriptions[].CanonicalHostedZoneName'

s3_cloudtrail_by_user.md

Asgard pings the aws API a lot. To look at cloudtrail logs w/o asgard you can do this:

gunzip -c /tmp/xxxxxxxxxxxxx_CloudTrail_us-east-1_20151021T1400Z_hfA6rpME8wAOxUH4.json.gz  \
  | jq ' .Records[] \
  | select(.userIdentity.userName != "asgard")'

That will let you look at a subset of names. You can also select for a specific username, etc. here.