Download a CloudWatch Logs group

download.sh

#!/bin/bash

log_group=$1
list_command="aws logs describe-log-streams --log-group-name $log_group"
group_next_token=''

while true; do
  if [[ -n $group_next_token ]]; then
    response=$($list_command --next-token $group_next_token)
  else
    response=$($list_command)
  fi
  group_next_token=$(echo $response | jq -r '.nextToken')
  if [[ ${#group_next_token} -lt 400 ]]; then
    break
  fi
  echo $response | jq -r '.logStreams[].logStreamName' | xargs -n 1 -P 20 ./get-stream.sh $log_group
done

get-stream.sh

#!/bin/bash

log_group=$1
stream=$2
get_command="aws logs get-log-events --log-group-name $log_group --start-from-head"
stream_next_token=''
counter=0

mkdir -p streams/$stream

while true; do
  output_path=$(printf "streams/%s/%s-%05d.log.gz" $stream $stream $counter)
  echo "$output_path" 1>&2
  if [[ -n $stream_next_token ]]; then
    response=$($get_command --log-stream-name $stream --next-token $stream_next_token)
  else
    response=$($get_command --log-stream-name $stream)
  fi
  echo $response | jq -r '.events[].message' | gzip > $output_path
  stream_next_token=$(echo $response | jq -r '.nextForwardToken')
  if [[ ${#stream_next_token} -lt 40 ]]; then
    break
  fi
  counter=$((counter + 1))
done

Usage: download.sh name-of-group. Will create a directory called "streams", and within that directory one directory per stream, and within those one file per stream page. The stream pages will be called "stream-N.log" where N is a sequence number with a fixed length of five.

When I execute get-stream.sh I get:
./get-cloudwatch-stream.sh: line 17: awssec: command not found

What is awssec that you have used in the script?

Sorry, awssec is an internal tool we use to manage temporary AWS credentials, I had missed to remove it in one place.