Skip to content

Instantly share code, notes, and snippets.

@nickwallen

nickwallen/Validation.md

Last active May 17, 2017 21:06
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save nickwallen/fed89757af41f07cc83513c9b967d568 to your computer and use it in GitHub Desktop.
Save nickwallen/fed89757af41f07cc83513c9b967d568 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
Validation.md

Calculate the rate of dating hitting HDFS. Should roughly match the incoming packet rate.

hadoop fs -ls /apps/metron/pcap | awk '{sum += $5} END { print sum}' && \
  sleep 60 && \
  hadoop fs -ls /apps/metron/pcap | awk '{sum += $5} END { print sum}'

Are the pcap files rolling correctly?

hdfs dfs -ls -r -t /apps/metron/pcap | grep "2017-05-17" | awk -F" " '{print $7}' | sort -n | uniq -c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment