Skip to content

Instantly share code, notes, and snippets.

@DecisionNerd
Last active November 13, 2015 02:37
Show Gist options
  • Save DecisionNerd/e42d51af0679acb8a2ad to your computer and use it in GitHub Desktop.
Save DecisionNerd/e42d51af0679acb8a2ad to your computer and use it in GitHub Desktop.
ingesting csv files into logstash requires removal of the header row in the first line of the file. This script processes csv files in the directory where it is executed. Script runs continuously until stopping with ctrl+Z
#!/bin/bash
# ingesting csv files into logstash requires removal of the header row in the
# first line of the file. This script processes csv files in the directory
# where it is executed. Script runs continuously until stopping with ctrl+Z
#specify directory to send files for ingest (without trailing /)
DEST=/home/user/Test/forIngest
echo "processing csv files in $(pwd) for ingest by logstash..."
while true
do
# if there are csv files in the current directory
if [ -a $(pwd)/*.csv ]
then
# select each csv file in the current directory
for file in $(pwd)/*.csv
do
echo "removing header row of $file"
# remove the first line in the csv file
sed -i 1d $file
echo "moving edited $file to $DEST"
# move file to where logstash will grab
mv $file $DEST
echo "waiting on csv files..."
# only run command every second
sleep 1
done
#else
fi
done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment