Created
January 23, 2019 09:10
-
-
Save jobel-code/e929470e9cd54a78ca402ab7c052cbca to your computer and use it in GitHub Desktop.
Aggregate text files with same header into single file
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# TO AGGREGATE RUN # THIS WILL ADD THE HEADERS ON FOR EACH FILE. USE ONLY FOR DEBUG | |
# cat *.tsv > aggregated_files_with_headers.csv | |
# TO FINAL AGGREGATION RUN IN CONSOLE | |
# SOURCE: https://unix.stackexchange.com/questions/60577/concatenate-multiple-files-with-same-header | |
# The first line of the awk script matches the first line of a file (FNR==1) | |
# except if it's also the first line across all files (NR==1). | |
# When these conditions are met, the expression while (/^<header>/) getline; is executed, | |
# which causes awk to keep reading another line (skipping the current one) as long as | |
# the current one matches the regexp ^<header>. | |
# The second line of the awk script prints everything except for the lines that were previously skipped. | |
## NOTE THE EXTENSION IS .aqs is a tab-separated-values as aqs, you can choose your own extension | |
awk 'FNR==1 && NR!=1{next;}{print}' *.tsv >> aggregated_file.aqs |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment