Skip to content

Instantly share code, notes, and snippets.

@hxmuller
Last active October 20, 2016 14:01
Show Gist options
  • Save hxmuller/11e986dedea48d3b0149bbe38be97faf to your computer and use it in GitHub Desktop.
Save hxmuller/11e986dedea48d3b0149bbe38be97faf to your computer and use it in GitHub Desktop.
delete duplicate lines while keeping order
# awk - pattern scanning and text processing language
# a) When the program is omitted, the default is { print $0 }
# b) Each line of filename is read by awk, one at a time.
# c) The line, or record, is stored into the field variable $0.
# d) If an array index consisting of the line is not (!) present
# in the array, then the line is printed.
# e) The array is incremented (current index added to the array).
awk '!array[$0]++' filename > filename2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment