Skip to content

Instantly share code, notes, and snippets.

@gzagatti
Created July 30, 2018 21:05
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save gzagatti/3bf037d397e6e47e47e81b7fc8293fd4 to your computer and use it in GitHub Desktop.
Save gzagatti/3bf037d397e6e47e47e81b7fc8293fd4 to your computer and use it in GitHub Desktop.
Remove consecutive duplicate lines with sed
# from https://www.linuxquestions.org/questions/programming-9/removing-duplicate-lines-with-sed-276169/
# $!N
# if not in the last line, append the next line of input
# pattern space (no duplicates): line1\nline2
# stdout: null
#
# /^\(.*\)\n\1$/
# does the pattern space (eg. line1\nline2) contains duplicate lines (eg. line1 == line2)?
#
# /<REGEX>/!P
# if the pattern space does not contain duplicate lines, print the pattern space up to \n
# pattern space: line1\nline2
# stdout: line1
# else
# pattern space: line1\nline2
# stdout: null
# D
# delete up to \n and restart the loop with line2
# pattern space when restarting cycle: line2
sed -n '$!N; /^\(.*\)\n\1$/!P; D' <file>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment