Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Remove consecutive duplicate lines with sed
# from https://www.linuxquestions.org/questions/programming-9/removing-duplicate-lines-with-sed-276169/
# $!N
# if not in the last line, append the next line of input
# pattern space (no duplicates): line1\nline2
# stdout: null
#
# /^\(.*\)\n\1$/
# does the pattern space (eg. line1\nline2) contains duplicate lines (eg. line1 == line2)?
#
# /<REGEX>/!P
# if the pattern space does not contain duplicate lines, print the pattern space up to \n
# pattern space: line1\nline2
# stdout: line1
# else
# pattern space: line1\nline2
# stdout: null
# D
# delete up to \n and restart the loop with line2
# pattern space when restarting cycle: line2
sed -n '$!N; /^\(.*\)\n\1$/!P; D' <file>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment