Skip to content

Instantly share code, notes, and snippets.

@rsvp
Created January 17, 2011 03:13
Show Gist options
  • Save rsvp/782451 to your computer and use it in GitHub Desktop.
Save rsvp/782451 to your computer and use it in GitHub Desktop.
uniqnosort : uniq replacement -- shell script to get unique lines WITHOUT prior sort -- preserves original ordering where any subsequent duplicates are removed. Duplicates do NOT have to be adjacent to each other!
#!/usr/bin/env bash
# bash 4.1.5(1) Linux Ubuntu 10.04 Date : 2011-01-16
#
# _______________| uniqnosort : get unique lines WITHOUT sorting.
#
# Usage: uniqnosort [file, or use as pipe]
#
# NB: More famous "uniq" requires duplicates on adjacent lines.
# If you really want to sort and get uniques,
# try "sort -u" rather than "sort | uniq".
#
# To reverse the line ordering, see my "tailflip".
#
# Dependencies: awk
# CHANGE LOG
# 2011-01-16 First version: amazing one-liner.
# _______________ :: BEGIN Script ::::::::::::::::::::::::::::::::::::::::
inf=${1:-'-'}
# ^stdin is default so that we can pipe in stuff.
awk '!x[$0]++' $inf
exit 0
# _______________ EOS :: END of Script ::::::::::::::::::::::::::::::::::::::::
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment