uniq command Linux
The uniq command can be removed to remove the duplicate lines from a text file in Linux or Unix
The duplicate lines should be right next to each other for uniq to work on them, so it is a good idea to first sort the file before passing it through uniq
Example of Uniq
suppose we have a text file
host1:/tmp # cat data
this is a uniq test
first sort the data before using uniq
uniq can be used to filter out duplicate entries
first to the data before using uniq
now, passing this file from uniq will NOT remove the duplicate entries
host1:/tmp # uniq data
this is a uniq test
first sort the data before using uniq
uniq can be used to filter out duplicate entries
first to the data before using uniq
see, that the input and output are same as no adjacent duplicate lines were present in the input file
however, if we first sort the file and than pipe the sort output to uniq than there results are different
host1:/tmp # sort data | uniq
first sort the data before using uniq
this is a uniq test
uniq can be used to filter out duplicate entries
Important Uniq Switches
uniq -u output only the uniq lines
uniq -d output only the non-uniq i.e. duplicate lines
uniq -c count the number of times each line is repeated
e.g.
host1:/tmp # sort data | uniq -c
2 first to the data before using uniq
1 this is a uniq test
1 uniq can be used to filter out duplicate entries
another very useful switch of uniq is uniq -f which can use to instruct uniq to ignore a certain filed while processing the input
d -h command use which proses