03. Finding Unique or Duplicate elements uniq

The uniq command takes in a sorted file and reports duplicated lines. With the proper options, it can be used to omit or report unique or repeated lines.

Use sort first!

If we use the uniq command on an unsorted list, it will report unexpected results, so make sure you sort before!

$ sort test.txt | uniq

Default output

The example file here simply lists the names of a few people. If we sort then call uniq, it will return all names in the file, just once.

$ cat names.txt
Bob
Chase
Jon
Clara
Bob
Theresa
Jon
Billy
Jonathan
Clara
Jonathan
Bob
Jonny
$ sort names.txt | uniq
Billy
Bob
Chase
Clara
Jon
Jonathan
Jonny
Theresa

Outputting repeated lines

To output lines that have repeated elements, we pass in the -d option.

$ sort names.txt | uniq -d
Bob
Clara
Jon
Jonathan

Outputting unique lines

To output lines that occur just once, pass in the -u option.

$ sort names.txt | uniq -c
Billy
Chase
Jonny
Theresa

With count

With the -c option, we can find how many times each occurrence occurs.

$ sort names.txt | uniq -c
   1 Billy
   3 Bob
   1 Chase
   2 Clara
   2 Jon
   2 Jonathan
   1 Jonny
   1 Theresa

More options

Here are a list of the most-used options. Check out the man page for more.

-c, --count
Precede each duplicate line occurrence by the number of times the duplicate occurs.
-d, --repeated
Output only repeated lines, rather than unique lines.
-f, --skip-fields=n
Ignore comparing the first n fields in each line.
-i, --ignore-case
Ignore case during the line comparisons.
-s --skip-chars=n
Skip the leading n characters of each line.
-u, --unique
Opposite of -d - output only unique lines. This is the default setting.

Aching back from coding all day?

Inversion Therapy Table

Aching back from coding all day? Try Back Problems

Stretch out your back and relieve your back muscles with inversion therapy. This device counteracts the forces of gravity on the body by decompressing and elongating the spine. By using this product just ten minutes a day, you can be well on your way to improved circulation and posture while relieving muscle aches, back pain and stress.

$$ Check price
119.98119.98Amazon 4.5 logo(1,700+ reviews)

More Back Problems resources

Take your Linux skills to the next level!

The Linux Command Line

Take your Linux skills to the next level! Try Linux & UNIX

The Linux Command Line takes you from your very first terminal keystrokes to writing full programs in Bash, the most popular Linux shell. Along the way you'll learn the timeless skills handed down by generations of gray-bearded, mouse-shunning gurus: file navigation, environment configuration, command chaining, pattern matching with regular expressions, and more.

$ Check price
39.9539.95Amazon 4.5 logo(274+ reviews)

More Linux & UNIX resources

Ad