On this page 10 sections
  1. cut — extract fields
  2. sort — order data
  3. uniq — identify duplicates
  4. Count frequency
  5. tr — translate characters
  6. wc — count content
  7. head and tail
  8. Follow logs in real time
  9. tee — display and save simultaneously
  10. Combine tools instead of doing everything with one command
Lab use only. Apply these techniques only to systems you own or have explicit permission to test.
01

cut — extract fields

cut extracts specific character positions or delimiter-separated fields. It is ideal for simple structured data where the delimiter is predictable.

Usernames from passwd
cut -d: -f1 /etc/passwd
Username and shell
cut -d: -f1,7 /etc/passwd
02

sort — order data

sort orders lines alphabetically by default. -n uses numeric ordering, -r reverses the result and -u removes duplicates while sorting.

Alphabetical
sort names.txt
Numeric descending
sort -nr numbers.txt
Sort and remove duplicates
sort -u hosts.txt
03

uniq — identify duplicates

uniq compares adjacent lines, so input should normally be sorted first. -c counts occurrences and -d prints only duplicated values.

Count unique values
sort hosts.txt | uniq -c
Only duplicates
sort hosts.txt | uniq -d
04

Count frequency

sort and uniq are frequently combined to determine which value appears most often.

Top repeated values
sort hosts.txt | uniq -c | sort -nr | head
Most common IPs in access log
awk '{print $1}' access.log | sort | uniq -c | sort -nr | head -20
05

tr — translate characters

tr replaces or deletes individual characters from a stream. It is useful for case conversion, delimiter conversion and whitespace cleanup.

Lowercase to uppercase
echo 'sam0x' | tr '[:lower:]' '[:upper:]'
Replace commas with newlines
tr ',' '\n' < values.csv
Remove carriage returns
tr -d '\r' < windows.txt > linux.txt
06

wc — count content

wc counts lines, words and bytes. -l is especially useful for counting results returned by other commands.

Count lines
wc -l hosts.txt
Count discovered PHP files
find /var/www -type f -name "*.php" | wc -l
07

head and tail

head reads the beginning of input while tail reads the end. They are useful for quickly inspecting large files and limiting pipeline output.

First 20 lines
head -n 20 application.log
Last 50 lines
tail -n 50 application.log
08

Follow logs in real time

tail -f keeps the file open and prints new lines as they are appended. This is useful when troubleshooting services while reproducing an issue.

Follow log
tail -f /var/log/nginx/access.log
Follow only errors
tail -f /var/log/nginx/error.log | grep -i "error"
09

tee — display and save simultaneously

tee copies standard input to both the terminal and one or more files. -a appends instead of overwriting.

Display and save
find /opt -type f | tee files.txt
Append results
echo "api.sam0x.me" | tee -a hosts.txt
10

Combine tools instead of doing everything with one command

Unix tools are designed to compose. A pipeline should normally perform one transformation per stage: select data, extract fields, normalise it, count it and finally limit or save the output.

Top source IPs
awk '{print $1}' access.log | sort | uniq -c | sort -nr | head
Unique shells configured on the system
cut -d: -f7 /etc/passwd | sort -u
Find PHP files containing passwords
find /var/www -type f -name "*.php" -exec grep -Hin "password" {} + 2>/dev/null