Lesson 10: Text Processing

Goal: Learn to transform, filter, and manipulate text data using powerful command-line tools.

Table of Contents

1. sort — Sort Lines
2. uniq — Remove Duplicates
3. cut — Extract Columns
4. tr — Translate Characters
5. sed — Stream Editor
6. awk — Pattern Processing
7. Combining Tools
8. Exercises

sort — Sort Lines

Basic sort (alphabetical)

echo -e "banana\napple\ncherry\ndate" | sort

Output:

apple
banana
cherry
date

Numeric sort

echo -e "10\n2\n33\n1\n5" | sort -n

Output:

1
2
5
10
33

Without -n, "10" would come before "2" (alphabetical: "1" < "2").

Reverse sort

sort -r file.txt        # reverse alphabetical
sort -rn file.txt       # reverse numeric (largest first)

Sort by column

# Sort /etc/passwd by UID (3rd field, numeric)
sort -t: -k3 -n /etc/passwd

# Sort by file size (5th column in ls -l)
ls -l | sort -k5 -n

# Sort CSV by 2nd column
sort -t, -k2 data.csv

Human-readable sort

# Sort sizes like 1K, 5M, 2G correctly
du -h /var/log/* 2>/dev/null | sort -h

Sort and remove duplicates

sort -u file.txt

uniq — Remove Duplicates

uniq removes consecutive duplicate lines. Always sort first.

Basic usage

# Remove consecutive duplicates
sort names.txt | uniq

# Count occurrences
sort names.txt | uniq -c

# Show only duplicates
sort names.txt | uniq -d

# Show only unique lines (no duplicates)
sort names.txt | uniq -u

Practical example

# Create test data
cat > access.log << EOF
192.168.1.1
192.168.1.5
192.168.1.1
192.168.1.3
192.168.1.1
192.168.1.5
192.168.1.3
EOF

# Count visits per IP, sorted by frequency
sort access.log | uniq -c | sort -rn

Output:

      3 192.168.1.1
      2 192.168.1.3
      2 192.168.1.5

cut — Extract Columns

cut extracts specific fields or character positions from each line.

Cut by delimiter and field

# Extract usernames from /etc/passwd (field 1, delimiter :)
cut -d: -f1 /etc/passwd

# Extract username and shell (fields 1 and 7)
cut -d: -f1,7 /etc/passwd

# Extract fields 1 through 3
cut -d: -f1-3 /etc/passwd

Cut by character position

# First 10 characters of each line
cut -c1-10 file.txt

# Characters 5 through 15
cut -c5-15 file.txt

# From character 20 to end of line
cut -c20- file.txt

CSV example

cat > employees.csv << EOF
Name,Department,Salary
Alice,Engineering,85000
Bob,Marketing,72000
Carol,Engineering,90000
Dave,Sales,65000
EOF

# Extract names (column 1)
cut -d, -f1 employees.csv

# Extract names and salaries
cut -d, -f1,3 employees.csv

tr — Translate Characters

tr translates (replaces) or deletes characters. It reads from stdin only.

Replace characters

# Convert lowercase to uppercase
echo "hello world" | tr 'a-z' 'A-Z'
# Output: HELLO WORLD

# Convert uppercase to lowercase
echo "HELLO WORLD" | tr 'A-Z' 'a-z'
# Output: hello world

# Replace spaces with underscores
echo "my file name.txt" | tr ' ' '_'
# Output: my_file_name.txt

# Replace colons with tabs
cat /etc/passwd | tr ':' '\t' | head -3

Delete characters

# Delete all digits
echo "abc123def456" | tr -d '0-9'
# Output: abcdef

# Delete all whitespace
echo "hello   world" | tr -d ' '
# Output: helloworld

# Delete newlines (join lines)
cat file.txt | tr -d '\n'

Squeeze repeated characters

# Squeeze multiple spaces into one
echo "too    many     spaces" | tr -s ' '
# Output: too many spaces

# Squeeze blank lines
cat file.txt | tr -s '\n'

sed — Stream Editor

sed is a powerful tool for finding and replacing text in streams or files.

Basic replacement

# Replace first occurrence per line
echo "hello world hello" | sed 's/hello/hi/'
# Output: hi world hello

# Replace ALL occurrences per line (global)
echo "hello world hello" | sed 's/hello/hi/g'
# Output: hi world hi

Replace in files

# Preview changes (print to stdout)
sed 's/old/new/g' config.txt

# Edit file in place
sed -i 's/old/new/g' config.txt

# Create a backup before editing
sed -i.bak 's/old/new/g' config.txt

Delete lines

# Delete line 3
sed '3d' file.txt

# Delete lines 2-5
sed '2,5d' file.txt

# Delete blank lines
sed '/^$/d' file.txt

# Delete comment lines
sed '/^#/d' config.txt

# Delete comments AND blank lines
sed '/^#/d; /^$/d' config.txt

Print specific lines

# Print only line 5
sed -n '5p' file.txt

# Print lines 10-20
sed -n '10,20p' file.txt

# Print lines matching a pattern
sed -n '/error/p' logfile.txt

Advanced sed

# Insert a line before line 3
sed '3i\New line inserted here' file.txt

# Append a line after line 5
sed '5a\Appended after line 5' file.txt

# Replace only on lines matching a pattern
sed '/server/s/80/8080/g' config.txt

# Multiple replacements
sed -e 's/foo/bar/g' -e 's/baz/qux/g' file.txt

Practical examples

# Remove trailing whitespace
sed 's/[[:space:]]*$//' file.txt

# Add prefix to every line
sed 's/^/>>> /' file.txt

# Extract domain from URLs
echo "https://example.com/path" | sed 's|https://\([^/]*\).*|\1|'
# Output: example.com

# Comment out a line in a config
sed -i 's/^PermitRootLogin yes/# PermitRootLogin yes/' /etc/ssh/sshd_config

awk — Pattern Processing

awk is a programming language for text processing. It processes files line by line, splitting each into fields.

Basic usage

By default, awk splits on whitespace. Fields are $1, $2, etc. $0 is the whole line.

# Print the first column
echo "Alice 25 Engineer" | awk '{print $1}'
# Output: Alice

# Print columns 1 and 3
echo "Alice 25 Engineer" | awk '{print $1, $3}'
# Output: Alice Engineer

# Print with custom separator
echo "Alice 25 Engineer" | awk '{print $1 " is an " $3}'
# Output: Alice is an Engineer

Custom field separator

# Parse /etc/passwd (colon-separated)
awk -F: '{print $1, $3}' /etc/passwd

# Parse CSV
awk -F, '{print $1, $3}' employees.csv

Conditions

# Print lines where column 3 > 80000
awk -F, '$3 > 80000 {print $1, $3}' employees.csv

# Print lines matching a pattern
awk '/error/ {print}' logfile.txt

# Print lines longer than 80 characters
awk 'length > 80' file.txt

# Print line numbers
awk '{print NR, $0}' file.txt

Built-in variables

Calculations

# Sum values in column 3
awk -F, 'NR>1 {sum += $3} END {print "Total:", sum}' employees.csv
# Output: Total: 312000

# Average
awk -F, 'NR>1 {sum += $3; count++} END {print "Average:", sum/count}' employees.csv
# Output: Average: 78000

# Min/Max
awk -F, 'NR>1 {if($3>max) max=$3; if(min=="" || $3<min) min=$3} END {print "Min:", min, "Max:", max}' employees.csv

Formatting output

# Printf for formatted output
awk -F, 'NR>1 {printf "%-10s %s\n", $1, $3}' employees.csv

Output:

Alice      85000
Bob        72000
Carol      90000
Dave       65000

Combining Tools

The real power of these tools emerges when you pipe them together.

# Top 10 most common words in a file
cat file.txt | tr ' ' '\n' | tr 'A-Z' 'a-z' | sort | uniq -c | sort -rn | head -10

# Active SSH config (no comments, no blanks)
cat /etc/ssh/sshd_config | sed '/^#/d; /^$/d'

# Disk usage by directory, sorted
du -h --max-depth=1 /home 2>/dev/null | sort -h

# Extract and count HTTP status codes from access log
awk '{print $9}' access.log | sort | uniq -c | sort -rn

# Find users with bash shell, sorted
grep "/bin/bash" /etc/passwd | cut -d: -f1 | sort

Exercises

Exercise 1: sort and uniq

# 1. Create test data
cat > words.txt << EOF
banana
apple
cherry
apple
banana
date
cherry
apple
EOF

# 2. Sort alphabetically
sort words.txt

# 3. Sort and remove duplicates
sort -u words.txt

# 4. Count occurrences of each word
sort words.txt | uniq -c | sort -rn

# 5. Clean up
rm words.txt

Exercise 2: cut and tr

# 1. Extract just usernames from /etc/passwd
cut -d: -f1 /etc/passwd | head

# 2. Convert a string to uppercase
echo "hello world" | tr 'a-z' 'A-Z'

# 3. Replace spaces with newlines
echo "one two three four" | tr ' ' '\n'

Exercise 3: sed and awk

# 1. Create a data file
cat > data.txt << EOF
Server: web01 Status: active Load: 0.5
Server: web02 Status: down Load: 0.0
Server: db01 Status: active Load: 1.2
Server: cache01 Status: active Load: 0.8
EOF

# 2. Extract server names with awk
awk '{print $2}' data.txt

# 3. Find servers that are down
awk '/down/ {print $2}' data.txt

# 4. Replace "down" with "OFFLINE" using sed
sed 's/down/OFFLINE/g' data.txt

# 5. Clean up
rm data.txt

Key Takeaways

• sort — sort lines (-n numeric, -r reverse, -k by column)
• uniq — remove duplicates (always sort first, -c to count)
• cut — extract columns (-d delimiter, -f field)
• tr — translate/delete characters (works on stdin only)
• sed — find & replace in streams (s/old/new/g, -i for in-place)
• awk — column-based processing ($1, $2, conditions, math)
• Pipe them together for powerful data transformations

Next Lesson: Lesson 11: System Information →