Lesson 16: Pipes, Redirection & Chaining

Goal: Master the Unix philosophy — connect small tools to build powerful workflows.

Table of Contents

1. Standard Streams
2. Output Redirection — > and >>
3. Input Redirection — <
4. Error Redirection — 2>
5. Pipes — |
6. tee — Split Output
7. xargs — Arguments from Stdin
8. Command Chaining — &&, ||, ;
9. Here Documents
10. Practical Pipelines
11. Exercises

Standard Streams

Every Linux process has three standard streams:

                    ┌──────────┐
  stdin (0)  ──────>│          │──────> stdout (1)
  (keyboard)        │ Process  │
                    │          │──────> stderr (2)
                    └──────────┘        (errors)

Output Redirection — > and >>

Write to a file (overwrite)

# Redirect stdout to a file
echo "Hello" > output.txt

# Command output to a file
ls -la > filelist.txt

# Date to a file
date > timestamp.txt

Append to a file

# Append to existing file
echo "Line 1" > log.txt
echo "Line 2" >> log.txt
echo "Line 3" >> log.txt

cat log.txt
# Line 1
# Line 2
# Line 3

Overwrite vs append

# > overwrites (dangerous — old content is gone!)
echo "new content" > file.txt

# >> appends (safe — old content preserved)
echo "new line" >> file.txt

Input Redirection — <

# Read from a file instead of keyboard
wc -l < /etc/passwd

# Sort a file
sort < unsorted.txt > sorted.txt

# Send mail from a file
mail user@example.com < message.txt

Error Redirection — 2>

Redirect errors to a file

# Send errors to a file
find / -name "*.conf" 2> errors.log

# Discard errors completely
find / -name "*.conf" 2>/dev/null

Redirect stdout and stderr separately

# stdout to one file, stderr to another
command > output.log 2> error.log

Redirect both to the same file

# Method 1: &> (bash shorthand)
command &> all.log

# Method 2: explicit
command > all.log 2>&1

Redirect stderr to stdout (merge)

# Useful for piping — errors also go through the pipe
command 2>&1 | grep "something"

Practical example

# Run a script, save all output, and still see it on screen
./deploy.sh 2>&1 | tee deploy.log

Pipes — |

The pipe | sends the output of one command as input to the next.

command1 | command2 | command3

Basic examples

# Count files in a directory
ls | wc -l

# Find a running process
ps aux | grep nginx

# Sort and remove duplicates
cat names.txt | sort | uniq

# Show disk usage, sorted by size
du -h --max-depth=1 | sort -rh

# Show the 5 largest files
ls -lS | head -5

Building pipelines

Think of each command as a step:

# Step by step:
# 1. Read the file
# 2. Extract column 1 (usernames)
# 3. Sort them
# 4. Count duplicates
# 5. Sort by count
# 6. Show top 10

cat access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -10

tee — Split Output

tee writes output to both a file AND stdout (like a T-pipe).

# Save to file AND display on screen
ls -la | tee filelist.txt

# Append instead of overwrite
ls -la | tee -a filelist.txt

# Save to multiple files
echo "Hello" | tee file1.txt file2.txt file3.txt

Practical uses

# Log a build while watching it
make 2>&1 | tee build.log

# Run and log a script
./deploy.sh 2>&1 | tee "deploy-$(date +%Y%m%d-%H%M).log"

# Write to a protected file with sudo
echo "127.0.0.1 myapp.local" | sudo tee -a /etc/hosts

xargs — Arguments from Stdin

xargs takes input from stdin and converts it into arguments for another command.

Basic usage

# Delete all .tmp files found by find
find . -name "*.tmp" | xargs rm

# Same with safety for filenames with spaces
find . -name "*.tmp" -print0 | xargs -0 rm

With placeholder

# Use {} as placeholder for each input item
echo "file1 file2 file3" | xargs -I {} touch {}.txt

# Copy files to a backup directory
find . -name "*.conf" | xargs -I {} cp {} /backup/

Limit parallel execution

# Process one item at a time
cat urls.txt | xargs -n 1 curl -O

# Process 4 items in parallel
cat urls.txt | xargs -P 4 -n 1 curl -O

Practical examples

# Kill all processes matching a name
pgrep -f "old-app" | xargs kill

# Compress all .log files
find /var/log -name "*.log" -mtime +7 | xargs gzip

# Count lines in all Python files
find . -name "*.py" | xargs wc -l

# Remove all Docker containers
docker ps -aq | xargs docker rm

Command Chaining — &&, ||, ;

Sequential execution with ;

Run commands in sequence regardless of success/failure:

echo "First" ; echo "Second" ; echo "Third"

Conditional AND — &&

Run next command ONLY if previous succeeds:

# Only upgrade if update succeeds
sudo apt update && sudo apt upgrade

# Only start if build succeeds
npm run build && npm start

# Chain multiple steps
mkdir project && cd project && git init && npm init -y

Conditional OR — ||

Run next command ONLY if previous fails:

# Show error message if command fails
cd /nonexistent || echo "Directory not found!"

# Fallback behavior
which nala || echo "nala not installed, using apt"

# Create directory if it doesn't exist
test -d /tmp/mydir || mkdir /tmp/mydir

Combining && and ||

# If-then-else pattern
test -f config.yaml && echo "Config found" || echo "Config missing!"

# Practical: check and act
ping -c 1 google.com > /dev/null 2>&1 && echo "Online" || echo "Offline"

Here Documents

Write multi-line input to a command.

# Create a file with content
cat > config.txt << EOF
server=localhost
port=8080
debug=true
EOF

# Use variables in heredoc (double-quoted behavior)
cat > greeting.txt << EOF
Hello, $USER!
Today is $(date).
EOF

# Literal heredoc (no variable expansion)
cat > template.txt << 'EOF'
Hello, $USER!
This $variable is not expanded.
EOF

Practical Pipelines

System administration

# Top 10 processes by memory
ps aux --sort=-%mem | head -11

# Failed SSH login attempts
grep "Failed password" /var/log/auth.log | awk '{print $11}' | sort | uniq -c | sort -rn | head -10

# Monitor disk space every 5 seconds
watch -n 5 'df -h | grep -E "^/dev"'

# Active network connections summary
ss -tulnp | awk 'NR>1 {print $1}' | sort | uniq -c

Log analysis

# HTTP status code distribution
awk '{print $9}' access.log | sort | uniq -c | sort -rn

# Requests per hour
awk '{print $4}' access.log | cut -d: -f1,2 | sort | uniq -c

# Slowest requests
awk '$NF > 5 {print $7, $NF}' access.log | sort -k2 -rn | head -10

File management

# Find duplicate files by checksum
find . -type f -exec md5sum {} + | sort | uniq -d -w32

# Rename all .jpeg to .jpg
find . -name "*.jpeg" | while read f; do mv "$f" "${f%.jpeg}.jpg"; done

# Bulk create dated backups
ls *.conf | xargs -I {} cp {} "{}.$(date +%Y%m%d).bak"

Exercises

Exercise 1: Redirection

# 1. Save ls output to a file
ls -la /etc > /tmp/etc-contents.txt

# 2. Append the date
date >> /tmp/etc-contents.txt

# 3. Count lines (using input redirection)
wc -l < /tmp/etc-contents.txt

# 4. Search with errors suppressed
find / -name "*.conf" 2>/dev/null | head -10

# 5. Clean up
rm /tmp/etc-contents.txt

Exercise 2: Pipes

# 1. Count unique shells used on the system
cut -d: -f7 /etc/passwd | sort | uniq -c | sort -rn

# 2. Find the 5 largest items in /var
sudo du -sh /var/* 2>/dev/null | sort -rh | head -5

# 3. Count running processes per user
ps aux | awk '{print $1}' | sort | uniq -c | sort -rn | head -10

Exercise 3: Chaining

# 1. Create a directory and enter it (fails gracefully)
mkdir /tmp/test-chain && cd /tmp/test-chain && echo "Success!" || echo "Failed!"

# 2. Check if a command exists
which docker && echo "Docker is installed" || echo "Docker is NOT installed"

# 3. Conditional file creation
test -f /tmp/marker.txt || echo "Created" > /tmp/marker.txt
cat /tmp/marker.txt
rm /tmp/marker.txt

Key Takeaways

• > overwrite, >> append, < input from file
• 2> redirect errors, 2>/dev/null discard errors
• | pipe output → input (the Unix superpower)
• tee — save to file AND display
• xargs — convert stdin to command arguments
• && — run next only if previous succeeds
• || — run next only if previous fails
• Build complex workflows from simple tools

Next Lesson: Lesson 17: Shell Configuration →