Lesson 7: Text Processing in Scripts

Goal: Use grep, sed, awk, and string manipulation inside scripts for real data processing.

Table of Contents

1. String Manipulation in Bash
2. grep in Scripts
3. sed in Scripts
4. awk in Scripts
5. Practical: Log Analyzer
6. Practical: Config File Parser
7. Exercises

String Manipulation in Bash

Bash has built-in string operations — no external tools needed.

Length, substrings, case

STR="Hello, World!"

echo "${#STR}"          # 13 (length)
echo "${STR:0:5}"       # Hello (substring)
echo "${STR:7}"         # World!
echo "${STR^^}"         # HELLO, WORLD! (uppercase)
echo "${STR,,}"         # hello, world! (lowercase)

Search and replace

FILE="report-2026-03-22.tar.gz"

echo "${FILE/report/backup}"     # backup-2026-03-22.tar.gz
echo "${FILE//-/_}"              # report_2026_03_22.tar.gz (all)

Remove prefix / suffix

FILE="report-2026-03-22.tar.gz"

echo "${FILE%.tar.gz}"    # report-2026-03-22 (remove suffix)
echo "${FILE%%.*}"        # report-2026-03-22 (remove longest suffix)
echo "${FILE#report-}"    # 2026-03-22.tar.gz (remove prefix)
echo "${FILE##*/}"        # (basename equivalent)

PATH_STR="/home/user/docs/file.txt"
echo "${PATH_STR%/*}"    # /home/user/docs (dirname)
echo "${PATH_STR##*/}"   # file.txt (basename)

Practical: Process filenames

#!/bin/bash
for file in *.log; do
    [[ -f "$file" ]] || continue
    base="${file%.log}"
    archive="${base}-$(date +%Y%m%d).log.gz"
    gzip -c "$file" > "$archive"
    echo "Archived: $file → $archive"
done

grep in Scripts

Check if a pattern exists

if grep -q "error" /var/log/syslog; then
    echo "Errors found in syslog!"
fi

-q (quiet) suppresses output — just sets exit code.

Extract matching lines

# Get all error lines
ERRORS=$(grep -i "error" /var/log/syslog | tail -5)
echo "$ERRORS"

# Count matches
ERROR_COUNT=$(grep -c "error" /var/log/syslog)
echo "Found $ERROR_COUNT errors"

grep with regex in scripts

#!/bin/bash
# Extract IP addresses from a log file
grep -oE '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' access.log | sort -u

grep as a filter

# Filter active config lines (no comments, no blanks)
grep -v '^#' /etc/ssh/sshd_config | grep -v '^$'

# Find processes
if ps aux | grep -q "[n]ginx"; then
    echo "nginx is running"
fi

sed in Scripts

In-place file editing

#!/bin/bash
CONFIG="/etc/myapp/config.conf"

# Change a setting
sed -i "s/^port=.*/port=8080/" "$CONFIG"

# Enable a commented-out setting
sed -i "s/^#max_connections/max_connections/" "$CONFIG"

# Add a line after a match
sed -i '/\[database\]/a host=localhost' "$CONFIG"

Template processing

#!/bin/bash
# Replace placeholders in a template
APP_NAME="myapp"
APP_PORT="3000"
DB_HOST="localhost"

sed -e "s|{{APP_NAME}}|$APP_NAME|g" \
    -e "s|{{APP_PORT}}|$APP_PORT|g" \
    -e "s|{{DB_HOST}}|$DB_HOST|g" \
    template.conf > output.conf

Extract data with sed

# Extract version number from a file
VERSION=$(sed -n 's/^version=//p' config.ini)
echo "Version: $VERSION"

# Get lines between markers
sed -n '/BEGIN/,/END/p' data.txt

awk in Scripts

Extract and process columns

#!/bin/bash
# Disk space alert
df -h | awk 'NR>1 {
    gsub(/%/, "", $5)
    if ($5+0 > 80)
        printf "WARNING: %s is %s%% full\n", $6, $5
}'

Summarize data

#!/bin/bash
# Sum file sizes in a directory
TOTAL=$(ls -l | awk 'NR>1 {sum += $5} END {print sum}')
echo "Total bytes: $TOTAL"

# Average response time from log
awk '{sum += $NF; count++} END {printf "Avg: %.2f ms\n", sum/count}' access.log

awk as a mini-program

#!/bin/bash
# Parse /etc/passwd and format output
awk -F: '
    $3 >= 1000 && $7 != "/usr/sbin/nologin" {
        printf "%-15s UID:%-5s Shell:%s\n", $1, $3, $7
    }
' /etc/passwd

Practical: Log Analyzer

#!/bin/bash
set -euo pipefail

LOG_FILE="${1:-/var/log/syslog}"

if [[ ! -f "$LOG_FILE" ]]; then
    echo "File not found: $LOG_FILE" >&2
    exit 1
fi

TOTAL=$(wc -l < "$LOG_FILE")
ERRORS=$(grep -ci "error" "$LOG_FILE" || true)
WARNINGS=$(grep -ci "warn" "$LOG_FILE" || true)

echo "=== Log Analysis: $(basename "$LOG_FILE") ==="
echo "Total lines:  $TOTAL"
echo "Errors:       $ERRORS"
echo "Warnings:     $WARNINGS"
echo ""

if [[ $ERRORS -gt 0 ]]; then
    echo "--- Last 5 Errors ---"
    grep -i "error" "$LOG_FILE" | tail -5
fi

Practical: Config File Parser

#!/bin/bash
set -euo pipefail

# Parse a simple key=value config file
parse_config() {
    local config_file="$1"

    if [[ ! -f "$config_file" ]]; then
        echo "Config not found: $config_file" >&2
        return 1
    fi

    while IFS='=' read -r key value; do
        # Skip comments and empty lines
        [[ "$key" =~ ^[[:space:]]*# ]] && continue
        [[ -z "$key" ]] && continue

        # Trim whitespace
        key=$(echo "$key" | xargs)
        value=$(echo "$value" | xargs)

        echo "  $key = $value"
        export "$key=$value" 2>/dev/null || true
    done < "$config_file"
}

# Usage
cat > /tmp/app.conf << 'EOF'
# Application Config
app_name = MyApp
port = 3000
debug = true
# database
db_host = localhost
db_port = 5432
EOF

echo "=== Parsed Config ==="
parse_config /tmp/app.conf
rm /tmp/app.conf

Exercises

Exercise 1: String manipulation

cat > strings.sh << 'EOF'
#!/bin/bash
set -euo pipefail

URL="https://api.example.com/v2/users?page=1"

echo "Full URL: $URL"
echo "Protocol: ${URL%%://*}"
echo "Domain:   $(echo "$URL" | sed 's|https://||; s|/.*||')"
echo "Path:     /$(echo "$URL" | cut -d/ -f4-)"

FILENAME="backup-2026-03-22.tar.gz"
echo ""
echo "File: $FILENAME"
echo "Without extension: ${FILENAME%.tar.gz}"
echo "Extension: ${FILENAME##*.}"
echo "Uppercase: ${FILENAME^^}"
EOF

chmod +x strings.sh && ./strings.sh
rm strings.sh

Exercise 2: Process checker

cat > proccheck.sh << 'EOF'
#!/bin/bash
set -euo pipefail

SERVICES=("sshd" "cron" "systemd")

echo "=== Process Status ==="
for svc in "${SERVICES[@]}"; do
    PID=$(pgrep -x "$svc" 2>/dev/null | head -1 || true)
    if [[ -n "$PID" ]]; then
        MEM=$(ps -p "$PID" -o rss= 2>/dev/null | awk '{printf "%.1f MB", $1/1024}')
        echo "  [RUNNING] $svc (PID: $PID, Mem: $MEM)"
    else
        echo "  [STOPPED] $svc"
    fi
done
EOF

chmod +x proccheck.sh && ./proccheck.sh
rm proccheck.sh

Key Takeaways

• Bash built-in string ops (${var%...}, ${var//...}) avoid spawning subprocesses
• grep -q for silent checks in if conditions
• sed -i for in-place editing; sed -e for template processing
• awk for column-based data processing and calculations
• Combine tools in pipelines for complex text transformations
• Always handle edge cases: empty files, missing patterns, special characters

Next Lesson: Lesson 8: Error Handling & Debugging →