Sed and Awk Guide: Powerful Text Processing

Linux Linux 8 min read 1634 words Beginner ExcellentWiki Editorial Team

Sed and awk are the two most powerful text processing tools on Unix-like systems. Sed excels at stream editing and substitutions. Awk is a full programming language designed for structured data processing and report generation.

Sed: Stream Editor

Sed processes text line by line, applying operations to lines that match patterns. It is ideal for find-and-replace, line deletion, insertion, and text transformations.

Basic Syntax

sed 's/pattern/replacement/flags' input.txt

# Flags:
# g — global (all occurrences on line)
# i — case-insensitive
# p — print line
# w file — write to file

Common Sed Commands

# Simple substitution (first occurrence per line)
sed 's/foo/bar/' file.txt

# Global substitution (all occurrences)
sed 's/foo/bar/g' file.txt

# Case-insensitive
sed 's/foo/bar/I' file.txt

# In-place editing (GNU sed)
sed -i 's/foo/bar/g' file.txt
sed -i.bak 's/foo/bar/g' file.txt    # backup to file.txt.bak

# Delete lines
sed '/pattern/d' file.txt            # delete matching lines
sed '5d' file.txt                    # delete line 5
sed '2,5d' file.txt                  # delete lines 2-5
sed '/^#/d' file.txt                 # delete comments
sed '/^$/d' file.txt                 # delete empty lines

# Print specific lines
sed -n '10p' file.txt                # print line 10 only
sed -n '5,10p' file.txt              # print lines 5-10
sed -n '/error/p' file.txt           # print matching lines

# Insert and append
sed '3i\inserted line' file.txt      # insert before line 3
sed '3a\appended line' file.txt      # append after line 3

# Multiple operations
sed -e 's/foo/bar/' -e 's/baz/qux/' file.txt
sed 's/foo/bar/; s/baz/qux/' file.txt

Practical Sed Examples

# Convert DOS line endings to Unix
sed 's/\r$//' windows.txt > unix.txt

# Strip HTML tags (basic)
sed 's/<[^>]*>//g' file.html

# Remove trailing whitespace
sed -i 's/[[:space:]]*$//' file.txt

# Add line numbers
sed = file.txt | sed 'N;s/\n/\t/'

# Replace only on lines matching pattern
sed '/production/s/debug/info/' config.txt

# Extract text between markers
sed -n '/START/,/END/p' file.txt

# Replace newlines with spaces (join lines)
sed ':a;N;$!ba;s/\n/ /g' file.txt

# Capitalize first letter of each word
sed 's/\b\(.\)/\u\1/g' file.txt

Awk: Pattern Scanning and Processing

Awk treats input as records (usually lines) and fields (words separated by whitespace). It is essentially a data processing language with C-like syntax, associative arrays, and built-in pattern matching.

Basic Structure

BEGIN { actions }          # run before processing input
/pattern/ { actions }      # run for lines matching pattern
{ actions }                # run for all lines
END { actions }            # run after processing all input

# Print specific fields
awk '{print $1, $3}' file.txt        # first and third fields
awk '{print $NF}' file.txt           # last field
awk '{print NR, $0}' file.txt        # line number and full line

# Field separator
awk -F',' '{print $1, $2}' data.csv
awk -F':' '/error/{print $1}' /var/log/syslog
awk -F'[ ,]+' '{print $1, $2}' file.txt  # multiple separators

# Pattern matching
awk '/error/ {print $0}' app.log
awk '$3 > 100 {print $1, $3}' data.txt
awk '$1 ~ /^admin/ {print $0}' /etc/passwd

# BEGIN and END blocks
awk 'BEGIN {print "Start"} {print $0} END {print "End"}' file.txt

Awk Variables

# Built-in variables
NR    # Current record number (line number)
NF    # Number of fields in current record
FS    # Field separator (default: whitespace)
OFS   # Output field separator (default: space)
RS    # Record separator (default: newline)
ORS   # Output record separator (default: newline)
$0    # Entire current line
$1, $2, ...  # Individual fields
FILENAME  # Current input filename

# Custom variables
awk -v threshold=100 '$3 > threshold {print}' data.txt
awk '{count++} END {print count, "lines processed"}' file.txt

Practical Awk Examples

# Sum a column
awk '{sum += $1} END {print sum}' numbers.txt
awk '{sum += $1; count++} END {print "Average:", sum/count}' numbers.txt

# Count occurrences
awk '{count[$1]++} END {for (item in count) print item, count[item]}' data.txt

# Filter and compute
awk '$3 >= 1000 {total += $3; count++} END {print total, count}' sales.txt

# Group by field
awk '{group[$1] += $2} END {for (g in group) print g, group[g]}' data.txt

# Process multiple files
awk 'FNR==1 {print "=== " FILENAME " ==="} {print}' *.txt

# Format output
awk '{printf "%-20s %5d\n", $1, $2}' data.txt

# Conditional formatting
awk '{if ($3 > 100) print $1, $3, "HIGH"; else print $1, $3, "LOW"}' report.txt

# Parse log file
awk '{print $1, $4, $7, $9}' /var/log/nginx/access.log | head

# Convert CSV to space-separated
awk -F',' '{OFS=" "; $1=$1; print}' data.csv

Real-World Pipeline Examples

# Top 10 IP addresses from access log
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -10

# HTTP status code distribution
awk '{print $9}' /var/log/nginx/access.log | sort | uniq -c | sort -rn

# Total size of files matching pattern
ls -l *.log | awk '{total += $5} END {print total " bytes"}'

# Find longest line in file
awk '{if (length > max) {max = length; line = $0}} END {print max, line}' file.txt

# Extract email addresses
grep -oE '[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}' file.txt | sort -u

# Remove duplicate lines (preserving order)
awk '!seen[$0]++' file.txt

# Format /etc/passwd nicely
awk -F':' '{printf "%-20s %-5s %-20s\n", $1, $3, $6}' /etc/passwd

# Monitor log file in real time
tail -f app.log | awk '/ERROR/ {print strftime(), $0}'

When to Use Which

Task	Tool
Simple find-and-replace	`sed`
Line deletion/insertion	`sed`
Extract columns from structured data	`awk`
Sum/average/aggregate columns	`awk`
Complex conditional processing	`awk`
Report generation	`awk`
Multi-line operations	`awk` or `sed` with hold space
CSV/TSV transformation	`awk` with `-F','`

Real-World Implementation Tips

Production Considerations

When moving from development to production, several factors become critical. Error handling should be comprehensive — every external call (database, API, file system) should have proper error checking, logging, and retry logic where appropriate. Performance monitoring through metrics and structured logging helps identify bottlenecks before they affect users.

Testing Strategy

A thorough testing approach combines multiple levels:

Unit tests verify individual functions and methods in isolation
Integration tests validate that components work together correctly
Edge case tests cover boundary conditions, empty inputs, and error states
Performance tests ensure the system meets latency and throughput requirements

Test data should be realistic but controlled. Mock external dependencies to make tests fast and deterministic. Aim for tests that are independent, repeatable, and fast enough to run on every commit.

Documentation

Good documentation is essential for maintainable code. Follow these principles:

Document the “why” not just the “what” — explain design decisions
Keep examples up to date with the code
Include usage examples for public APIs
Document configuration options and their defaults
Explain error conditions and recovery strategies

Security Best Practices

Security should be considered throughout development:

Validate all inputs at system boundaries
Use parameterized queries for database access
Store secrets in environment variables or secret managers
Keep dependencies updated to patch vulnerabilities
Apply the principle of least privilege

Performance Optimization

Optimize based on measured data, not assumptions:

Profile before optimizing — identify actual bottlenecks
Measure the impact of each change
Consider the trade-off between speed and readability
Cache expensive operations with appropriate invalidation
Use connection pooling for database and network resources

Monitoring and Observability

Production systems need visibility:

Structured logging with correlation IDs for request tracking
Metrics for latency, throughput, error rates, and resource usage
Health check endpoints for load balancers and orchestration
Distributed tracing for request flows across services
Alerts for anomaly detection based on baselines

These patterns apply across all programming languages and frameworks. The specific implementation varies, but the principles remain consistent.

FAQ

Q: What is the difference between sed and awk? A: Sed is a stream editor focused on line-based text transformations (substitution, deletion, insertion). Awk is a full programming language designed for field-based data processing with variables, arrays, and control flow.

Q: How do I edit files in-place with sed on macOS? A: macOS sed requires an argument for -i: sed -i '' 's/foo/bar/g' file.txt. GNU sed uses sed -i 's/foo/bar/g' file.txt.

Q: Can awk handle CSV data with quoted fields? A: Not reliably with default field splitting. Use FPAT in GNU awk: awk -v FPAT='"[^"]*"|[^,]*' '{print $1}' file.csv.

Q: How do I pass shell variables to awk? A: Use -v: awk -v pattern="$search" '$0 ~ pattern' file.txt.

Q: What does awk ‘!seen[$0]++’ do? A: It removes duplicate lines. The first time a line is seen, seen[$0] is 0, !0 is true, the line is printed, and seen is incremented. Subsequent occurrences have seen[$0] > 0, !n is false, so they are not printed.

For a comprehensive overview, read our article on Bash Scripting Basics.

For a comprehensive overview, read our article on Cron Jobs Guide.