Linux Text Processing: grep, sed, awk Guide
The Unix Text Processing Philosophy
Linux text processing tools embody the Unix philosophy: small, focused programs that do one thing well, combined through pipes to perform complex transformations. A single pipeline — combining grep, sort, uniq, and awk — can analyze gigabytes of log data, extract insights, and produce formatted reports, all without loading the data into memory more than once. These tools operate on streams of text lines, making them memory-efficient for arbitrarily large files. Understanding them is essential for system administration, DevOps, and data engineering on Linux. Brian Kernighan and Rob Pike’s “The Unix Programming Environment” (Prentice Hall, 1984) remains the definitive reference on this tool-based programming philosophy. Modern Linux distributions provide GNU versions of these tools, which include extended features beyond the POSIX standard. While modern scripting languages like Python offer alternatives, the Unix text tools remain unmatched for interactive use and one-liner data transformations.
grep: Pattern Search
grep is the foundational text search tool. It reads lines from input and prints those matching a pattern:
grep -rn "FATAL" /var/log/app/ # Recursive search with line numbers
grep -l "ERROR" *.log # Only print filenames with matches
grep -c "timeout" access.log # Count matching lines
grep -B5 -A10 "CRITICAL" syslog # Context: 5 before, 10 after
grep -i "warning\|error" # Case-insensitive multiple patterns
grep -v "^#" config # Exclude commentsgrep -P enables Perl-compatible regex (PCRE) for lookaheads and backreferences. For literal string search, grep -F is significantly faster using the Aho-Corasick algorithm. The ripgrep (rg) command offers dramatically faster performance for large codebases through SIMD-accelerated search. For JSON log formats, jq provides query capabilities that complement grep.
sed: Stream Editor
sed applies scripted transformations to text streams. Its core operation is s/regex/replacement/flags:
sed -i 's/old-host/new-host/g' /etc/config/*.yaml # In-place replacement
sed -i.bak 's/debug: true/debug: false/' config.yml # Backup before edit
sed '/^#/d; /^$/d' config.yml # Remove comments and blanks
sed 's/ */\t/g' data.txt # Convert spaces to tabs
sed -n '50,100p' large-file.log # Extract line range
sed '/BEGIN/,/END/!d' output.txt # Extract block between markersThe -i flag modifies files in place. On macOS (BSD sed), -i '' requires an empty backup extension argument. For complex transformations, write sed scripts to files (sed -f script.sed). The hold space and pattern space provide two buffers for multi-line operations, enabling advanced transformations like reversing file order or removing duplicate consecutive lines.
awk: Pattern Scanning and Processing
awk is a full programming language for field-based text processing. It splits each input line into fields ($1, $2, … $NF) using a configurable field separator (default: whitespace):
# Column extraction
awk '{print $1, $NF}' data.txt # First and last field
awk -F: '{print $1, $3}' /etc/passwd # Custom delimiter
# Aggregation
awk '{sum += $3; count++} END {print avg = sum/count}' budget.csv
awk '/ERROR/ {count++} END {print count " errors"}' app.log
# Conditional processing
awk '$4 > 1000 {print $1, $4}' sales.tsv # Rows where col4 > 1000
awk 'NR > 1 {print $1, $2 * $3}' prices.txt # Skip header row
# Pattern range
awk '/^START/,/^END/' document.txt # Extract between markersGNU awk (gawk) adds associative arrays, multi-dimensional arrays, network access, and time functions. Built-in variables like NR (record number), NF (field count), FS (field separator), and RS (record separator) provide fine-grained control.
Composing Pipelines
The true power emerges through composition:
# Top 10 HTTP 404 offenders from nginx access log
grep ' 404 ' access.log \
| awk '{print $1}' \
| sort \
| uniq -c \
| sort -rn \
| head -10
# Response time distribution
awk '{print $NF}' access.log \
| grep -oP '^\d+' \
| sort -n \
| awk 'BEGIN{n=0} {arr[n++]=$1} END \
{print "p50:", arr[int(n*0.5)]; \
print "p95:", arr[int(n*0.95)]; \
print "p99:", arr[int(n*0.99)]}'The cut command extracts fixed columns when field splitting is simpler than awk. tr translates or deletes characters. paste merges lines from multiple files column-wise. xargs converts standard input into command arguments, with -P for parallel execution.
Performance Considerations
For large files, pipeline order matters: filter early with grep before processing with awk. Use LC_ALL=C for significant speedups on ASCII data. The sort -S option controls memory usage for large datasets. For truly massive datasets, consider using zcat for compressed logs or find with -exec for parallel processing.
Advanced awk Programming
Beyond one-liners, awk scripts can implement substantial data processing logic. Associative arrays serve as hash maps: awk '{count[$1]++} END {for (k in count) print k, count[k]}' produces frequency distributions. Multi-dimensional arrays are simulated through concatenated indices: awk '{items[$1,$2]++}' creates two-dimensional frequency tables. The split() function parses fixed-width records into arrays. Custom functions enable code reuse: function min(a,b) { return a < b ? a : b }. Awk’s pattern-action model naturally handles report generation: BEGIN blocks print headers, pattern blocks process data, and END blocks compute summaries. For CSV processing, the FPAT variable defines field patterns rather than simple delimiters, handling quoted fields with embedded commas: awk -v FPAT='[^,]*|"[^"]*"' '{print $1}'. Awk scripts can include other scripts via the -f option, supporting reusable library patterns. The gawk extension provides networking (/inet/tcp/0/host/port), profiling (--profile), and debugger (--debug) capabilities that extend awk beyond traditional text processing.
Processing Structured Data
Modern log processing often involves structured formats like JSON, YAML, or XML. The jq tool is essential for JSON processing: jq '.errors[] | select(.severity == "critical") | {message, timestamp}' app.log extracts and filters specific fields from JSON logs. For YAML processing, yq provides similar capabilities. The mlr (Miller) tool processes CSV, JSON, and other structured formats with SQL-like operations: mlr --csv cut -f host,status then sort -f host access.csv. Combining jq with traditional text tools enables powerful pipelines: jq -r '.requests[] | [.ip, .duration] | @tsv' | sort -k2 -rn | head -20. For Apache and Nginx access logs, goaccess generates real-time HTML dashboards. The lnav tool provides an interactive log file viewer with automatic format detection. For binary log formats, strings extracts readable content, and xxd or od provides hex dumps. The pcregrep tool supports multi-line regex patterns across log entries that span multiple lines. Structured log processing with these tools eliminates the need for custom scripts in many data extraction and analysis scenarios.
Advanced Sed and Awk Techniques
Sed one-liners solve many text transformation tasks efficiently. In-place editing with backup: sed -i.bak 's/old/new/g' file. Multi-line pattern matching requires the N command to append the next line: sed '/start/,/end/{/start/{h;d};/end/{H;x;s/pattern/replacement/}}'. Deleting blank lines: sed '/^$/d'. Printing specific line ranges: sed -n '10,20p'. The hold space enables cross-line operations: sed '/pattern1/{h;d};/pattern2/{H;x;s/ //}'.
Awk excels at field-based processing. Summing a column: awk '{sum+=$1} END {print sum}'. Filtering by field comparison: awk '$3 > 100'. The BEGIN and END blocks handle initialization and finalization. Awk’s associative arrays enable grouping: awk '{count[$1]++} END {for (k in count) print k, count[k]}' | sort -k2 -rn. Multi-file processing uses FILENAME and FNR variables. The system() function calls external commands but has security implications for untrusted input. The getline function reads the next line into $0 or a variable, useful for multi-record processing. Both sed and awk benefit from Shepherd’s -i scripts from the shed project, which provides an interactive tutorial for learning advanced techniques.
Parallel Processing with GNU Parallel
GNU Parallel executes tasks across CPU cores for significant speedup in text processing pipelines. Basic usage: parallel gzip ::: *.log compresses files in parallel. The ::: syntax provides input arguments, :::: reads arguments from files. The --jobs flag controls concurrency: --jobs 0 uses all cores, --jobs 50% uses half the cores. The --pipe mode splits stdin into chunks processed by workers: cat hugefile | parallel --pipe wc counts lines in parallel.
The replacement strings control argument insertion: {} for full argument, {.} without extension, {/} basename, {//} dirname, {#} job number. The --dry-run flag previews commands without execution. The --eta flag shows estimated completion time. The --progress flag displays per-job completion. The --sshlogin flag distributes jobs across network hosts. The --semaphore flag controls concurrent access to resources. For complex pipelines, use --rpl to define custom replacement patterns. The --halt now,fail=1 stops on first failure. Parallel combined with awk, jq, and sed creates powerful distributed text processing pipelines that scale across all available CPU resources.
Frequently Asked Questions
What is the difference between grep and awk? grep finds lines matching a pattern. awk splits lines into fields for column-based processing, computation, and formatted output.
How do I replace text in multiple files? Use sed -i 's/old/new/g' *.txt or find . -name "*.yml" -exec sed -i 's/old/new/g' {} +.
What is the fastest way to count lines in a file? wc -l is the fastest line counter. For pattern matching, grep -c is faster than grep | wc -l.
How do I extract columns from a CSV file? Use awk -F, '{print $1, $3}' for simple CSV. For quoted fields with embedded commas, use csvtool or a dedicated parser.
What is the difference between single and double quotes in shell pipelines? Single quotes prevent all variable expansion. Double quotes allow shell variables.
Related: Linux Systemd Guide | Linux Namespaces | Linux Storage Guide