Regular Expressions: A Complete Guide

Regular expressions (regex) are patterns that match character combinations in strings. They are used in text editors, search tools, programming languages, and databases. Regex is not a language itself — it is a mini-language embedded in tools like grep, sed, awk, Python, JavaScript, and VS Code.

How Regex Works

A regex engine walks through your pattern and the target string simultaneously, character by character. It tries to match the pattern at each position in the string. When a match fails, the engine backtracks to try alternative paths. Understanding this backtracking behavior is key to writing efficient regexes.

Literal Characters

Most characters in a regex match themselves. The pattern hello matches the string “hello” and nothing else.

Pattern: hello
Matches: "hello"
No match: "Hello", "helloo", "hel lo"

Regex is case-sensitive by default. Use the i flag for case-insensitive matching.

Character Classes

Match one character from a set:

Shorthand Classes

The dot . matches any character except newline (in most flavors). Use [\s\S] or the s flag to match everything including newlines.

Quantifiers

Quantifiers specify how many times a character or group must appear:

Greedy vs Lazy

String: "<div>hello</div><span>world</span>"

Greedy:  <.+>   matches "<div>hello</div><span>world</span>"
Lazy:    <.+?>  matches "<div>", "</div>", "<span>", "</span>"

Greedy quantifiers match as much as possible and backtrack. Lazy quantifiers (with ? suffix like *?, +?, ??) match as little as possible. Use lazy quantifiers when you want to match the smallest possible substring, such as individual HTML tags.

Possessive Quantifiers

Possessive quantifiers (*+, ++, ?+) match as much as possible but never backtrack. They are useful for optimization when you know backtracking will never succeed:

Pattern: \w++@\w++\.\w++

This pattern matches email-like strings faster than a greedy version because once \w++ matches, it never gives up characters — and since \w can’t match @, the regex engine moves on immediately rather than trying different split points.

Anchors

Anchors do not match characters — they match positions:

Pattern: ^Hello
Matches: "Hello world" (at start)
No match: "Say Hello"

Pattern: world$
Matches: "Hello world" (at end)
No match: "world peace"

Pattern: \bword\b
Matches: "word" as whole word
No match: "sword" or "words"

Groups and Capturing

Capturing Groups

Pattern: (https?://)([\w.]+)
String: "Visit https://example.com today"
Group 1: "https://"
Group 2: "example.com"

Parentheses () group part of a pattern and capture the matched text. Captured groups are available by number (\1, \2 in the pattern, $1, $2 in replacements):

# Reorder date format
echo "2024-01-15" | sed -E 's/([0-9]+)-([0-9]+)-([0-9]+)/\2\/\3\/\1/'
# Output: 01/15/2024

Non-Capturing Groups

Pattern: (?:https?://)?[\w.]+

Use (?:...) when you need grouping but do not need to capture. Non-capturing groups are faster and avoid cluttering your match results.

Named Groups

Pattern: (?<year>\d{4})-(?<month>\d{2})

Named groups ((?<name>...)) make patterns self-documenting. In replacements, use ${name} to reference them instead of numbered backreferences.

Alternation

The pipe | acts as OR within a group:

Pattern: cat|dog|fish
Matches: "cat", "dog", or "fish"

Pattern: (apple|banana) pie
Matches: "apple pie" or "banana pie"

Pattern: ^(Error|Warning|Info):

Alternation stops at the group boundaries. cat|dog|fish tries each alternative left to right and stops at the first match.

Escaping

Use backslash to match literal special characters:

\. \* \+ \? \^ \$ \( \) \[ \] \{ \} \\ \| \/

To match a literal period in a filename: file\.txt. Without the backslash, . matches any character.

Lookahead and Lookbehind

Lookarounds match positions without consuming characters:

# Match "foo" only when followed by "bar"
foo(?=bar)

# Match "foo" only when NOT followed by "bar"
foo(?!bar)

# Match "bar" only when preceded by "foo"
(?<=foo)bar

# Match "bar" only when NOT preceded by "foo"
(?<!foo)bar

Practical Lookaround Examples

# Find all prices ($ followed by numbers)
grep -P '\$\d+\.?\d*' prices.txt

# Find lines with "password" NOT followed by "expired"
grep -P 'password(?!.*expired)' config.txt

# Extract values from key=value pairs (value part only)
echo "name=alice" | grep -oP '(?<==).+'

Flags (Modifiers)

Flags change how the regex engine interprets the pattern:

Flag Syntax by Tool

# grep
grep -i 'pattern' file.txt
grep -P '(?i)pattern' file.txt

# sed
sed -E 's/pattern/replacement/I' file.txt

# JavaScript
/pattern/gi

# Python
re.search(r'pattern', text, re.IGNORECASE)

Regex in Common Tools

grep

# Extended regex (-E)
grep -E 'error|failed' log.txt

# Perl-compatible (-P) for lookaheads
grep -P '(?<=\$)\d+' prices.txt

# Show only matched text (-o)
grep -oP 'id="\K[^"]+' page.html

VS Code Search

Find: (TODO|FIXME): (.+)
Replace: $1: **$2**

sed

# Basic regex (BRE) by default
sed 's/\(error\)/\1 (FIXED)/' log.txt

# Extended regex (-E)
sed -E 's/(error|warning)/[LOG] \1/' log.txt

Python

import re

pattern = r'\b[A-Z][a-z]+ \d{4}\b'  # "January 2024"
matches = re.findall(pattern, text)

JavaScript

// Validate email format
const emailRegex = /^[^\s@]+@[^\s@]+\.[^\s@]+$/;
console.log(emailRegex.test("user@example.com")); // true

// Extract all URLs from text
const urlRegex = /https?:\/\/[^\s]+/g;
const text = "Visit https://example.com and http://test.com";
console.log(text.match(urlRegex)); // ["https://example.com", "http://test.com"]

Performance: Catastrophic Backtracking

Some patterns cause exponential backtracking, freezing your application:

Bad:   (<.+>)+
Good:  (<[^>]+>)+
Better: use proper parser for complex HTML/XML

The pattern (<.+>)+ on the string <div><span>text</span></div> forces the engine to try millions of combinations. Nested quantifiers with overlapping matches are the primary cause. Use specific character classes ([^>]+) instead of . to constrain what can match.

Atomic Groups for Performance

Atomic groups (?>...) prevent backtracking into the group. If the group matches, the engine never tries different permutations within it:

Pattern: (?>\w+):(\d+)
String: "username:1234"

Without the atomic group, if the overall pattern fails, the engine backtracks through \w+ character by character. The atomic group locks in the match and fails fast if the rest of the pattern doesn’t work.

FAQ

What is the difference between regex flavors?

Different tools and languages implement regex with subtle differences. PCRE (Perl Compatible) is the most powerful, supporting lookarounds and atomic groups. POSIX ERE (used by grep -E) supports only basic features. JavaScript regex lacks lookbehind in older versions. Always test your pattern in the specific environment you’re targeting.

How do I match a multiline pattern?

Use the s (dotall) flag so . matches newlines. For line-by-line matching, use the m (multiline) flag so ^ and $ match line boundaries. In Python, combine flags with re.DOTALL | re.MULTILINE.

Why is my regex slow?

Nested quantifiers ((.+)+) cause catastrophic backtracking. Use possessive quantifiers, atomic groups, or more specific character classes. Tools like regex101.com provide debuggers that show exactly how many steps the engine takes.

How do I test and debug regex patterns?

Use online tools like regex101.com, regexr.com, or regExr. They provide real-time matching, explanation of each pattern component, and a debugger that shows backtracking steps. Most programming languages also offer verbose flags that make patterns self-documenting.

What is the \K escape in Perl-compatible regex?

\K resets the start of the match. Everything before \K is consumed for matching but not included in the result. It is useful as an alternative to variable-length lookbehinds, which some regex flavors do not support.

Related: Learn sed and awk and grep command for practical regex usage.