NLP and Transformers Guide
Transformer architectures have revolutionized natural language processing. Models like BERT, GPT, T5, and their successors achieve state-of-the-art results on virtually every NLP task. This guide covers the fundamentals and practical implementation.
The Transformer Architecture
The Transformer, introduced in “Attention Is All You Need” (Vaswani et al., 2017), replaced recurrent neural networks with a self-attention mechanism. Key innovations:
- Self-attention — each token attends to all tokens in the sequence
- Positional encoding — injects position information
- Multi-head attention — multiple attention patterns in parallel
- Feed-forward networks — per-token transformations
- Residual connections — enable training deep networks
Self-Attention
The core of the Transformer is the scaled dot-product attention:
import numpy as np
def scaled_dot_product_attention(Q, K, V, mask=None):
"""
Q: query matrix (batch, seq_len, d_k)
K: key matrix (batch, seq_len, d_k)
V: value matrix (batch, seq_len, d_v)
"""
d_k = Q.shape[-1]
scores = np.matmul(Q, K.transpose(0, 2, 1)) / np.sqrt(d_k)
if mask is not None:
scores = scores + mask
attention_weights = softmax(scores, axis=-1)
output = np.matmul(attention_weights, V)
return output, attention_weights
def multi_head_attention(Q, K, V, num_heads, d_model):
"""Split queries into multiple heads, compute attention, concatenate."""
d_k = d_model // num_heads
# Project and split into heads
Q = linear(Q, d_model).reshape(-1, num_heads, d_k)
K = linear(K, d_model).reshape(-1, num_heads, d_k)
V = linear(V, d_model).reshape(-1, num_heads, d_k)
attention, weights = scaled_dot_product_attention(Q, K, V)
concat = attention.reshape(-1, d_model)
return linear(concat, d_model)Tokenization
Transformers use subword tokenization — splitting text into pieces smaller than words but larger than characters:
from transformers import AutoTokenizer
# BERT uses WordPiece tokenization
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
tokens = tokenizer(
"Transformers are amazing!",
padding=True,
truncation=True,
max_length=128,
return_tensors='pt'
)
print(tokens.input_ids)
# tensor([[ 101, 19081, 2024, 6429, 999, 102]])
print(tokenizer.decode(tokens.input_ids[0]))
# "[CLS] transformers are amazing! [SEP]"Special Tokens
[CLS]— classification token (BERT-style)[SEP]— separator between sequences[PAD]— padding token[UNK]— unknown token[MASK]— masked token (masked language modeling)
BERT: Bidirectional Encoder Representations
BERT (Devlin et al., 2018) uses a masked language model objective to learn bidirectional representations:
from transformers import BertForSequenceClassification, BertTokenizer
import torch
model = BertForSequenceClassification.from_pretrained(
'bert-base-uncased',
num_labels=2 # binary classification
)
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
def predict_sentiment(text):
inputs = tokenizer(
text,
return_tensors='pt',
padding=True,
truncation=True,
max_length=512
)
with torch.no_grad():
outputs = model(**inputs)
probabilities = torch.nn.functional.softmax(outputs.logits, dim=-1)
prediction = torch.argmax(probabilities, dim=-1)
return {
'sentiment': 'positive' if prediction.item() == 1 else 'negative',
'confidence': probabilities[0][prediction].item()
}GPT: Generative Pre-trained Transformer
GPT (Radford et al., 2018) uses a causal (left-to-right) language model objective for text generation:
from transformers import GPT2LMHeadModel, GPT2Tokenizer
model = GPT2LMHeadModel.from_pretrained('gpt2')
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
def generate_text(prompt, max_length=100):
inputs = tokenizer(prompt, return_tensors='pt')
with torch.no_grad():
outputs = model.generate(
inputs.input_ids,
max_length=max_length,
temperature=0.7,
top_p=0.9,
top_k=50,
do_sample=True,
num_return_sequences=1,
pad_token_id=tokenizer.eos_token_id
)
return tokenizer.decode(outputs[0], skip_special_tokens=True)
# Usage
print(generate_text("The future of AI is"))Key Generation Parameters
temperature— controls randomness (lower = more deterministic)top_p(nucleus sampling) — cumulative probability thresholdtop_k— sample from top K tokens onlyrepetition_penalty— discourage repetitive textdo_sample— sample vs greedy decoding
Fine-Tuning Transformers
from transformers import (
AutoTokenizer,
AutoModelForSequenceClassification,
TrainingArguments,
Trainer
)
from datasets import load_dataset
# Load dataset
dataset = load_dataset('imdb')
# Tokenize
tokenizer = AutoTokenizer.from_pretrained('distilbert-base-uncased')
def tokenize_function(examples):
return tokenizer(
examples['text'],
padding='max_length',
truncation=True,
max_length=512
)
tokenized_datasets = dataset.map(tokenize_function, batched=True)
# Define model
model = AutoModelForSequenceClassification.from_pretrained(
'distilbert-base-uncased',
num_labels=2
)
# Training arguments
training_args = TrainingArguments(
output_dir='./results',
evaluation_strategy='epoch',
save_strategy='epoch',
learning_rate=2e-5,
per_device_train_batch_size=16,
per_device_eval_batch_size=16,
num_train_epochs=3,
weight_decay=0.01,
load_best_model_at_end=True,
metric_for_best_model='accuracy',
)
# Trainer
trainer = Trainer(
model=model,
args=training_args,
train_dataset=tokenized_datasets['train'].select(range(1000)),
eval_dataset=tokenized_datasets['test'].select(range(100)),
)
trainer.train()Common NLP Tasks
# Text classification
classifier = pipeline('sentiment-analysis', model='distilbert-base-uncased-finetuned-sst-2-english')
result = classifier("This movie was fantastic!")
print(result) # [{'label': 'POSITIVE', 'score': 0.999}]
# Named Entity Recognition
ner = pipeline('ner', model='dbmdz/bert-large-cased-finetuned-conll03-english')
entities = ner("Apple is looking to buy a startup in London.")
# [{'entity': 'ORG', 'word': 'Apple'}, {'entity': 'LOC', 'word': 'London'}]
# Question Answering
qa = pipeline('question-answering', model='distilbert-base-cased-distilled-squad')
result = qa(
question="What is the capital of France?",
context="Paris is the capital of France. It is known for the Eiffel Tower."
)
print(result) # {'answer': 'Paris', 'score': 0.99}
# Text Summarization
summarizer = pipeline('summarization', model='facebook/bart-large-cnn')
summary = summarizer(long_text, max_length=130, min_length=30)Transfer Learning with Transformers
The pre-train then fine-tune paradigm:
- Pre-training — train on massive unlabeled data (masked language modeling, next sentence prediction)
- Fine-tuning — adapt to specific task with labeled data
- Inference — use fine-tuned model for predictions
This approach requires dramatically less labeled data than training from scratch. A pre-trained BERT model fine-tuned on just a few thousand examples often outperforms a from-scratch model trained on millions.
Real-World Implementation Tips
Production Considerations
When moving from development to production, several factors become critical. Error handling should be comprehensive — every external call (database, API, file system) should have proper error checking, logging, and retry logic where appropriate. Performance monitoring through metrics and structured logging helps identify bottlenecks before they affect users.
Testing Strategy
A thorough testing approach combines multiple levels:
- Unit tests verify individual functions and methods in isolation
- Integration tests validate that components work together correctly
- Edge case tests cover boundary conditions, empty inputs, and error states
- Performance tests ensure the system meets latency and throughput requirements
Test data should be realistic but controlled. Mock external dependencies to make tests fast and deterministic. Aim for tests that are independent, repeatable, and fast enough to run on every commit.
Documentation
Good documentation is essential for maintainable code. Follow these principles:
- Document the “why” not just the “what” — explain design decisions
- Keep examples up to date with the code
- Include usage examples for public APIs
- Document configuration options and their defaults
- Explain error conditions and recovery strategies
Security Best Practices
Security should be considered throughout development:
- Validate all inputs at system boundaries
- Use parameterized queries for database access
- Store secrets in environment variables or secret managers
- Keep dependencies updated to patch vulnerabilities
- Apply the principle of least privilege
Performance Optimization
Optimize based on measured data, not assumptions:
- Profile before optimizing — identify actual bottlenecks
- Measure the impact of each change
- Consider the trade-off between speed and readability
- Cache expensive operations with appropriate invalidation
- Use connection pooling for database and network resources
Monitoring and Observability
Production systems need visibility:
- Structured logging with correlation IDs for request tracking
- Metrics for latency, throughput, error rates, and resource usage
- Health check endpoints for load balancers and orchestration
- Distributed tracing for request flows across services
- Alerts for anomaly detection based on baselines
These patterns apply across all programming languages and frameworks. The specific implementation varies, but the principles remain consistent.
FAQ
Q: What is the difference between BERT and GPT? A: BERT is encoder-only (bidirectional context) — best for understanding tasks (classification, QA, NER). GPT is decoder-only (left-to-right) — best for generation tasks (text completion, chat, creative writing).
Q: How do I choose a pre-trained model? A: Start with a general-purpose model (BERT-base, GPT-2, T5-small). Consider model size, inference speed, and task suitability. Distilled models (DistilBERT) offer 60% speed with 95% performance.
Q: What hardware do I need? A: Fine-tuning BERT-base requires ~8GB GPU RAM (batch size 16, max length 512). Larger models (LLaMA, GPT-3) need multiple GPUs or quantization. For CPU inference, use ONNX or quantized models.
Q: How do I handle long documents? A: Transformers have a maximum context length (BERT: 512, GPT-2: 1024, GPT-4: 128K). For longer documents: truncate, chunk and aggregate, or use long-context models (Longformer, BigBird).
Q: What is the attention mechanism? A: Attention allows each token to “look at” every other token in the sequence and compute a weighted combination. It captures contextual relationships regardless of distance — unlike RNNs which process sequentially.
Q: How do I deploy a transformer model? A: Convert to ONNX, optimize with TensorRT, use Hugging Face Inference API, or serve with FastAPI + PyTorch. For large models, use vLLM, Text Generation Inference (TGI), or Llama.cpp for efficient serving.
Transformer Model Architecture Details
Self-Attention Mechanism
The core innovation of transformers is self-attention, which computes relevance scores between all pairs of positions in a sequence:
def scaled_dot_product_attention(Q, K, V, mask=None):
scores = tf.matmul(Q, K, transpose_b=True)
scores = scores / tf.math.sqrt(tf.cast(tf.shape(Q)[-1], tf.float32))
if mask is not None:
scores += (mask * -1e9)
attention_weights = tf.nn.softmax(scores, axis=-1)
output = tf.matmul(attention_weights, V)
return output, attention_weightsPositional Encoding
Since attention is position-invariant, positional encodings are added to input embeddings:
import numpy as np
def positional_encoding(position, d_model):
angles = np.arange(position)[:, np.newaxis] / np.power(
10000, 2 * (np.arange(d_model)[np.newaxis, :] // 2) / d_model
)
pe = np.zeros((position, d_model))
pe[:, 0::2] = np.sin(angles[:, 0::2])
pe[:, 1::2] = np.cos(angles[:, 1::2])
return pe[np.newaxis, :, :]Fine-Tuning Pretrained Models
from transformers import AutoTokenizer, AutoModelForSequenceClassification
model_name = "bert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(
model_name, num_labels=3
)
inputs = tokenizer(
text_list, padding=True, truncation=True, max_length=128, return_tensors="pt"
)
outputs = model(**inputs, labels=labels)
loss = outputs.loss
logits = outputs.logitsFor a comprehensive overview, read our article on Deep Learning Guide.
For a comprehensive overview, read our article on Ensemble Methods Guide.