Natural Language Processing: Text Analysis and Models

Natural Language Processing (NLP) enables computers to understand, interpret, and generate human language. From search engines and chatbots to machine translation and sentiment analysis, NLP is one of the most impactful areas of data science.

Text Preprocessing

Raw text must be cleaned and normalized before any analysis:

import re
import nltk
from nltk.corpus import stopwords
from nltk.stem import PorterStemmer, WordNetLemmatizer

def preprocess(text):
    # Lowercase
    text = text.lower()
    # Remove punctuation and digits
    text = re.sub(r'[^\w\s]', '', text)
    text = re.sub(r'\d+', '', text)
    # Tokenize
    tokens = nltk.word_tokenize(text)
    # Remove stopwords
    tokens = [t for t in tokens if t not in stopwords.words('english')]
    # Stemming or lemmatization
    stemmer = PorterStemmer()
    tokens = [stemmer.stem(t) for t in tokens]
    return tokens

Lemmatization is more accurate than stemming but requires a dictionary and is slower. For most modern NLP pipelines (especially with transformers), minimal preprocessing is needed — the models handle casing and punctuation natively.

Text Representation

Bag of Words

Represent a document as a vector of word counts:

from sklearn.feature_extraction.text import CountVectorizer

corpus = ["I love NLP", "NLP is amazing", "I love deep learning"]
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(corpus)
print(vectorizer.get_feature_names_out())
# ['amazing', 'deep', 'is', 'learning', 'love', 'nlp']
print(X.toarray())
# [[0 0 0 0 1 1]
#  [1 0 1 0 0 1]
#  [0 1 0 1 1 0]]

TF-IDF (Term Frequency — Inverse Document Frequency)

Downweights common words and upweights rare ones:

from sklearn.feature_extraction.text import TfidfVectorizer

vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(corpus)

TF-IDF weight for term t in document d: tf-idf(t, d) = tf(t, d) × log(N / df(t)) where N is the total number of documents and df(t) is the number of documents containing t.

Word Embeddings

Dense vector representations where semantically similar words have similar vectors:

Word2Vec (2013) — two architectures: CBOW (predict word from context) and Skip-gram (predict context from word)

from gensim.models import Word2Vec

sentences = [["i", "love", "nlp"], ["deep", "learning", "is", "fun"]]
model = Word2Vec(sentences, vector_size=100, window=5, min_count=1)
model.wv.most_similar("nlp")
# [("machine_learning", 0.85), ("ai", 0.82), ...]

GloVe (2014) — factorizes a word co-occurrence matrix. Pre-trained GloVe vectors (trained on 6 billion tokens) are available in dimensions 50, 100, 200, and 300.

FastText (2016) — extends Word2Vec by learning embeddings for character n-grams, enabling representation of out-of-vocabulary words.

Contextual Embeddings (2018+) — models like BERT and GPT produce different embeddings for the same word depending on its context:

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModel.from_pretrained("bert-base-uncased")
inputs = tokenizer("I love NLP", return_tensors="pt")
outputs = model(**inputs)
# outputs.last_hidden_state has shape (1, 4, 768)
# — one 768-dim vector per token, conditioned on context

Sentiment Analysis

Classifying text as positive, negative, or neutral:

from transformers import pipeline

classifier = pipeline("sentiment-analysis")
result = classifier("This product is amazing!")
# [{'label': 'POSITIVE', 'score': 0.9998}]

Custom Sentiment with ML

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression

vectorizer = TfidfVectorizer(max_features=5000)
X_train = vectorizer.fit_transform(train_texts)
model = LogisticRegression()
model.fit(X_train, train_labels)

VADER (Valence Aware Dictionary for Sentiment Reasoning)

Rule-based sentiment analyzer tuned for social media:

from nltk.sentiment.vader import SentimentIntensityAnalyzer

analyzer = SentimentIntensityAnalyzer()
scores = analyzer.polarity_scores("This movie was surprisingly good!")
# {'neg': 0.0, 'neu': 0.464, 'pos': 0.536, 'compound': 0.7351}

Topic Modeling

Discover latent topics across a collection of documents:

Latent Dirichlet Allocation (LDA)

from sklearn.decomposition import LatentDirichletAllocation

lda = LatentDirichletAllocation(n_components=5, random_state=42)
lda.fit(X_tfidf)
for topic_idx, topic in enumerate(lda.components_):
    top_words = [feature_names[i] for i in topic.argsort()[:-11:-1]]
    print(f"Topic {topic_idx}: {' '.join(top_words)}")

BERTopic

Modern topic modeling using sentence embeddings and clustering:

from bertopic import BERTopic

topic_model = BERTopic()
topics, probs = topic_model.fit_transform(documents)
topic_model.visualize_topics()

Modern NLP Architectures

| Model | Year | Parameters | Key Innovation | |

NLP Pipeline

Natural Language Processing converts human language into structured representations that machines can process. A typical NLP pipeline includes these stages:

Text Preprocessing

Raw text requires cleaning before analysis:

• Tokenization — Split text into words or subword units. NLTK and spaCy provide language-aware tokenizers. Byte-pair encoding (BPE) and WordPiece handle out-of-vocabulary words by splitting into known subwords.
• Lowercasing — Reduces vocabulary size but may lose information (US vs us).
• Stop word removal — Remove common words (the, a, is) that add little meaning. Apply domain-specific stop word lists carefully.
• Stemming and lemmatization — Reduce words to base form. Stemming (Porter, Lancaster) chops suffixes crudely. Lemmatization uses vocabulary analysis (running → run).

Text Representation

Bag of Words (BoW) — Count word occurrences, ignoring order. Simple but loses context.

TF-IDF — Weight words by frequency in document vs rarity across documents.

Word Embeddings — Word2Vec, GloVe, FastText produce dense vectors where similar words have similar vectors.

Contextual Embeddings — BERT, RoBERTa, GPT produce different embeddings for same word in different contexts.

Common NLP Tasks

Text classification, Named Entity Recognition (NER), Part-of-Speech Tagging, Dependency Parsing, Machine Translation, Summarization, Question Answering.

Evaluation Metrics

Classification: accuracy, precision, recall, F1. Sequence labeling: exact match, token-level F1. Generation: BLEU, ROUGE, perplexity.

—|—|—|—| | BERT | 2018 | 110M–340M | Bidirectional pretraining, masked language model | | GPT | 2018 | 117M | Autoregressive, left-to-right language modeling | | RoBERTa | 2019 | 125M–355M | Optimized BERT pretraining | | T5 | 2019 | 60M–11B | Text-to-text unified framework | | GPT-3 | 2020 | 175B | In-context learning, scaling laws | | LLaMA | 2023 | 7B–65B | Open-source, efficient training | | GPT-4 | 2023 | 1.8T (estimated) | Multi-modal, instruction following |

Transfer Learning in NLP

Modern NLP uses a two-stage approach:

1. Pre-training — train a large model on massive text corpora (books, web pages) using self-supervised objectives
2. Fine-tuning — adapt the pre-trained model to a specific task with a small labeled dataset

from transformers import AutoModelForSequenceClassification, Trainer

model = AutoModelForSequenceClassification.from_pretrained(
    "bert-base-uncased", num_labels=2
)
trainer = Trainer(model=model, train_dataset=train_dataset)
trainer.train()

This transfer learning approach has achieved state-of-the-art results on virtually every NLP benchmark with minimal task-specific engineering.

Evaluation Metrics

For text generation, human evaluation remains the gold standard despite being expensive and slow. Automated metrics like BLEU and ROUGE correlate moderately with human judgment but can be gamed and miss semantic quality.

Advanced NLP Techniques

Sequence-to-Sequence Architectures

Sequence-to-sequence models form the backbone of modern NLP tasks like machine translation, summarization, and text generation. The encoder-decoder architecture uses an RNN, LSTM, or transformer encoder to process the input sequence into a context vector, then a decoder generates the output sequence token by token. Attention mechanisms improved seq2seq by allowing the decoder to focus on different parts of the input at each generation step, solving the bottleneck problem of fixed-length context vectors. The transformer architecture, introduced in the 2017 paper “Attention Is All You Need,” replaced recurrence entirely with multi-head self-attention and positional encodings, enabling parallel computation and capturing long-range dependencies more effectively.

Fine-Tuning and Transfer Learning

Pre-trained language models like BERT, GPT, RoBERTa, and T5 have transformed NLP by enabling transfer learning across tasks. These models are pre-trained on massive text corpora (Wikipedia, books, web crawl) using self-supervised objectives — masked language modeling for BERT, autoregressive language modeling for GPT. Fine-tuning adapts the pre-trained model to a specific task by continuing training on labeled data with a task-specific classification or generation head. This approach achieves state-of-the-art results with far less labeled data than training from scratch. Parameter-efficient fine-tuning methods like LoRA (Low-Rank Adaptation) and adapter layers reduce the computational cost by updating only a small fraction of model parameters.

Evaluation Metrics

NLP tasks require specialized evaluation metrics beyond standard accuracy. BLEU (Bilingual Evaluation Understudy) measures n-gram overlap between generated and reference text, commonly used for machine translation. ROUGE evaluates summarization by comparing recall of n-grams. METEOR considers synonymy and stemming for more robust evaluation. Perplexity measures how well a language model predicts a sample — lower perplexity indicates better prediction. For semantic similarity and embedding quality, cosine similarity and Spearman correlation against human judgments are standard. Human evaluation remains the gold standard for tasks involving creativity, tone, and factual accuracy, as automated metrics cannot capture all aspects of language quality.

Frequently Asked Questions

What is the difference between NLP and NLU? NLP (Natural Language Processing) encompasses all computational processing of human language, including generation, translation, and speech recognition. NLU (Natural Language Understanding) is a subfield focused specifically on comprehension — extracting meaning, intent, and entities from text. All NLU is NLP, but not all NLP requires deep understanding (e.g., text formatting or spell checking is NLP but not NLU).

How much data do I need to train an NLP model? Training a transformer from scratch requires millions to billions of tokens. For fine-tuning, as few as a few hundred labeled examples can produce good results with modern pre-trained models. For few-shot or zero-shot scenarios, large language models like GPT-3+ can perform tasks with just a few examples in the prompt or with no examples at all.

What is tokenization and why does it matter? Tokenization splits text into smaller units (words, subwords, or characters) that the model processes. Subword tokenization methods like Byte-Pair Encoding (BPE), WordPiece, and SentencePiece balance vocabulary size against handling out-of-vocabulary words. The choice of tokenizer affects model capacity: a larger vocabulary captures more semantics per token but increases embedding matrix size.

For a comprehensive overview, read our article on Bayesian Statistics Guide.

For a comprehensive overview, read our article on Big Data Tools Guide.