Statistics for Data Science: Key Concepts Explained
Statistics provides the mathematical foundation for data science. Every prediction, every confidence interval, every hypothesis test relies on statistical principles. You do not need a PhD in statistics to be an effective data scientist, but you do need a solid grasp of core concepts and the intuition to apply them correctly.
Descriptive Statistics
Descriptive statistics summarize and describe the main features of a dataset.
Measures of Central Tendency
The mean is the arithmetic average. It is sensitive to outliers — a single extreme value can distort it significantly.
The median is the middle value when data is sorted. It is robust to outliers. Use the median instead of the mean when your data is skewed or contains outliers.
The mode is the most frequent value. It is useful for categorical data where mean and median are meaningless.
Measures of Dispersion
Variance measures the average squared deviation from the mean. Higher variance means data points are more spread out.
Standard deviation is the square root of variance, expressed in the same units as the original data. It is the most commonly used measure of spread.
The interquartile range is the range between the 25th and 75th percentiles. It is robust to outliers and drives box plot whiskers.
Skewness measures asymmetry. Positive skew means a long right tail. Negative skew means a long left tail. Knowing the skew helps choose appropriate statistical methods.
Probability Fundamentals
Probability quantifies uncertainty. It is the language of risk, prediction, and inference.
Basic Rules
Probability ranges from 0 (impossible) to 1 (certain). The probability of an event and its complement sum to 1. For mutually exclusive events, the probability of either occurring is the sum of their individual probabilities.
Conditional Probability
Conditional probability measures the likelihood of an event given that another event has occurred. It is the foundation of Bayesian thinking and many machine learning algorithms. The formula is P(A|B) = P(A and B) / P(B).
Bayes’ Theorem
Bayes’ Theorem updates probabilities based on new evidence. It describes the relationship between prior beliefs and posterior probabilities after observing data:
P(H|E) = P(E|H) × P(H) / P(E)
Bayesian methods are increasingly important in modern data science, especially when data is scarce or uncertainty quantification is critical.
Key Probability Distributions
The normal distribution is the most important distribution in statistics. It is symmetric, bell-shaped, and defined by its mean and standard deviation. The central limit theorem — which states that the sum or mean of many independent random variables is approximately normal — makes it central to statistical inference.
The binomial distribution models the number of successes in a fixed number of independent trials. Each trial has two outcomes with constant probability. Applications include A/B testing and survey analysis.
The Poisson distribution models the number of events occurring in a fixed interval given a constant average rate. Used for modeling counts — customer arrivals, defects, website visits.
Hypothesis Testing
Hypothesis testing determines whether observed effects are statistically significant or likely due to random chance.
Null and Alternative Hypotheses
The null hypothesis (H0) represents the default assumption — typically no effect or no difference. The alternative hypothesis (H1) represents what you want to prove. You never accept the null hypothesis; you either reject it or fail to reject it.
p-Values
The p-value is the probability of observing results at least as extreme as yours, assuming the null hypothesis is true. A small p-value (typically below 0.05) suggests the null hypothesis is unlikely. Common misinterpretation: the p-value is NOT the probability that the null hypothesis is true.
Type I and Type II Errors
Type I error (false positive) — rejecting a true null hypothesis. The significance level α controls this risk. Type II error (false negative) — failing to reject a false null hypothesis. Statistical power (1 - β) measures the ability to detect real effects.
Common Tests
The t-test compares means between two groups. Independent t-test for separate groups. Paired t-test for before-after measurements.
ANOVA compares means across three or more groups to determine whether at least one group differs significantly.
The chi-square test assesses association between categorical variables by comparing observed frequencies to expected frequencies under independence.
Regression Analysis
Regression models the relationship between a dependent variable and one or more independent variables.
Linear regression assumes a linear relationship: y = β0 + β1x + ε. Ordinary least squares estimates coefficients by minimizing the sum of squared residuals.
Multiple regression extends linear regression to multiple predictors. Coefficients represent the effect of each predictor while controlling for others.
Logistic regression models binary outcomes using the logit function. The output is a probability between 0 and 1.
Interpreting Regression Output
The R-squared value measures the proportion of variance explained by the model. Coefficients represent the change in the dependent variable for a one-unit change in the predictor. p-values for coefficients test whether the predictor has a statistically significant effect.
Correlation
Correlation measures the strength and direction of a linear relationship between two variables. The Pearson correlation coefficient ranges from -1 (perfect negative) to +1 (perfect positive), with 0 indicating no linear relationship.
Correlation does not imply causation. Two variables can correlate without one causing the other. Spurious correlations arise from lurking variables, coincidence, or selection bias. Establishing causation requires controlled experiments or advanced causal inference methods.
Sampling
Data science rarely has access to the entire population. Sampling allows inference about a population from a subset.
Random sampling ensures each member has equal selection probability. Stratified sampling divides the population into subgroups and samples from each. The choice of sampling method affects the validity of conclusions.
Practical Application
Statistics is not an abstract exercise. Every data science project involves statistical thinking. When you calculate an average, you use descriptive statistics. When you assess whether a feature improves model performance, you use hypothesis testing. When you quantify prediction uncertainty, you use probability distributions.
The key is not memorizing formulas but developing intuition. Statistics gives you the framework to answer questions rigorously and communicate uncertainty honestly.
Practical Applications of Statistics
Bayesian vs. Frequentist Statistics
Both Bayesian and frequentist frameworks are essential tools in data science. Frequentist statistics treats probability as the long-run frequency of events — parameters are fixed unknown constants, and confidence intervals contain the true parameter in a certain proportion of repeated experiments. Bayesian statistics treats probability as a degree of belief — parameters are random variables with prior distributions, updated to posterior distributions after observing data. Bayesian methods excel when prior information is available, when interpretability matters (credible intervals are more intuitive than confidence intervals), and when dealing with complex hierarchical models. Practical choices: use frequentist methods for straightforward A/B testing with large samples, and Bayesian methods for sequential testing, small samples, or when incorporating domain knowledge into the analysis.
Effect Size and Practical Significance
Statistical significance does not imply practical significance. With large sample sizes, even trivially small effects become statistically significant. Effect size measures — Cohen’s d for mean differences, Pearson’s r for correlations, and odds ratios for binary outcomes — quantify the magnitude of an effect independent of sample size. A statistically significant result with a very small effect size may not warrant business action. Conversely, a non-significant result with a moderate effect size may indicate insufficient power rather than no effect. Best practice is to report both the p-value and the effect size with a confidence interval.
Multiple Testing Correction
When testing many hypotheses simultaneously, the probability of at least one false positive increases substantially. Running 20 independent tests at α = 0.05 gives a family-wise error rate of 1 - (0.95)^20 ≈ 64%. The Bonferroni correction (divide α by the number of tests) is simple but conservative, reducing power. The Holm-Bonferroni method provides a step-down procedure with more power. The Benjamini-Hochberg procedure controls the false discovery rate (FDR) rather than the family-wise error rate and is preferred for exploratory analyses and high-dimensional data. For modern machine learning with thousands of feature significance tests, FDR control is the standard approach.
Frequently Asked Questions
What is the difference between descriptive and inferential statistics? Descriptive statistics summarize and describe features of a dataset — mean, median, standard deviation, percentiles, and visualizations like histograms and box plots. Inferential statistics draws conclusions about a population from a sample — hypothesis tests, confidence intervals, and regression models. Data science uses both: descriptive for exploratory analysis and inferential for drawing conclusions from experiments.
When should I use a parametric vs. nonparametric test? Use parametric tests (t-test, ANOVA, Pearson correlation) when the data approximately follows a known distribution (typically normal) and sample sizes are adequate. Use nonparametric tests (Mann-Whitney U, Kruskal-Wallis, Spearman correlation) when assumptions are violated, data is ordinal, or sample sizes are very small. Nonparametric tests have lower power but make fewer assumptions.
What is the central limit theorem and why does it matter? The central limit theorem states that the sampling distribution of the mean approaches a normal distribution as sample size increases, regardless of the population distribution. This justifies using normal-theory inference (z-tests, t-tests, confidence intervals) even when the underlying data is not normally distributed, provided the sample is large enough (typically n > 30).
How do I choose a significance level? The conventional α = 0.05 represents a 5% risk of false positive. Choose a stricter α (e.g., 0.01) when false positives are costly — medical trials, regulatory decisions. Choose a more lenient α (e.g., 0.10) for exploratory analyses where false negatives are more costly than false positives. Always pre-register the significance level rather than choosing after seeing results.
For a comprehensive overview, read our article on Bayesian Statistics Guide.
For a comprehensive overview, read our article on Big Data Tools Guide.