Performance Testing Guide: Load, Stress & Scalability Testing

Performance testing evaluates how a system behaves under various load conditions. It identifies bottlenecks, validates reliability under peak traffic, and confirms that the application meets speed and scalability requirements. Unlike functional testing, which verifies correctness, performance testing answers questions about speed, stability, and capacity.

According to the Software Performance Testing Handbook by the IEEE Computer Society, performance issues are the leading cause of production incidents after deployment. The Google SRE book Site Reliability Engineering dedicates multiple chapters to load testing and capacity planning, emphasising that performance testing must be continuous rather than a one-time activity before release. Without regular performance testing, teams deploy blind to how their system will behave under real-world traffic.

Types of Performance Testing

Load Testing

Load testing simulates expected user traffic to measure response times, throughput, and resource utilisation under normal conditions. It answers the question: “Can the system handle the expected number of concurrent users within acceptable response times?” Load tests should model realistic user behaviour — reading pages, submitting forms, browsing search results — rather than hammering a single endpoint. The difference between a naive load test and a realistic one can be orders of magnitude in the conclusions it produces.

Stress Testing

Stress testing pushes the system beyond its expected capacity to determine the breaking point. It answers: “At what point does the system fail, and how does it fail — gracefully with degraded responses or catastrophically with crashes?” Stress testing is essential for capacity planning and determining auto-scaling triggers. The Google SRE book defines “stress testing” as finding the maximum load the system can handle while maintaining acceptable performance, then adding a safety margin for production operations.

Endurance Testing

Endurance testing, also called soak testing, runs a moderate load for an extended period — hours or days — to detect memory leaks, resource exhaustion, and performance degradation over time. Garbage collection patterns in JVM applications, connection pool leaks in database drivers, and file descriptor exhaustion in web servers are classic endurance issues. An application that performs perfectly for the first hour may crash on hour three due to a slow memory leak. The Software Performance Testing Handbook recommends endurance tests at 80 percent of expected peak load for at least 24 hours for production-critical applications.

Spike Testing

Spike testing rapidly increases load to extreme levels, simulating flash crowds, viral events, or sudden traffic surges after a marketing campaign. It tests auto-scaling mechanisms, load balancer configurations, and recovery behaviour when load subsides. The key metric in spike testing is recovery time — how quickly the system returns to normal after the spike passes. A system that handles the spike but fails to release resources afterwards may degrade over successive spike events.

Scalability Testing

Scalability testing measures how effectively the system handles increased load by adding resources — more application instances, larger databases, or increased bandwidth. It determines whether the system scales linearly, sub-linearly, or hits a hard ceiling.

Key Performance Metrics

Response time measures the duration from request initiation to full response. Aim for sub-500 milliseconds for API endpoints and sub-2 seconds for page loads. Throughput measures requests per second the system can handle. Error rate tracks the percentage of failed requests — aim for below 1 percent under normal load. Latency percentiles — P50, P95, and P99 — provide a more accurate picture than averages, as averages can hide severe outliers. A system with average response time of 200ms might have P99 of 5 seconds, meaning one in every hundred requests experiences a multi-second delay. Resource utilisation tracks CPU, memory, disk I/O, and network bandwidth saturation. Monitoring resource utilisation during tests helps identify the specific bottleneck component — CPU-bound applications benefit from faster processors while memory-bound applications require larger instances.

Establishing Performance Baselines

Before making any changes, establish baseline performance metrics for your system. Run load tests at multiple load levels — 25 percent, 50 percent, 75 percent, and 100 percent of expected peak — and record response time percentiles, throughput, and resource utilisation at each level. These baselines become the reference point for detecting regressions. Store baseline results in a time-series database and visualise trends over weeks and months. A gradual increase in P99 response time might indicate accumulating technical debt that requires investigation before it becomes a production incident.

Performance Testing Tools

Apache JMeter

JMeter is the most widely used open-source performance testing tool. It supports HTTP, JDBC, JMS, FTP, and dozens of other protocols through a plugin architecture. Its GUI-based test plan creation reduces the learning curve for non-developers.

k6

k6 is a modern load testing tool with a JavaScript scripting API. It integrates natively with CI/CD pipelines, supports threshold-based pass/fail criteria, and provides cloud execution for distributed load generation.

import http from 'k6/http';
import { check, sleep } from 'k6';
import { Rate } from 'k6/metrics';

const failureRate = new Rate('failed_requests');

export const options = {
    stages: [
        { duration: '2m', target: 100 },
        { duration: '5m', target: 100 },
        { duration: '2m', target: 200 },
        { duration: '5m', target: 200 },
        { duration: '2m', target: 0 }
    ],
    thresholds: {
        http_req_duration: ['p(95)<500', 'p(99)<1500'],
        failed_requests: ['rate<0.01']
    }
---;

export default function () {
    const res = http.get('https://api.example.com/products');
    check(res, {
        'status is 200': (r) => r.status === 200,
        'response time OK': (r) => r.timings.duration < 500
    });
    failureRate.add(res.status !== 200);
    sleep(1);
---

Locust

Locust uses Python and provides a real-time web UI for monitoring test progress. It supports distributed execution across multiple worker machines for generating high load from a single control node.

Performance Testing Process

Define clear performance goals before writing tests — target response times, throughput, and error rates. Create realistic test scenarios based on production traffic patterns. Set up an isolated test environment that mirrors production infrastructure. Execute a baseline test with minimal load, then incrementally increase load to target levels. Analyse results to identify bottlenecks, optimise, and repeat until goals are met. The iterative nature of performance testing is critical — the first test almost never meets goals, and the process of optimise-and-retest is where the real value emerges.

Common Bottlenecks

Database bottlenecks dominate performance issues — slow queries, missing indexes, connection pool exhaustion, and lock contention. Application code bottlenecks include inefficient algorithms, blocking I/O operations, memory leaks, and excessive garbage collection. Infrastructure bottlenecks include insufficient CPU, memory, or network bandwidth. Configuration issues — thread pool limits, connection timeouts, cache settings — are the easiest to fix but often the last to be diagnosed. A systematic approach to bottleneck identification uses the “universal scalability model” described by Neil Gunther in Guerrilla Capacity Planning: measure throughput as a function of concurrency, identify where throughput stops scaling linearly, and investigate the bottleneck at that point.

Best Practices

Test early and often. Performance is not something to fix at the end of the development cycle. Use production-like data volumes — small datasets hide performance issues. Monitor system resources during tests: CPU, memory, disk I/O, and network. Test from multiple geographic locations if your user base is global. Automate performance tests in CI/CD pipelines and fail builds that exceed response time thresholds.

Performance Metrics to Track

Key performance metrics include: latency (p50, p95, p99 percentiles), throughput (requests per second), error rate, resource utilization (CPU, memory, disk I/O, network), and saturation (queue depth, connection pool usage). P99 latency is more meaningful than average — it represents the worst-case experience for 99% of users. Monitor all metrics together — reducing latency by increasing resource utilization may lead to saturation. Set SLOs (Service Level Objectives) based on business requirements and alert when approaching breach.

Bottleneck Identification

Performance bottlenecks follow predictable patterns. CPU-bound: scaling horizontally or optimizing algorithms. Memory-bound: increasing capacity or fixing leaks. I/O-bound: adding caching, connection pooling, or async processing. Network-bound: optimizing payload size, adding CDN, or using compression. Thread/connection pool exhaustion: increasing pool size or reducing blocking operations. Use profiling tools (cProfile, py-spy, async-profiler) to identify the specific bottleneck rather than guessing.

FAQ

Q: What is the difference between load testing and stress testing?
A: Load testing validates behaviour under expected traffic. Stress testing pushes beyond expected limits to find the breaking point and observe failure behaviour.

Q: How many concurrent users should I test with?
A: Test at your expected peak concurrent user count plus a 50 percent buffer. If you expect 1,000 concurrent users, test at 1,500.

Q: What is a good P99 response time target?
A: For API endpoints, aim for P99 under 1 second. For web pages, aim for P99 under 3 seconds. Adjust based on your application domain.

Q: How often should I run performance tests?
A: Run smoke performance tests on every significant code change. Run full performance test suites before every release and on a weekly schedule.

Internal Links

• Explore Performance Testing Tools for an in-depth comparison of k6, Locust, and JMeter configurations.
• Read about CI/CD Testing to integrate performance checks into deployment pipelines.
• Learn about Testing Fundamentals for the role of non-functional testing in quality strategy.