Python Requests Guide: HTTP, Sessions, and REST APIs

Python Python 8 min read 1612 words Beginner ExcellentWiki Editorial Team

The requests library is the de facto standard for HTTP communication in Python. It provides a clean, intuitive API that handles everything from simple GET requests to complex session management, file uploads, OAuth authentication, and streaming responses — without the boilerplate that Python’s standard library urllib requires. This guide covers the requests library comprehensively, from installation through advanced patterns used in production API clients.

Introduction to Requests

The requests library was created by Kenneth Reitz with the philosophy of providing “HTTP for Humans.” Its design prioritizes readability and simplicity. A GET request is requests.get(url) — not a multi-line incantation of URL opening, connection handling, and response parsing. Requests is built on top of urllib3, which handles connection pooling, retries, and SSL verification. The library abstracts these concerns behind a simple API while exposing configuration options when needed.

pip install requests

Requests supports all standard HTTP methods: GET, POST, PUT, PATCH, DELETE, HEAD, and OPTIONS. Each method has a corresponding function: requests.get(), requests.post(), and so on. All return a Response object. The Response object provides the server’s response data through several attributes. response.text returns the response body as a string (decoded using the detected encoding). response.content returns raw bytes. response.json() parses the body as JSON and raises ValueError on invalid JSON. response.status_code provides the HTTP status code. response.headers is a case-insensitive dictionary of response headers.

The requests official documentation describes the API in detail, including the comprehensive set of exceptions available in requests.exceptions.

Making HTTP Requests

GET requests retrieve resources. Query parameters are passed as a dictionary:

import requests

response = requests.get(
    "https://api.github.com/search/repositories",
    params={"q": "python", "sort": "stars", "per_page": 10}
)
data = response.json()

POST requests create resources. The json parameter automatically serializes the payload and sets the Content-Type header:

response = requests.post(
    "https://api.example.com/todos",
    json={"title": "New Task", "completed": False}
)
created_todo = response.json()

The data parameter sends form-encoded data (application/x-www-form-urlencoded). The files parameter sends multipart form data for file uploads:

# Form data
response = requests.post(
    "https://api.example.com/login",
    data={"username": "alice", "password": "secret"}
)

# File upload
with open("report.pdf", "rb") as f:
    response = requests.post(
        "https://api.example.com/upload",
        files={"file": ("report.pdf", f, "application/pdf")}
    )

PUT replaces resources entirely; PATCH applies partial updates:

# PUT replaces the resource
requests.put(
    "https://api.example.com/todos/1",
    json={"title": "Updated", "completed": True, "userId": 1}
)

# PATCH updates specified fields only
requests.patch(
    "https://api.example.com/todos/1",
    json={"completed": True}
)

Understanding when to use PUT vs PATCH is important for REST API design. PUT replaces the entire resource — omitted fields may be reset to defaults. PATCH applies partial updates, leaving unspecified fields unchanged.

Working with Headers

Custom headers are passed as a dictionary to the headers parameter. Common use cases include setting User-Agent, Accept, Content-Type, and Authorization:

headers = {
    "User-Agent": "MyApp/1.0 (myapp@example.com)",
    "Accept": "application/json",
    "Accept-Language": "en-US",
    "Cache-Control": "no-cache",
---
response = requests.get("https://api.example.com/data", headers=headers)

The Session object persists headers across multiple requests. Set default headers on the session to avoid repeating them:

session = requests.Session()
session.headers.update({
    "User-Agent": "MyApp/1.0",
    "Accept": "application/json",
---)

Response headers are accessible as a case-insensitive dictionary. Common patterns include checking Content-Type, pagination links, and rate limit headers:

response = requests.get("https://api.github.com/search/repositories")
content_type = response.headers.get("content-type")
rate_limit_remaining = response.headers.get("x-ratelimit-remaining")

The case-insensitive dictionary behavior means response.headers['Content-Type'] and response.headers['content-type'] both work. This matches HTTP header specification where header names are case-insensitive.

Authentication Methods

Requests supports multiple authentication mechanisms through the auth parameter. HTTP Basic Authentication is the simplest:

from requests.auth import HTTPBasicAuth

response = requests.get(
    "https://api.example.com/protected",
    auth=HTTPBasicAuth("username", "password")
)

# Shorthand — Requests creates HTTPBasicAuth automatically
response = requests.get(
    "https://api.example.com/protected",
    auth=("username", "password")
)

Bearer token authentication, common with OAuth 2.0 and JWT, uses a header:

headers = {"Authorization": "Bearer eyJhbGciOiJIUzI1NiJ9..."}
response = requests.get("https://api.example.com/me", headers=headers)

For API keys sent as query parameters:

response = requests.get(
    "https://api.example.com/data",
    params={"api_key": "abc123def456"}
)

More complex authentication flows — OAuth 2.0, NTLM, AWS Signature V4 — are supported through the requests-oauthlib, requests-ntlm, and requests-aws4auth companion libraries. The requests-oauthlib documentation covers OAuth 2.0 flows including authorization code grant and client credentials grant.

Webhook Handlers and Callback Receivers

Many modern APIs use webhooks to push events to your application rather than requiring polling. A webhook receiver is an HTTP server endpoint that accepts POST requests from the API provider. Flask or FastAPI are commonly used to implement webhook receivers:

from flask import Flask, request, jsonify
import hmac
import hashlib

app = Flask(__name__)
WEBHOOK_SECRET = b"your-webhook-secret-key"

@app.route("/webhooks/stripe", methods=["POST"])
def handle_webhook():
    signature = request.headers.get("Stripe-Signature")
    payload = request.get_data()

    expected_sig = hmac.new(
        WEBHOOK_SECRET, payload, hashlib.sha256
    ).hexdigest()

    if not hmac.compare_digest(signature, expected_sig):
        return jsonify({"error": "Invalid signature"}), 401

    event = request.json
    process_event(event)
    return jsonify({"status": "ok"}), 200

Webhook receivers must respond quickly — typically within 5-10 seconds — or the provider may retry or drop the connection. Heavy processing should be offloaded to a task queue like Celery or RQ. Idempotency handling prevents duplicate processing when providers retry due to network issues.

Sessions and Connection Pools

requests.Session provides connection reuse, cookie persistence, and default configuration across multiple requests. Without a session, each request opens a new TCP connection — adding latency for TLS handshakes and DNS resolution.

session = requests.Session()
session.auth = ("username", "password")
session.headers.update({"User-Agent": "MyApp/1.0"})

# Login — cookie stored in session
session.post("https://api.example.com/login", json={"user": "alice"})

# Subsequent requests use stored cookies
profile = session.get("https://api.example.com/me")
orders = session.get("https://api.example.com/orders")

Session-mounted adapters provide per-host configuration. The HTTPAdapter with Retry configuration is the standard pattern for production API clients:

from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

retry_strategy = Retry(
    total=3,
    backoff_factor=0.5,
    status_forcelist=[429, 500, 502, 503, 504],
    allowed_methods=["GET", "POST", "PUT", "PATCH"]
)

adapter = HTTPAdapter(max_retries=retry_strategy)

session = requests.Session()
session.mount("https://", adapter)
session.mount("http://", adapter)

The backoff_factor controls delay between retries using exponential backoff. With a factor of 0.5, retries wait 0.5, 1.0, and 2.0 seconds. The status_forcelist defines which HTTP status codes trigger retries — 429 (rate limited) and 5xx (server errors) are standard. The urllib3 Retry documentation covers all available parameters.

Error Handling and Timeouts

Timeout configuration prevents requests from hanging indefinitely. Always set timeouts in production code:

# Single timeout for the entire request (connect + read)
response = requests.get("https://api.example.com", timeout=5)

# Separate connect and read timeouts
response = requests.get(
    "https://api.example.com",
    timeout=(3.5, 10)  # 3.5s connect, 10s read
)

The connect timeout limits how long to wait for the TCP connection. The read timeout limits how long to wait between bytes of the response body. For streaming responses, the read timeout restarts on each received chunk. Comprehensive error handling separates connection errors, timeout errors, HTTP errors, and encoding errors:

from requests.exceptions import (
    RequestException, ConnectionError, Timeout, HTTPError
)

try:
    response = requests.get(
        "https://api.example.com/data",
        timeout=(3.5, 10)
    )
    response.raise_for_status()  # Raises HTTPError for 4xx/5xx
    data = response.json()

except ConnectionError:
    # DNS failure, refused connection, network unreachable
    logger.error("Failed to connect to API server")

except Timeout:
    # Request exceeded timeout
    logger.error("API request timed out")

except HTTPError as e:
    # 4xx or 5xx response
    logger.error("HTTP %d: %s", e.response.status_code, e.response.text)

except ValueError as e:
    # Invalid JSON response
    logger.error("Invalid JSON response: %s", e)

except RequestException as e:
    # Catch-all for any requests-related error
    logger.error("Request failed: %s", e)

The exception hierarchy in requests.exceptions allows fine-grained handling. ConnectionError, Timeout, and HTTPError all inherit from RequestException, so the catch-all at the end handles any unexpected requests errors.

Streaming Responses

For large responses, streaming avoids loading the entire body into memory. Use stream=True and iter_content or iter_lines:

# Stream a file download to disk
response = requests.get(
    "https://example.com/large-file.zip",
    stream=True
)
response.raise_for_status()

with open("large-file.zip", "wb") as f:
    for chunk in response.iter_content(chunk_size=8192):
        f.write(chunk)

For streaming API responses such as Server-Sent Events (SSE):

response = requests.get(
    "https://api.example.com/events",
    stream=True
)
for line in response.iter_lines():
    if line:
        event = json.loads(line.decode("utf-8"))
        process_event(event)

Production REST API Client Pattern

Production API clients typically wrap requests in a client class that encapsulates URL construction, authentication, and error handling:

class APIClient:
    BASE_URL = "https://api.example.com/v2"

    def __init__(self, api_key: str):
        self._session = requests.Session()
        self._session.headers.update({
            "Authorization": f"Bearer {api_key}",
            "Accept": "application/json",
            "User-Agent": "MyApp/1.0"
        })
        retry = Retry(total=3, backoff_factor=0.5)
        adapter = HTTPAdapter(max_retries=retry)
        self._session.mount("https://", adapter)

    def get(self, path: str, **kwargs) -> dict:
        response = self._session.get(f"{self.BASE_URL}{path}", **kwargs)
        response.raise_for_status()
        return response.json()

    def post(self, path: str, data: dict = None) -> dict:
        response = self._session.post(f"{self.BASE_URL}{path}", json=data)
        response.raise_for_status()
        return response.json()

Paginated APIs are handled with generators that abstract page iteration:

def paginate(self, path: str, per_page: int = 100):
    page = 1
    while True:
        data = self.get(path, params={"page": page, "per_page": per_page})
        if not data:
            break
        yield from data
        page += 1

This client pattern is used by major API wrapper libraries including the GitHub Python SDK and Stripe’s Python library. It centralizes configuration, reduces code duplication, and makes testing easier through dependency injection of the session.

FAQ

Do I need to install requests separately?

Yes. Requests is not part of the Python standard library. Install it with pip install requests. In Python 3, the standard library’s urllib.request is available but has a more verbose API.

How do I handle rate limiting with requests?

Use the Retry adapter with status_forcelist including 429. For proactive rate limiting, implement a rate limiter using time.sleep() between requests or use pyrate-limiter or requests-ratelimiter libraries.

Can requests handle HTTPS and SSL certificates?

Yes. Requests verifies SSL certificates by default. Use verify=False to disable verification (not recommended for production) or verify="/path/to/cert.pem" for custom certificate authorities.

How do I upload multiple files with requests?

Pass a list of tuples to the files parameter: files=[("images", ("photo1.jpg", f1)), ("images", ("photo2.jpg", f2))]. Multiple files with the same form field name are sent as an array.

What is the difference between response.text and response.content?

response.text is the response body decoded as a string using the detected encoding (from Content-Type header or chardet detection). response.content is the raw bytes. Use .content when working with binary data or when you need to control encoding manually.

Related: Explore Python error handling for robust exception management patterns. Learn Python concurrency for concurrent HTTP requests with asyncio. Study Python design patterns for client wrapper patterns used in production API clients.

Share this article

X LinkedIn Facebook Email