Deep RL: DQN, PPO, SAC, and Multi-Agent Algorithms

Deep reinforcement learning combines the representational power of neural networks with reinforcement learning algorithms, enabling agents to solve complex decision-making problems that were previously intractable. Deep Q-Networks (DQN), Proximal Policy Optimization (PPO), and Soft Actor-Critic (SAC) are the three most influential algorithms in modern deep RL, each representing a different approach to the core challenge of learning effective policies from high-dimensional observations.

The field has progressed rapidly since DQN achieved human-level performance on Atari games in 2015. AlphaGo used deep RL to defeat the world champion in Go — a game once thought decades away from machine mastery. PPO has become the default algorithm for continuous control tasks in robotics and simulation. SAC adds entropy maximization for better exploration, and multi-agent RL extends these ideas to systems with multiple interacting agents.

Deep Q-Networks

DQN (Mnih et al., 2015) was the first algorithm to successfully combine neural networks with Q-learning, using raw pixel inputs from Atari games. Its key innovations addressed the fundamental instability of combining non-linear function approximation with off-policy learning.

Experience Replay Implementation

Experience replay stores transitions in a buffer and samples randomly to decorrelate consecutive experiences:

import random
import numpy as np
from collections import deque

class ReplayBuffer:
    def __init__(self, capacity: int = 100000):
        self.buffer = deque(maxlen=capacity)

    def push(self, state, action, reward, next_state, done):
        self.buffer.append((state, action, reward, next_state, done))

    def sample(self, batch_size: int):
        batch = random.sample(self.buffer, batch_size)
        states, actions, rewards, next_states, dones = map(np.stack, zip(*batch))
        return states, actions, rewards, next_states, dones

    def __len__(self):
        return len(self.buffer)

Target Network and Training Loop

A separate target network provides stable Q-value targets. The target network parameters are copied from the online network at regular intervals:

import torch
import torch.nn as nn
import torch.nn.functional as F

class DQNAgent:
    def __init__(self, state_dim, action_dim, hidden_dim=256):
        self.q_network = nn.Sequential(
            nn.Linear(state_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, action_dim)
        )
        self.target_network = nn.Sequential(*[copy.deepcopy(layer) for layer in self.q_network])
        self.target_network.eval()
        self.optimizer = torch.optim.Adam(self.q_network.parameters(), lr=3e-4)
        self.gamma = 0.99
        self.buffer = ReplayBuffer(capacity=100000)
        self.batch_size = 64
        self.target_update_freq = 1000
        self.steps = 0

    def update(self):
        if len(self.buffer) < self.batch_size:
            return

        states, actions, rewards, next_states, dones = self.buffer.sample(self.batch_size)

        states = torch.FloatTensor(states)
        actions = torch.LongTensor(actions)
        rewards = torch.FloatTensor(rewards)
        next_states = torch.FloatTensor(next_states)
        dones = torch.FloatTensor(dones)

        # Current Q values
        current_q = self.q_network(states).gather(1, actions.unsqueeze(1)).squeeze()

        # Target Q values using frozen target network
        with torch.no_grad():
            max_next_q = self.target_network(next_states).max(1)[0]
            target_q = rewards + self.gamma * max_next_q * (1 - dones)

        # Loss and update
        loss = F.mse_loss(current_q, target_q)
        self.optimizer.zero_grad()
        loss.backward()
        torch.nn.utils.clip_grad_norm_(self.q_network.parameters(), 1.0)
        self.optimizer.step()

        # Periodic target network update
        self.steps += 1
        if self.steps % self.target_update_freq == 0:
            self.target_network.load_state_dict(self.q_network.state_dict())

Prioritized experience replay (Schaul et al., 2016) improves on uniform sampling by sampling more frequently from transitions with larger TD errors — the ones the agent has most to learn from.

Proximal Policy Optimization

PPO (Schulman et al., 2017) has become the default algorithm for continuous control due to its stability, simplicity, and strong performance. It introduces a clipped surrogate objective that prevents policy updates from being too large:

def compute_ppo_loss(actor, critic, states, actions, old_log_probs, advantages, returns, epsilon=0.2):
    # New log probabilities under current policy
    action_probs = actor(states)
    dist = torch.distributions.Categorical(action_probs)
    log_probs = dist.log_prob(actions)

    # Probability ratio
    ratio = torch.exp(log_probs - old_log_probs)

    # Clipped surrogate objective
    surr1 = ratio * advantages
    surr2 = torch.clamp(ratio, 1.0 - epsilon, 1.0 + epsilon) * advantages
    actor_loss = -torch.min(surr1, surr2).mean()

    # Value function loss
    values = critic(states).squeeze()
    critic_loss = F.mse_loss(values, returns)

    # Entropy bonus (optional, encourages exploration)
    entropy = dist.entropy().mean()
    entropy_coef = 0.01

    total_loss = actor_loss + 0.5 * critic_loss - entropy_coef * entropy
    return total_loss

The clipping mechanism restricts the probability ratio to [1-epsilon, 1+epsilon], meaning the new policy cannot deviate too far from the old policy in a single update. This prevents the destructive policy updates that plagued earlier policy gradient methods. PPO typically performs multiple epochs of mini-batch updates on the same trajectory data (typically 3-10 epochs), making it sample-efficient compared to pure on-policy methods.

PPO Training Loop

class PPOAgent:
    def __init__(self, state_dim, action_dim):
        self.actor = self._build_network(state_dim, action_dim)
        self.critic = self._build_network(state_dim, 1)
        self.optimizer = torch.optim.Adam(
            list(self.actor.parameters()) + list(self.critic.parameters()),
            lr=3e-4
        )
        self.gamma = 0.99
        self.gae_lambda = 0.95
        self.ppo_epochs = 10
        self.clip_epsilon = 0.2

    def update(self, trajectories):
        states, actions, rewards, dones, values, log_probs = trajectories

        # Compute advantages using Generalized Advantage Estimation
        advantages = []
        gae = 0
        for t in reversed(range(len(rewards))):
            if t == len(rewards) - 1:
                next_value = 0  # Terminal state
            else:
                next_value = values[t + 1]
            delta = rewards[t] + self.gamma * next_value * (1 - dones[t]) - values[t]
            gae = delta + self.gamma * self.gae_lambda * (1 - dones[t]) * gae
            advantages.insert(0, gae)

        advantages = torch.FloatTensor(advantages)
        returns = advantages + torch.FloatTensor(values)
        advantages = (advantages - advantages.mean()) / (advantages.std() + 1e-8)

        # Multiple PPO epochs
        for _ in range(self.ppo_epochs):
            loss = compute_ppo_loss(
                self.actor, self.critic,
                states, actions, log_probs,
                advantages, returns, self.clip_epsilon
            )
            self.optimizer.zero_grad()
            loss.backward()
            self.optimizer.step()

Soft Actor-Critic

SAC (Haarnoja et al., 2018) adds entropy maximization to the objective function. The agent learns to maximize both expected return and policy entropy, encouraging exploration and preventing premature convergence to narrow policies:

def sac_update(actor, qf1, qf2, value, target_value, batch, alpha, gamma=0.99):
    states, actions, rewards, next_states, dones = batch

    # Compute target Q value
    with torch.no_grad():
        next_actions, next_log_probs = actor.sample(next_states)
        next_q = torch.min(
            qf1(next_states, next_actions),
            qf2(next_states, next_actions)
        )
        target_q = rewards + gamma * (1 - dones) * (next_q - alpha * next_log_probs)

    # Q-function losses (clipped double Q-learning)
    qf1_loss = F.mse_loss(qf1(states, actions), target_q)
    qf2_loss = F.mse_loss(qf2(states, actions), target_q)

    # Policy loss with entropy
    new_actions, log_probs = actor.sample(states)
    q_values = torch.min(
        qf1(states, new_actions),
        qf2(states, new_actions)
    )
    actor_loss = (alpha * log_probs - q_values).mean()

    # Temperature parameter alpha (auto-tuned)
    alpha_loss = -(alpha.log() * (log_probs.detach() + target_entropy).mean())

SAC uses two Q-functions and takes the minimum to reduce overestimation bias — a problem where Q-learning systematically overestimates action values. The entropy coefficient alpha can be automatically tuned to a target entropy value.

Multi-Agent Reinforcement Learning

In multi-agent environments, each agent’s policy must account for the non-stationarity caused by other agents learning and changing their behavior simultaneously. MADDPG (Lowe et al., 2017) uses a centralized critic that observes all agents’ actions, with decentralized actors that only use local observations:

class MADDPG:
    def __init__(self, n_agents, obs_dims, action_dims):
        self.agents = []
        for i in range(n_agents):
            agent = DDPGAgent(obs_dims[i], action_dims[i])
            self.agents.append(agent)

    def update(self, experiences, agent_id):
        # Centralized critic sees all observations and actions
        all_obs = torch.cat([exp['obs'] for exp in experiences], dim=-1)
        all_actions = torch.cat([exp['action'] for exp in experiences], dim=-1)

        # Update critic with global information
        target_q = self.compute_target_q(experiences, agent_id)
        current_q = self.agents[agent_id].critic(all_obs, all_actions)
        critic_loss = F.mse_loss(current_q, target_q)

        # Actor uses only local observation
        local_obs = experiences[agent_id]['obs']
        actor_loss = -self.agents[agent_id].critic(
            all_obs,
            self.combine_actions(experiences, agent_id)
        ).mean()

Hyperparameter Sensitivity

Deep RL algorithms are notoriously sensitive to hyperparameters. The learning rate, discount factor, entropy coefficient, network architecture, and buffer size all interact in complex ways. Key guidelines:

• Learning rate: 3e-4 is a good default for Adam across DQN, PPO, and SAC
• Discount factor (gamma): 0.99 for long-horizon tasks, 0.9-0.95 for shorter episodes
• Network width: 256-512 units per hidden layer for most continuous control tasks
• Replay buffer size: 1e5 for simple tasks, 1e6 for complex environments
• Target network update frequency: every 1000 steps for DQN

Population-based training and Bayesian optimization are more effective than manual tuning for scaling RL hyperparameter search.

FAQ

Which deep RL algorithm should I start with? PPO is the most practical starting point — it works well across both discrete and continuous action spaces, requires minimal hyperparameter tuning, and is relatively stable. DQN is limited to discrete actions. SAC generally outperforms PPO on continuous control but is more complex.

How do I handle continuous action spaces? DQN cannot handle continuous actions directly. Use PPO or SAC for continuous control. Both output parameters of a probability distribution (typically Gaussian) from which actions are sampled. For high-dimensional continuous actions, SAC with automatic entropy tuning is often the best choice.

What is the sample efficiency of these algorithms? From most to least sample efficient: SAC > DQN > PPO. SAC and DQN are off-policy (reuse past experiences). PPO is on-policy (uses each experience once). However, PPO’s multi-epoch updates within each trajectory partially offset the sample efficiency gap.

Why does RL training oscillate or diverge? Common causes: learning rate too high, reward scaling too large, insufficient exploration, or catastrophic forgetting where the agent loses previously learned skills. Use gradient clipping, reward normalization, and maintain multiple parallel environments to stabilize training.

Can I train deep RL on a laptop CPU? Simple environments like CartPole, MountainCar, and LunarLander train in minutes on CPU. Atari games benefit from GPU (hours per experiment). MuJoCo continuous control tasks need GPU for reasonable iteration speed.

Internal Links

• For foundational RL concepts and Q-learning, see Reinforcement Learning: Agents, Rewards, and Policy.
• For the neural network architectures underlying these algorithms, read Neural Networks: Perceptrons, Activation, and Backpropagation.
• For choosing the right framework to implement deep RL, see PyTorch vs TensorFlow: Choosing Your Framework.

Related Concepts and Further Reading

Understanding rl deep dive requires familiarity with several interconnected ideas and principles that together form a complete picture. Exploring these related concepts deepens your knowledge and provides context that makes the core material more meaningful and applicable. Each concept builds on the others, creating a web of understanding that supports deeper learning and practical application. Taking time to explore how these elements connect reveals patterns that accelerate comprehension and retention of new information.

The relationship between rl deep dive and adjacent fields is worth particular attention. Many of the most important insights emerge at the boundaries between disciplines, where ideas from different areas combine to create new approaches and solutions that neither field could produce alone. Exploring these connections pays dividends in both breadth and depth of understanding, revealing patterns and principles that might otherwise remain hidden from view. Cross-disciplinary knowledge is increasingly valued as problems become more complex and interconnected.

For those looking to go beyond introductory material, several excellent resources provide deeper treatment of specific aspects of rl deep dive. Academic journals, industry publications, authoritative reference works, and online courses each offer different perspectives and levels of detail. The key is to match your reading to your current learning goals and build knowledge progressively, focusing on quality over quantity in your study materials. A well-chosen resource that matches your current level is worth more than dozens of resources that are too basic or too advanced.