How to Set Up SSH Keys: A Complete Guide

SSH keys are more secure and convenient than passwords for remote server access. Once configured, you can log into servers, push to GitHub, and run remote commands without typing a password each time.

How SSH Keys Work

┌─────────────────┐         ┌─────────────────┐
│   Your Machine   │         │   Server         │
│                   │         │                  │
│  Private Key     │───────→│  Public Key      │
│  (secret.key)    │  Auth  │  (authorized_keys)│
└─────────────────┘         └─────────────────┘

You generate a pair of keys:

• Private key — stays on your machine. Never share it.
• Public key — goes on every server you access.

The Cryptography Behind SSH Keys

SSH key pairs use asymmetric (public-key) cryptography. When you connect to a server, the server generates a random challenge, encrypts it with your public key, and sends it to your client. Your client decrypts it with your private key and sends the decrypted challenge back. Only someone holding the private key can successfully respond. The private key never travels over the network — it’s used only locally for decryption.

ED25519, introduced in OpenSSH 6.5, uses elliptic-curve cryptography. It’s faster than RSA (roughly 3x faster for key generation and signing), produces smaller keys (256-bit vs 4096-bit), and is considered mathematically stronger at equivalent sizes. RSA 4096 is still secure but requires more CPU time for authentication, noticeable when you’re running many SSH commands in a script.

Step 1: Generate a Key Pair

# Generate an ED25519 key (recommended)
ssh-keygen -t ed25519 -C "your@email.com"

# Or RSA (for older systems)
ssh-keygen -t rsa -b 4096 -C "your@email.com"

You’ll be asked:

• Where to save — default ~/.ssh/id_ed25519 is fine
• Passphrase — strongly recommended (encrypts your key)

Passphrase Best Practices

A passphrase encrypts your private key on disk using symmetric encryption (AES-256 by default in OpenSSH). Without a passphrase, anyone who gains access to your ~/.ssh/id_ed25519 file immediately has SSH access to every server that trusts that public key. With a passphrase, the key is useless without the password.

Use a password manager to store your passphrase. 16+ characters with mixed case and symbols is ideal. If you’re worried about typing it frequently, the SSH agent caches your passphrase for the session duration, so you only enter it once per login.

Step 2: Add to SSH Agent

# Start the agent
eval "$(ssh-agent -s)"

# Add your key
ssh-add ~/.ssh/id_ed25519

Persistent SSH Agent Configuration

To avoid running ssh-agent manually every time, configure your shell to start it automatically:

# Add to ~/.zshrc or ~/.bashrc
if [ -z "$SSH_AUTH_SOCK" ]; then
  eval "$(ssh-agent -s)" > /dev/null
  ssh-add ~/.ssh/id_ed25519 2> /dev/null
fi

For macOS users, the system keychain can store passphrases persistently:

ssh-add --apple-use-keychain ~/.ssh/id_ed25519

This stores your passphrase in the macOS keychain so you never type it again, even across reboots. Linux users can achieve similar persistence with ssh-agent in .profile and desktop keyring integrations like gnome-keyring or seahorse.

Step 3: Copy to Remote Server

Option A: Using ssh-copy-id

ssh-copy-id user@server-ip

Option B: Manual

cat ~/.ssh/id_ed25519.pub | ssh user@server-ip "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"

Option C: Manual via SSH

# On your machine, view the key
cat ~/.ssh/id_ed25519.pub
# Copy the output

# On the server
echo "your-public-key-here" >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
chmod 700 ~/.ssh

Why Permission Matter

The .ssh directory and its contents must have restricted permissions. OpenSSH is strict about this — if permissions are too loose, it refuses to use the keys:

chmod 700 ~/.ssh          # directory: owner read/write/execute only
chmod 600 ~/.ssh/authorized_keys  # file: owner read/write only
chmod 644 ~/.ssh/id_ed25519.pub   # public key: can be world-readable

If you get “Permissions 0644 for ‘authorized_keys’ are too open,” it means the file is readable by group or others. OpenSSH requires private keys and authorized_keys to be accessible only by the file owner.

Step 4: Test the Connection

ssh user@server-ip

If everything works, you should be logged in without a password prompt.

Debugging Connection Issues

If the connection fails or still asks for a password, run SSH in verbose mode:

ssh -vvv user@server-ip

Look for lines like:

• Authenticated with partial success — your key was offered but rejected
• Permission denied (publickey) — the server has no matching public key
• debug1: Offering public key — shows which key was tried

Verbose output is invaluable for diagnosing the exact failure point.

Setting Up for GitHub

# 1. Generate a key if you don't have one
ssh-keygen -t ed25519 -C "your@email.com"

# 2. Add to SSH agent
eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_ed25519

# 3. Copy the public key
cat ~/.ssh/id_ed25519.pub

# 4. Add to GitHub:
#    Settings → SSH and GPG Keys → New SSH Key → Paste → Save

# 5. Test
ssh -T git@github.com
# Output: "Hi username! You've successfully authenticated..."

GitHub Key Requirements

GitHub requires keys to be at least 2048-bit RSA or any ED25519 key. If you’re using an older key (1024-bit RSA), generate a new one. GitHub supports up to 100 keys per account, so you can have separate keys for work and personal machines.

SSH Config File

Simplify connections with ~/.ssh/config:

Host github.com
  HostName github.com
  User git
  IdentityFile ~/.ssh/id_ed25519

Host myserver
  HostName 192.168.1.100
  User alice
  Port 2222
  IdentityFile ~/.ssh/id_ed25519

Host *.internal
  User admin
  IdentityFile ~/.ssh/internal-key

Now you can type ssh myserver instead of ssh alice@192.168.1.100 -p 2222.

Advanced Config Options

The SSH config file supports many useful options:

Host bastion
  HostName bastion.company.com
  User alice
  ForwardAgent yes        # forward SSH agent for jump host access
  ServerAliveInterval 60  # keep connection alive (prevent timeout)
  ProxyJump bastion       # connect through bastion host

ForwardAgent yes is powerful but risky — it allows the remote server to use your local SSH keys. Only enable it for trusted servers. For accessing machines behind a bastion, ProxyJump (or ProxyCommand) automatically routes connections through the jump host.

Wildcard and Multi-Host Config

Host *.aws.example.com
  User ec2-user
  IdentityFile ~/.ssh/aws-key
  StrictHostKeyChecking no
  UserKnownHostsFile /dev/null

Wildcard patterns match multiple hosts, reducing duplication. Use StrictHostKeyChecking no only for ephemeral environments like auto-scaling groups where host keys change frequently.

Key Management

# List loaded keys
ssh-add -l

# List ALL keys in your .ssh directory
ls ~/.ssh/

# View a public key
cat ~/.ssh/id_ed25519.pub

# Generate a new key for a different service
ssh-keygen -t ed25519 -f ~/.ssh/github-key -C "github"

Multi-Key Strategy

Manage multiple keys for different contexts:

~/.ssh/
├── id_ed25519          # personal GitHub
├── id_ed25519.pub
├── work-github         # work GitHub
├── work-github.pub
├── server-prod         # production servers
├── server-prod.pub
├── config              # maps keys to hosts
└── authorized_keys     # (on servers only)

Each key should have a descriptive filename using -f at generation time. The config file maps them to specific hosts, preventing you from accidentally using your personal key on a work server.

Security Best Practices

• Use ED25519 instead of RSA — it’s faster and more secure
• Always use a passphrase — encrypts your private key at rest
• Never share your private key — it’s called private for a reason
• Set permissions correctly:

chmod 700 ~/.ssh
chmod 600 ~/.ssh/id_ed25519       # private key
chmod 644 ~/.ssh/id_ed25519.pub   # public key (can be world-readable)
chmod 600 ~/.ssh/authorized_keys  # on servers
chmod 644 ~/.ssh/config

• Use different keys for different services — if one is compromised, the others are safe
• Rotate keys periodically — generate new keys every 1-2 years
• Add key expiration — ssh-keygen supports expiry dates

Key Rotation Procedure

When rotating keys, follow this process:

1. Generate a new key pair locally
2. Add the new public key to all servers and services (while keeping the old one)
3. Test the new key works (ssh -i ~/.ssh/id_ed25519_new user@server)
4. Remove the old public key from all servers and services
5. Delete the old private key from your machine

This ensures you always have a fallback if the new key doesn’t work on a particular server.

Recovering from a Lost Private Key

1. Generate a new key pair
2. Log into each server (via password or console) and add the new public key
3. Remove the old public key from all servers
4. Update GitHub/GitLab/other services

Preventing Key Loss

Key loss is preventable. Back up your .ssh directory to an encrypted USB drive or a password manager that supports file attachments. Never back up unencrypted keys to cloud storage. Services like Keybase offer encrypted file storage suitable for key backups.

SSH Tunneling and Port Forwarding

SSH keys also enable secure tunnels for accessing services behind firewalls:

# Local port forwarding: access remote DB locally
ssh -L 5432:localhost:5432 user@server

# Dynamic port forwarding (SOCKS proxy)
ssh -D 8080 user@server
# Configure your browser to use localhost:8080 as SOCKS proxy

# Remote port forwarding: expose local server
ssh -R 8080:localhost:3000 user@server

Local forwarding maps a local port to a remote service. Dynamic forwarding creates a SOCKS proxy that routes all traffic through the SSH connection. Remote forwarding makes a local service accessible from the remote server.

FAQ

What is the difference between SSH keys and GPG keys?

SSH keys are used for authentication and secure connections. GPG keys are used for signing and encryption (e.g., signing Git commits). They use different key formats and serve different purposes, though you can use the same key material for both in some configurations.

Can I use the same SSH key on multiple machines?

Yes, you can copy your private key to multiple machines, but this weakens security. Instead, generate separate keys for each machine and add each public key to your servers. The ~/.ssh/config file keeps everything organized.

How do I disable password authentication for SSH?

On your server, set PasswordAuthentication no in /etc/ssh/sshd_config and restart the SSH service. Only users with SSH keys will be able to log in. Always keep a root SSH session open when making this change to avoid locking yourself out.

What is SSH agent forwarding?

SSH agent forwarding lets you use your local SSH keys on a remote server. When enabled, the remote server forwards authentication requests back to your local machine. Use it sparingly because any user with root access on the remote server can use your forwarded agent.

How do I customize SSH connection timeouts?

Add ServerAliveInterval 60 and ServerAliveCountMax 3 to your SSH config. This sends a keepalive every 60 seconds, and drops the connection after 3 missed responses (180 seconds). Prevents SSH sessions from hanging indefinitely.

Related: Learn Linux file permissions and bash scripting.