SSH Key Authentication on Linux: Setup, Security, and Things That Go Wrong

By /

I’ve set up SSH key authentication on probably hundreds of servers at this point. It’s one of those tasks that should be simple and usually is — until it isn’t. This is a practical walkthrough of how I do it, with the security considerations that actually matter and the troubleshooting steps that have saved me more than once.

Why Keys Instead of Passwords

The reasons are well-known but worth stating briefly: SSH keys are asymmetric cryptography. The private key never leaves your machine. The server stores only your public key. Even if someone intercepts the connection, they can’t derive your private key. And brute-forcing a 256-bit ECDSA key is computationally infeasible, unlike passwords.

From a practical standpoint: keys are also faster (no typing) and easy to revoke (just delete the public key from authorized_keys). Once you’re used to key-based auth, going back to passwords feels wrong.

Generating a Key Pair

On your local machine:

ssh-keygen -t ed25519 -C "user@hostname-$(date +%Y%m)"

I prefer Ed25519 these days. It’s fast, secure, and produces shorter keys than RSA-4096 with equivalent security. The -C comment field is just a label — I include a date so I can tell when the key was created.

If you’re connecting to older systems that don’t support Ed25519:

ssh-keygen -t rsa -b 4096 -C "user@hostname-2024"

4096-bit RSA is fine. I don’t bother with 2048 anymore since the performance difference is negligible on modern hardware.

The keygen tool asks where to save the key — the default is ~/.ssh/id_ed25519 (and id_ed25519.pub for the public key). Then it asks for a passphrase. Use a passphrase. If your laptop is stolen and the private key has no passphrase, whoever has the laptop has access to every server that trusts that key. The passphrase is only decrypted locally by your SSH client — it never goes over the wire.

Use ssh-agent to avoid typing the passphrase every time:

eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_ed25519

On macOS, keychain integration handles this automatically. On Linux with a desktop environment, most setups start an agent automatically.

Copying the Public Key to the Server

The easiest way:

ssh-copy-id -i ~/.ssh/id_ed25519.pub user@server

This appends your public key to ~/.ssh/authorized_keys on the server. It also creates the directory and sets permissions correctly if they don’t exist — which is something I appreciate because wrong permissions on ~/.ssh are a common source of failures.

If ssh-copy-id isn’t available or you need to do it manually:

cat ~/.ssh/id_ed25519.pub | ssh user@server "mkdir -p ~/.ssh && chmod 700 ~/.ssh && cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"

The permissions matter. ~/.ssh must be 700 (owner-only rwx), and authorized_keys must be 600 (owner-only rw). If they’re world-readable, sshd ignores the file by default.

Disabling Password Authentication

Once keys are working, disable password auth. Edit /etc/ssh/sshd_config:

PasswordAuthentication no
ChallengeResponseAuthentication no
UsePAM yes

Then restart sshd:

sudo systemctl restart sshd

Before you do this, make absolutely sure you can log in with your key. Open a second terminal and test. If you lock yourself out, you’ll need console access (KVM, cloud provider console, etc.) to fix it.

I also typically set:

PermitRootLogin no

Or at minimum PermitRootLogin prohibit-password if some automation requires root access via key. Direct root login with a password should never be permitted.

Managing Multiple Keys

I have different keys for different purposes: personal projects, work servers, staging environments. SSH config lets you specify which key to use for which hosts:

# ~/.ssh/config

Host prod-server
  HostName 203.0.113.10
  User deploy
  IdentityFile ~/.ssh/id_ed25519_prod

Host staging-server
  HostName 203.0.113.11
  User deploy
  IdentityFile ~/.ssh/id_ed25519_staging

Host github.com
  IdentityFile ~/.ssh/id_ed25519_github
  AddKeysToAgent yes

With this, ssh prod-server automatically uses the right key and user. The AddKeysToAgent yes line automatically adds the key to the agent when it’s first used.

Authorized Keys File Options

The authorized_keys file supports options on each line that restrict what a key can do. This is useful for automation keys:

command="/usr/local/bin/backup.sh",no-pty,no-x11-forwarding,no-agent-forwarding ssh-ed25519 AAAA...

This key can only run /usr/local/bin/backup.sh — it can’t open an interactive shell. no-pty prevents tty allocation. This pattern is what I use for dedicated backup or monitoring keys: the key is limited to exactly one operation.

You can also restrict by source IP:

from="203.0.113.0/24",no-pty ssh-ed25519 AAAA...

This only accepts connections from that IP range, adding another layer of restriction.

Key Rotation

Keys should be rotated periodically or immediately after a compromise. My rough rotation schedule:

  • Production server keys: annually or when team members leave
  • Personal keys: when I get a new machine or laptop

Rotation is simple:

  1. Generate a new key pair
  2. Add the new public key to authorized_keys (the old one stays until you confirm the new one works)
  3. Test login with the new key
  4. Remove the old public key from authorized_keys
  5. Delete or archive the old private key

The tricky part with rotation is when one key is deployed across many servers. Keeping track of which keys are on which servers is where I’ve started maintaining a simple spreadsheet. For larger environments, tools like HashiCorp Vault or AWS Systems Manager handle key distribution and rotation properly.

If you’re also managing SSL/TLS certificates on those same servers, SSL certificate monitoring is another area where having a central inventory pays off — the same discipline of tracking expiry and rotation applies.

Troubleshooting

“Permission denied (publickey)”

This is the most common error. The cause is almost always one of:

  1. Wrong key: SSH is trying the wrong private key. Use -v flag to see which keys are being offered: ssh -v user@server
  2. Permissions on authorized_keys: must be 600, owner must match the user
  3. Permissions on ~/.ssh directory: must be 700
  4. The public key wasn’t added correctly: double-check ~/.ssh/authorized_keys on the server

I always start with ssh -vvv user@server when something isn’t working. The verbose output shows exactly which keys are being tried and where they’re failing.

“Server refused our key”

Usually a permissions issue on the server side. Run:

ls -la ~/.ssh/
ls -la ~/.ssh/authorized_keys

Also check the sshd logs:

sudo journalctl -u ssh -n 50
# or
sudo tail -f /var/log/auth.log

The log usually says exactly why the key was rejected.

“WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED”

This is sshd telling you the server’s host key has changed since your last connection. It could mean the server was rebuilt, or (in the scary version) someone is doing a man-in-the-middle attack.

If you’re sure the server was rebuilt:

ssh-keygen -R server-hostname

Then reconnect and accept the new key. If you’re not sure why the host key changed, investigate before accepting.

One More Thing: Authorized Keys for Multiple Users

On a fresh server, I often need to add keys for several team members. Instead of running ssh-copy-id multiple times, I keep a local file with all the public keys:

cat team-keys.pub | ssh admin@server "cat >> ~/.ssh/authorized_keys"

Or put each person’s key on a separate line in the authorized_keys file with a comment identifying who it belongs to. When someone leaves the team, you know exactly which line to remove.

SSH key authentication is one of those things that, once you’ve set it up correctly and understand how it works, becomes second nature. The important habits are: use passphrases, disable password auth, keep permissions correct, and document which keys go where.

Related Posts

Scroll to Top