🎉 NEW YEAR SALE! 40% OFF on Annual Premium+ Plan - Till 31st Dec! Use SUPERSALE40 Shop Now →

How to Set Up Passwordless SSH in RHEL: A Step-by-Step Guide

Published On: 20 March 2026

Objective

If you're still typing passwords every time you SSH into a server, you're doing it the hard way  and the less secure way. Key-based authentication isn't just a convenience thing. It's genuinely better cryptography. A well-generated Ed25519 key is harder to crack than any password a human would actually remember and type regularly. There's also the automation angle. The moment you try to run Ansible, set up CI/CD pipelines, or write scripts that move between machines, password auth becomes a blocker. You either store the password in plaintext somewhere (bad) or you set up keys properly (correct). This guide covers the correct path.

A few other reasons this matters in practice:

  • No password means nothing for a brute-force attack to guess against
  • Revoking access is as simple as deleting a line from authorized_keys - no password resets, no coordination
  • Meets PCI-DSS, HIPAA, and similar frameworks that require strong authentication methods
  • Logins are instant; no waiting on password prompts across a hundred servers

How It Actually Works 

SSH key auth is asymmetric cryptography in practice. You generate two mathematically linked keys - one private, one public. The public key goes on the server. The private key never leaves your machine. When you connect, the server issues a cryptographic challenge that only the holder of the matching private key can solve. Your client solves it, the server verifies the answer, and you're in. No password ever crosses the wire. Even if someone compromises the server completely, they still can't get your private key - it was never there.

The flow step by step:

  1. Client sends a connection request referencing your public key
  2. Server checks ~/.ssh/authorized_keys for that key
  3. Server sends an encrypted challenge only the private key can answer
  4. Client solves it using your private key
  5. Server verifies and grants access - no password involved

Which Key Type Should You Use?

 

Method Key Type Security When to Use
Password Login N/A Low Don't, if you can help it
RSA 2048-bit RSA Medium Only for legacy compatibility
RSA 4096-bit RSA Medium-High General use, older systems
ECDSA Elliptic Curve High FIPS-compliant environments
Ed25519 EdDSA Very High Recommended default
FIDO2 / Hardware Token Hardware Highest Privileged and admin access

 

Go with Ed25519 unless you have a specific reason not to. It's fast, compact, and the security margin is excellent. The one exception: if you're operating under FIPS 140-2/140-3 compliance - common in government and finance on RHEL - Ed25519 isn't FIPS-approved. Use ECDSA with nistp256 or nistp384 instead.

Step 1: Verify OpenSSH Is Running on Your RHEL Server

OpenSSH ships with RHEL by default, but it's worth confirming the service is actually running before you do anything else.

# Confirm openssh-server is installed
rpm -qa | grep openssh-server

# Install it if somehow missing
sudo dnf install openssh-server -y

# Enable and start sshd
sudo systemctl enable sshd -now

# Check it's actually running
sudo systemctl status sshd

# Open SSH through the firewall
sudo firewall-cmd -permanent -add-service=ssh
sudo firewall-cmd -reload

Step 2: Generate Your Key Pair (On Your Local Machine)

Run key generation on the client - the machine you're connecting from, not the server. This is a mistake people make once and remember forever.

Ed25519 (Recommended)

ssh-keygen -t ed25519 -C "your_email@example.com"

ECDSA (For FIPS Environments)

ssh-keygen -t ecdsa -b 384 -C "your_email@example.com"

RSA (Legacy Compatibility Only)

ssh-keygen -t rsa -b 4096 -C "your_email@example.com"

When the prompt asks for a save location, hitting Enter takes the default (~/.ssh/id_ed25519), which is fine for most setups. If you manage keys for multiple environments, use a descriptive custom path instead. On the passphrase question: for interactive logins, add one - it protects your private key if your laptop is ever stolen or compromised. For automation and scripts, leave it empty. Either way, the server connection itself is passwordless. The passphrase just guards the key file sitting on your disk.

After generation you'll have two files:

  • ~/.ssh/id_ed25519 - your private key; guard this carefully
  • ~/.ssh/id_ed25519.pub - your public key; this is what gets copied to servers

Step 3: Get the Public Key onto the Server

Three ways to do this, depending on what you have available.

Method 1: ssh-copy-id (Easiest)

ssh-copy-id -i ~/.ssh/id_ed25519.pub username@server_ip_or_hostname

This handles everything - appends the key to authorized_keys and sets correct permissions automatically. Use this when you still have password access to the server.

Method 2: Manual Copy

# Print your public key
cat ~/.ssh/id_ed25519.pub

# On the server, create the .ssh directory and add the key
mkdir -p ~/.ssh && chmod 700 ~/.ssh
echo "paste_your_public_key_here" >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys

Method 3: Via scp

# Copy the public key file to the server
scp ~/.ssh/id_ed25519.pub username@server_ip:~/uploaded_key.pub

# On the server, append and clean up
cat ~/uploaded_key.pub >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
rm ~/uploaded_key.pub

Step 4: Test the Connection

ssh username@server_ip_or_hostname

You should land in a shell without any password prompt. If you added a passphrase to your private key, you'll be asked for that - which is expected and correct. That's the passphrase protecting your local key file, not the server asking for a password. Still getting a password prompt? Jump to the troubleshooting section below.

Step 5: Harden Your SSH Config

Once key auth is confirmed working, disable password login entirely. This is the step that actually closes the door on brute-force attacks.

sudo vi /etc/ssh/sshd_config

Settings to apply:

# Kill password auth
PasswordAuthentication no

# No direct root login
PermitRootLogin no

# Block empty passwords
PermitEmptyPasswords no

# Restrict to specific users (optional but worth doing)
AllowUsers alice bob deploy_user

# Tighten key exchange algorithms
KexAlgorithms curve25519-sha256,ecdh-sha2-nistp384

# Cap failed auth attempts
MaxAuthTries 3

# Drop idle sessions after 10 minutes
ClientAliveInterval 300
ClientAliveCountMax 2

Before restarting sshd, validate the config first. A syntax error here with no fallback session open means you're locked out.

# Test config before applying
sudo sshd -t

# Restart SSH daemon
sudo systemctl restart sshd

Important: Keep an active SSH session open while testing these changes. If something breaks and you get disconnected, that existing session stays alive and you can fix it. Starting fresh without a fallback is how people end up filing support tickets at 2am.

Managing Multiple Keys With ~/.ssh/config

Once you're managing more than two or three servers with different keys, typing out full hostnames and identity file paths gets old fast. The SSH config file fixes this.

# ~/.ssh/config

# Production web server
Host prod-web
    HostName 192.168.1.10
    User deploy
    IdentityFile ~/.ssh/id_ed25519_prod
    Port 22

# Staging
Host staging
    HostName 192.168.1.20
    User admin
    IdentityFile ~/.ssh/id_ed25519_staging

# Internal server reached through a jump host
Host internal-db
    HostName 10.0.0.5
    User dbadmin
    IdentityFile ~/.ssh/id_ed25519_internal
    ProxyJump bastion_user@bastion.example.com

With this in place, connections become:

ssh prod-web
ssh staging
ssh internal-db

Using Passwordless SSH With Ansible

Ansible assumes key-based SSH auth. Once your keys are deployed, ad-hoc commands and playbooks just work:

# inventory.ini
[webservers]
server1.example.com
server2.example.com

# Quick connectivity check
ansible webservers -i inventory.ini -m ping

# Run a playbook
ansible-playbook -i inventory.ini site.yml

For deploying keys at scale, use the authorized_key module rather than doing it by hand on each server:

- name: Deploy SSH public key to all servers
  hosts: all
  tasks:
    - name: Add SSH key
      ansible.posix.authorized_key:
        user: deploy
        state: present
        key: "{{ lookup('file', '~/.ssh/id_ed25519.pub') }}"

SELinux and SSH: When Things Break for No Obvious Reason

File permissions look right, the key is in authorized_keys, and you're still getting a password prompt. SELinux is often the culprit - especially if the home directory is in a non-standard path.

# Look for SELinux denials related to sshd
sudo ausearch -m avc -ts recent | grep sshd

# Restore the correct SELinux context on .ssh
sudo restorecon -Rv ~/.ssh

# Verify the context on authorized_keys
ls -laZ ~/.ssh/authorized_keys
# Should show: system_u:object_r:ssh_home_t:s0

Troubleshooting: Still Getting a Password Prompt?

Work through this list before assuming something unusual is wrong. The answer is almost always one of these:

  • Wrong permissions: ~/.ssh must be 700; authorized_keys must be 600
  • Wrong ownership: both must be owned by the actual user, not root
  • SELinux context: run sudo restorecon -Rv ~/.ssh
  • PubkeyAuthentication not enabled: check that PubkeyAuthentication yes is set in sshd_config (it's the default, but double-check)
  • AuthorizedKeysFile pointing somewhere wrong: verify the directive in sshd_config

SSH verbose mode tells you exactly where it's failing:

ssh -vvv username@server_ip

Check sshd logs on the server side:

sudo journalctl -u sshd -n 50
# or on older setups
sudo tail -f /var/log/secure

Fix permissions quickly:

chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
chown -R $USER:$USER ~/.ssh

Verify the private key actually matches the public key on the server:

ssh-keygen -y -f ~/.ssh/id_ed25519
# Compare the output against the entry in the server's authorized_keys

Key Management Habits Worth Building

  • Rotate keys at least once a year  and immediately when someone with key access leaves the team
  • One key per person per environment shared private keys are an audit and revocation nightmare
  • Protect private keys with a passphrase use ssh-agent to cache it so you're not typing it constantly
  • Audit authorized_keys regularly stale keys from departed employees or decommissioned systems should go
  • Use a bastion host for external access internal servers shouldn't be directly reachable from the internet
  • Look into short-lived SSH certificates tools like HashiCorp Vault can issue time-limited certificates instead of managing long-lived keys
  • Keep a key inventory know which keys have access to what, or you'll find out the hard way during an incident
  • Monitor /var/log/secure unusual login patterns show up here before they become bigger problems

Conclusion

The whole process takes maybe 15 minutes if your environment is straightforward. Generate a key pair on your client, push the public key to the server, lock down sshd_config to kill password auth, check your SELinux contexts, and test. That's it. On RHEL 10 specifically, the default crypto policy is already tighter than RHEL 9, and post-quantum key exchange support is there if you want to opt in early. Neither requires extra configuration out of the box - they're just worth knowing about as your environment evolves. Once keys are deployed and password auth is off, the attack surface on your SSH setup shrinks dramatically. Brute-force attempts become irrelevant. Automation works cleanly. And you stop typing passwords into terminals a dozen times a day. To practicse hands-on, try lab Configure and Secure SSH on LinuxCert.Guru