SSH was working but now "Permission denied (publickey)"

[jimjamz]

SSH has been working successfully for the past few months between my Raspberry Pi and other devices. However, since yesterday, every device I used to SSH to the Raspberry Pi, now results in:

Code: Select all

Permission denied (publickey)

The Raspberry Pi has a fixed LAN IP so is always the same, as do all the other devices on the same WLAN that try to connect to it. Password authentication is off so only keys can be used to connect.
I plugged the Pi's SD card into my Linux laptop and inspected the authorized_keys file and ensure all key entries are correct, which they are.

I SSHed to the Pi verbosely:

Code: Select all

ssh -v -p 29922 jimjamz@192.168.0.5 -i ~/.ssh/my-keyfile
OpenSSH_6.7p1 Debian-5+deb8u3, OpenSSL 1.0.1t  3 May 2016
debug1: Reading configuration data /home/jimjamz/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to 192.168.0.5 [192.168.0.5] port 29922.
debug1: Connection established.
debug1: identity file /home/jimjamz/.ssh/my-keyfile type 1
debug1: key_load_public: No such file or directory
debug1: identity file /home/jimjamz/.ssh/my-keyfile-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.7p1 Debian-5+deb8u3
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4p1 Raspbian-10+deb9u1
debug1: match: OpenSSH_7.4p1 Raspbian-10+deb9u1 pat OpenSSH* compat 0x04000000
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr umac-64-etm@openssh.com none
debug1: kex: client->server aes128-ctr umac-64-etm@openssh.com none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA 22:90:11:f1:79:0f:7c:22:37:75:90:f0:23:97:f4:0f
debug1: Host '[192.168.0.5]:29922' is known and matches the ECDSA host key.
debug1: Found key in /home/jimjamz/.ssh/known_hosts:1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/jimjamz/.ssh/my-keyfile
debug1: Authentications that can continue: publickey
debug1: No more authentication methods to try.
Permission denied (publickey).

The Pi must be at fault as they keys of several connecting devices can't all be wrong.

As a test, I changed the sshd_config file by plugging the SD into my laptop and changing PasswordAuthentication to yes. When I tried to connecting to the Pi again, I could see it was now asking for a password, so I can see at least the changes I make to the SD via my laptop are being persisted when plugged into the Pi.

I also managed to take a copy of the /var/log/auth.log from the SD card after my attempts to SSH onto the Pi:

Code: Select all

Nov 26 12:16:01 FLORENCE-PI CRON[1506]: pam_unix(cron:session): session closed for user jimjamz
Nov 26 12:17:01 FLORENCE-PI CRON[1516]: pam_unix(cron:session): session opened for user jimjamz by (uid=0)
Nov 26 12:17:01 FLORENCE-PI CRON[1515]: pam_unix(cron:session): session opened for user root by (uid=0)
Nov 26 12:17:01 FLORENCE-PI CRON[1516]: pam_unix(cron:session): session closed for user jimjamz
Nov 26 12:17:01 FLORENCE-PI CRON[1515]: pam_unix(cron:session): session closed for user root
Nov 26 12:18:01 FLORENCE-PI CRON[1558]: pam_unix(cron:session): session opened for user jimjamz by (uid=0)
Nov 26 12:18:01 FLORENCE-PI CRON[1558]: pam_unix(cron:session): session closed for user jimjamz
Nov 26 12:19:01 FLORENCE-PI CRON[1567]: pam_unix(cron:session): session opened for user jimjamz by (uid=0)
Nov 26 12:19:02 FLORENCE-PI CRON[1567]: pam_unix(cron:session): session closed for user jimjamz

Strange, because I don't remember trying to login as root.

[SurferTim]

Have you tried refreshing the key? On the Linux box:

Code: Select all

ssh-keygen -f "/home/user/.ssh/known_hosts" -R 192.168.0.5

Change user and the IP to match your system.

[jimjamz]

Have you tried refreshing the key? On the Linux box:

Code: Select all

ssh-keygen -f "/home/user/.ssh/known_hosts" -R 192.168.0.5

Change user and the IP to match your system.

There is an added caveat here. The password for the local user on the Raspberry Pi has been forgotten. I haven't authenticated as a local user in so long and had only used it as a headless server, that I never used the local authentication to remember it. Yes, foolish, I know. Any other possibilities?

I plugged the Pi SD into my Linux laptop and renamed the known_hosts to known_hosts.old but that still results in the same outcome.

[cpc464]

Try checking the permissions on ~/.ssh and the key files, both on the Pi and your local server. SSH will fail if they are too loose.

[jimjamz]

As a workaround, I have a shell script file on the Pi that gets run every minute as a cron job. I plugged the Pi SD into my laptop and modified the shell script to run:

Code: Select all

ssh-keygen -f "/home/jimjamz/.ssh/known_hosts" -R 192.168.0.5

However, it did not solve the problem.