Here's an interesting bug I ran into recently: After transferring some files to my server via scp, I couldn't log in via SSH anymore - my key was rejected by sshd.

user@host: Permission denied (publickey).

The thing is, as luck would have it, I had previously set up a WebDAV server on the same machine - I've been using it for a number of things, for example as a makeshift synchronization solution for my KeePass database - and since it was just sshd acting up and everything else still worked fine, I could still access the entire file system via WebDAV.

My first thought was that some application had overwritten my authorized_keys file. Forgejo used to do this in the past every time it was updated, but I believe they fixed this. I certainly haven't had that issue for quite a while now. And sure enough, my key was still in there.

I tried creating a new key, putting it into authorized_keys and logging in using that, but to no avail. I tried logging in from a different machine - still nothing. I temporarily modified the sshd configuration to allow logging in via a password again, which worked flawlessly, but of course this was a workaround and not a solution. A reboot didn't help either (but I thought I might as well).

Luckily, when I say "server", what I really mean is a single-board computer sitting in my own four walls. So eventually, I gave in, took out the SD card and put it into my machine. I thought I'd run fsck for good measure, but it didn't report any issues. (And I hoped it wouldn't, the card is barely a few months old after all!)

But then I noticed something odd while cd'ing and ls'ing my way around the file system: The file system permissions of my home directory were set to rwxrwxrwx (777). I was certain that I didn't make this change - not least because ls helpfully highlighted the respective entry with a green background, something I no doubt would've noticed in the past.

If you're using SSH yourself, you may have run into a curious issue in the past: sshd telling you that the permissions of your private key file are too open. OpenSSH will refuse to use a key to connect to any server if said file is readable by any user but yourself:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0644 for 'id_ed25519' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
Load key "id_ed25519": bad permissions
user@host: Permission denied (publickey).

Well, it seems like there's a similar check on the server as well. As soon as I reverted the permissions of my home directory to rwx------ (700), everything went back to normal. And sure enough, between thousands of authentication failures from bots and scrapers, there was a revealing line in the OpenSSH logs:

sshd-session: Authentication refused: bad ownership or modes for directory /home/user

Alright, problem solved. But why did the permissions of my home directory change all of a sudden?

Well, let's replicate what happened right before all of this went down: the SCP file transfer. The exact command I used at the time (save for the redacted hostname) was:

$ scp -r . host:

This transfers everything in the current local directory to my home directory on the server. Another part that's going to be of importance is that, for reasons, the directory I wanted to transfer had the permissions of rwxrwxrwx (777). Can you tell where this is going already?

First, let's create the directory, as well as some sample files to transfer:

$ mkdir local_directory
$ chmod 777 local_directory
$ cd local_directory
$ touch test_file_1 test_file_2 test_file_3

Next is the destination directory. We could also use the home directory again, but I'm not going to take any more risks:

$ ssh host mkdir remote_directory
$ ssh host chmod 700 remote_directory

The moment of truth:

$ ssh host ls -ld remote_directory | cut -d " " -f 1
drwx------

$ scp -r . host:remote_directory
test_file_1	100%	0	0.0KB/s	00:00
test_file_2	100%	0	0.0KB/s	00:00
test_file_3	100%	0	0.0KB/s	00:00

$ ssh host ls -ld remote_directory | cut -d " " -f 1
drwxrwxrwx

It seems that OpenSSH's scp modifies the permissions of the transferred files and directories to match their local counterparts. This makes sense, except it also does so if the target directory already exists!

So, in short: When I transferred the files within a directory with the permissions of rwxrwxrwx (777) to my server, scp modified the permissions of the target (and, in this case, home) directory to match. Then, when I tried to log in via SSH, sshd determined that these permissions were too broad, too open to other users, to continue in a safe manner, and rejected my key.

I reported my findings to OpenSSH's bug tracker. Within hours, a fix was pushed, and the author told me that it's going to be included in the next release, 10.3. Kudos to them for such a quick response!

Links of interest: