I found the problem in Raspbian Buster. /etc/ssh/ssh_config has two items activated:
batchmode yes
connecttimeout 1000

If you comment out both of these by inserting a # in from of them, and reboot you Pi, all will be good.