SSH Woes


23 posts
by floogle » Wed Jan 09, 2013 12:02 am
The problem:
Remote SSH login is very slow, and SSH connection are very slow and unreliable.

The system:
Model B Pi
Fully up to date Rasbian
High quality SD card
Power supply says 5v 1A
RJ45 LAN connected
HDMI output works
Powered USB hub with external HD (which is also powered) (and sometimes mouse and keyboard).
Using stock memory split and no over-clocking.

A Bit More Description
I have used the Pi for general fiddling and playing around with, and it works nicely. While trying to set it up to run headless I ran into the problems.

Logging in via SSH (from a winbox using putty) takes a very long time, anywhere between 1 second and 30 seconds. I read somewhere that adding an entry to /etc/hosts for the connecting machine might help. It didn't.

Once I manage to connect via SSH the connection will seem to be working fine before it randomly hangs. I don't get any kind of timeout message in Putty, it just stops doing anything and I don't know why. Sometimes the connection recovers if I wait long enough (minutes).

I have tried adding a keepalive in putty, and fiddling with sshd's keepalive setting, but no joy.

After reading around it seems that a lot of problems are caused by poor power supplies, but since my Pi has been running just fine until now, even under heavy load, I consider that unlikely in this case.

I can ping the Pi from windows without any problem, the Pi has no network access problems (web browsing works just fine).

Watching /var/log/messages and /var/log/dmesg during a SSH session shows nothing interesting.

Does anybody have any advice for me?
Posts: 11
Joined: Tue Jan 08, 2013 11:32 pm
by k4gbb » Wed Jan 09, 2013 1:59 am
I have the same problem from time to time.
It happens with almost any and all of my SSh connections.
No matter if they are across the LAN or Inet. It's a latency thing.
The Grass may be greener on the other side of the fence, but it still has to be mowed.
User avatar
Posts: 50
Joined: Sun Aug 12, 2012 5:33 am
Location: Dunnellon, FL USA - EL88tx
by bgreat » Wed Jan 09, 2013 3:27 am
I use two Pi's exclusively via SSH with minimal lag and no hang issues. One is connected via wireless adapter and the other is hard wired via the onboard Ethernet port. My first thought for any issue similar to this is to verify the processor load via top or uptime to confirm no task is causing you to have communication issues.

If you can not isolate it to another application causing a resource issue, then the next thing I would confirm is the power supply. I know you do not notice any network issue when connecting from the Pi, but this is still my number one item for testing. I had one Pi working flawlessly for two months until I switched power adapters. I tested it for several hours and it appeared to be working without issue, then the next day I had a corrupted SD card. Fixed the card and a day later the same thing happened. Switched back the power supply and it has been running flawlessly again.

Enjoy!
Bill
User avatar
Posts: 235
Joined: Mon Jan 23, 2012 2:09 pm
by sharow » Wed Jan 09, 2013 3:46 am
Try these
1. run "ifconfig" and see what errors/dropped value after slow ssh connection.
2. install ethtool, and see what output 'ethtool eth0' and 'ethtool -k eth0'
3. Disable IPv6
ArchLinuxARM@RaspberryPi
User avatar
Posts: 9
Joined: Wed Oct 17, 2012 7:24 am
Location: Tokyo
by milhouse » Wed Jan 09, 2013 3:58 am
DNS issues can slow down SSH authentication - are you sure your network is resolving local hostnames correctly?

Try adding "UseDNS no" to /etc/ssh/sshd_config and see if it has any effect.
Posts: 555
Joined: Mon Jan 16, 2012 12:59 pm
by floogle » Wed Jan 09, 2013 1:03 pm
k4gbb wrote:I have the same problem from time to time.
It happens with almost any and all of my SSh connections.
No matter if they are across the LAN or Inet. It's a latency thing.


I can rule out general network latency as all other network services are running just fine, including SSH between other machines on the same network. As a precaution I am minimising network traffic while I am fiddling with this issue.

bgreat wrote:... My first thought for any issue similar to this is to verify the processor load via top or uptime to confirm no task is causing you to have communication issues.

If you can not isolate it to another application causing a resource issue, then the next thing I would confirm is the power supply. I know you do not notice any network issue when connecting from the Pi, but this is still my number one item for testing. I had one Pi working flawlessly for two months until I switched power adapters. I tested it for several hours and it appeared to be working without issue, then the next day I had a corrupted SD card. Fixed the card and a day later the same thing happened. Switched back the power supply and it has been running flawlessly again.


I had considered that perhaps it was a processor load issue with the Pi, but no significant processes are running, I am logged in the CLI, with no X running. Uptime shows loads of 0.03, 0.16, 0.09. This is fairly typical as the device is essentially idle. Running top has the top process as using the most CPU by far, at about 3%. During a remote SSH login there is a small spike where the sshd process jumps to the top of the list for a moment or two.

I will still consider the power supply as a potential cause, but it still seems unlikely since the device is essentially idle it won't be drawing much power, it doesn't even need to power peripherals as they are all on a powered USB hub. But even if it was a power supply issue it strikes me as odd that it would manifest itself through SSH, especially when everything else is working, even under high load.

sharow wrote:Try these
1. run "ifconfig" and see what errors/dropped value after slow ssh connection.
2. install ethtool, and see what output 'ethtool eth0' and 'ethtool -k eth0'
3. Disable IPv6


ifconfig shows 987 RX errors, and 143 TX errors on eth0. I don't recall if these numbers are within acceptable limits or not. This numbers increase at a rate of roughly 50 per minute (estimated).

EDIT: I mis-read the output (silly me), the number are the packet counts, there are zero errors.

Currently looking at the output from "sudo ethtool eth0". I am typing this from a different machine so I can't paste the output. Nothing is jumping out at me however. The link is reported at running at 10mb/s full duplex, I don't know why it isn't running at 100.

As I write I luckily had an SSH session that lasted long enough to run the command:
Code: Select all
pi@raspberrypi ~ $ sudo ethtool eth0
Settings for eth0:
        Supported ports: [ TP MII ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
        Supported pause frame use: No
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
        Advertised pause frame use: Symmetric Receive-only
        Advertised auto-negotiation: Yes
        Link partner advertised link modes:  10baseT/Half 10baseT/Full
        Link partner advertised pause frame use: Symmetric
        Link partner advertised auto-negotiation: Yes
        Speed: 10Mb/s
        Duplex: Full
        Port: MII
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: on
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes


Code: Select all
pi@raspberrypi ~ $ sudo ethtool -k eth0
Features for eth0:
rx-checksumming: on
tx-checksumming: on
        tx-checksum-ipv4: off [fixed]
        tx-checksum-unneeded: off [fixed]
        tx-checksum-ip-generic: on
        tx-checksum-ipv6: off [fixed]
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
scatter-gather: off
        tx-scatter-gather: off [fixed]
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
        tx-tcp-segmentation: off [fixed]
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp6-segmentation: off [fixed]
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: off [requested on]
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: on
loopback: off [fixed]


This output doesn't really mean much to me I'm afraid.

I just disabled IPv6 and rebooting now.
Last edited by floogle on Wed Jan 09, 2013 1:18 pm, edited 1 time in total.
Posts: 11
Joined: Tue Jan 08, 2013 11:32 pm
by floogle » Wed Jan 09, 2013 1:10 pm
sharow wrote:3. Disable IPv6


Reboot complete, this didn't make any difference.

milhouse wrote:DNS issues can slow down SSH authentication - are you sure your network is resolving local hostnames correctly?

Try adding "UseDNS no" to /etc/ssh/sshd_config and see if it has any effect.


I'm not aware that my local network has any hostnames, although I did try adding an entry for the windows laptop to /etc/hosts and it made no difference.

Adding "UseDNS no" to the sshd config made no difference either.

At this point I would like to say that I do appreciate everyone's help!
Posts: 11
Joined: Tue Jan 08, 2013 11:32 pm
by bgreat » Wed Jan 09, 2013 1:17 pm
If you are using a 100/1000 network switch, I would be investigating the 10 Mbit connection rate. This would seem to indicate you are connected to a 10 Mbit network hub. Coupled with the TX and RX errors, I would suspect the network connection. If you are connected to a 10 Mbit hub, or as an experiment, try setting your Pi to use half duplex instead of full duplex.

ethtool -s eth0 speed 10 duplex half

And, try a new network cable or connection point.

Enjoy!
Bill
Last edited by bgreat on Wed Jan 09, 2013 1:20 pm, edited 1 time in total.
User avatar
Posts: 235
Joined: Mon Jan 23, 2012 2:09 pm
by RaTTuS » Wed Jan 09, 2013 1:18 pm
floogle wrote:...

ifconfig shows 987 RX errors, and 143 TX errors on eth0. I don't recall if these numbers are within acceptable limits or not. This numbers increase at a rate of roughly 50 per minute (estimated).
...

this^
Code: Select all
 uprecords
     #               Uptime | System                                     Boot up
----------------------------+---------------------------------------------------
->   1    52 days, 02:34:35 | Linux 3.2.27+             Sun Nov 18 10:41:10 2012
 ifconfig  eth0
eth0      Link encap:Ethernet  HWaddr b8:27:eb:ec:4e:32
          inet addr:192.168.8.33  Bcast:192.168.31.255  Mask:255.255.224.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:57731929 errors:0 dropped:0 overruns:0 frame:0
          TX packets:515135 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:4027635809 (3.7 GiB)  TX bytes:39207000 (37.3 MiB)

you should have no errors really - try another cable
http://www.catb.org/esr/faqs/smart-questions.html <- ask smart Questions
"That's not right, the badgers have moved the goalposts."
1QC43qbL5FySu2Pi51vGqKqxy3UiJgukSX - Prosliver FTW
User avatar
Posts: 5333
Joined: Tue Nov 29, 2011 11:12 am
Location: North West UK
by floogle » Wed Jan 09, 2013 1:23 pm
Sorry guys, I mis-read the output from ifconfig, there are no errors!

I edited my earlier post so I don't cause any more confusion!

The hub the cable is connected to is a 10/100 switch, I don't know why the Pi connects at 10. The speed doesn't really bother me though, just this SSH issue.

As an extra note: when I connect my windows machine using the same cable it connects at 100 and has no network problems with it.

I just tried setting the Pi to half-duplex: no change.
Posts: 11
Joined: Tue Jan 08, 2013 11:32 pm
by bgreat » Wed Jan 09, 2013 2:10 pm
Now that you report no errors on the interface, I would not expect the half duplex to help. Have you tried forcing 100 Mbit?

Enjoy!
Bill
User avatar
Posts: 235
Joined: Mon Jan 23, 2012 2:09 pm
by cyrano » Wed Jan 09, 2013 2:21 pm
floogle wrote:The hub the cable is connected to is a 10/100 switch, I don't know why the Pi connects at 10. The speed doesn't really bother me though, just this SSH issue.



This could explain why the RPi is connected at 10 mbits: Most switches test speed from 10 to 100 to 1000. Windows does it the other way, from gigabit to 100 mbit to 10 mbit. If the timing is just right, you end up on the wrong, lower speed setting. I've never knwon it to be a problem with 10/100 though...

A fix could be to hard lock the RPi's port in settings to 100 mbit. The switch will keep cycling until it locks at the right speed.

I hope this can put you in the right direction.
User avatar
Posts: 496
Joined: Wed Dec 05, 2012 11:48 pm
Location: Belgium
by floogle » Wed Jan 09, 2013 3:49 pm
I have tried putting into 100mb/s mode with sudo ethtool -s eth0 speed 100 duplex full but it doesn't seem to want to do it. I have restarted networking, and brought the interface down and up, but it won't do it.

I have another power supply hanging around so I can try that to see if it makes any difference.
Posts: 11
Joined: Tue Jan 08, 2013 11:32 pm
by floogle » Wed Jan 09, 2013 3:57 pm
The second power supply (also 5V, 1A) gave no improvement.

The thought occurs to me that perhaps it is putty causing the problem? The only settings I changed in putty are Connection > Data > auto-login username, and Connection > Seconds between keepalives (10).

I also tried ssh'ing from the Pi to the Pi, admittedly only over loopback, but that worked without problem, I will try doing it over eth0 next time I fire it up.
Posts: 11
Joined: Tue Jan 08, 2013 11:32 pm
by sharow » Wed Jan 09, 2013 8:42 pm
> floogle
thanks for response.

hmmm :| , it looks no network issue.
maybe SSH issue.

1. can you try another SSH-client? openssh or teraterm(http://ttssh2.sourceforge.jp/index.html.en) or something.
2. set 'Compression no' to /etc/ssh/sshd_config
3. set 'Protocol 2' to /etc/ssh/sshd_config, but I don't know Putty supported it.
ArchLinuxARM@RaspberryPi
User avatar
Posts: 9
Joined: Wed Oct 17, 2012 7:24 am
Location: Tokyo
by floogle » Wed Jan 16, 2013 2:15 pm
Sorry for the delay in replying.

sharow wrote:1. can you try another SSH-client? openssh or teraterm(http://ttssh2.sourceforge.jp/index.html.en) or something.
2. set 'Compression no' to /etc/ssh/sshd_config
3. set 'Protocol 2' to /etc/ssh/sshd_config, but I don't know Putty supported it.


1. I tried using ttssh but it was no different from Putty, I even tried an SSH app on my phone, but they all have the same problem.
2. Done, no difference.
3. This was already set.
Posts: 11
Joined: Tue Jan 08, 2013 11:32 pm
by floogle » Wed Jan 16, 2013 2:40 pm
OK, so here's a small update.

Just for fun I connected from my windows laptop with putty, and also from my phone. In both terminals I pinged the local router. Both terminals were running well for many minutes, which is a new record for me.

Then, on one client I stopped ping and ran top instead. That client then froze after only displaying the header in top, at the same time the other client (which is still pinging) slowed to a crawl. And I mean so slow it looks dead.

At this time running uptime on the pi shows < 0.1 load.

Killing both SSH terminals left the sshd on the Pi unresponsive, but after restarting the service I can connect again, at least from my phone.
Posts: 11
Joined: Tue Jan 08, 2013 11:32 pm
by floogle » Wed Jan 16, 2013 7:01 pm
subsequent to my last post I suddenly started to suffer from major network problems. eth0 refused to get an IP address, I fiddled with ifconfig and dhclient and it really struggled.

I have installed a fresh OS on the memory card and now my main problem is the networking. I have to run dhclient a few times in order to get an IP address, and I suffer packet loss of 24-50%. I can't do apt-get update because it simply times out.

I have tested another machine using the same cable in the same location and that worked without any problem at all, so I can rule out a cable problem.

Since this new problem rather precludes my SSH issues I propose that this thread be closed without a solution being found, and I will have a think about what I am going to do with my Pi.

Thanks for all your help guy & gals, I'm just sorry that your efforts have been in vain.
Posts: 11
Joined: Tue Jan 08, 2013 11:32 pm
by efflandt » Wed Jan 16, 2013 10:55 pm
Check /var/log/syslog for any error messages related to ethernet (whether using ethernet or WiFi).

Have you tried doing running rpi-update to get newer firmware? The script and things need to be installed first https://github.com/Hexxeh/rpi-update

Until I did that I had random network drop outs (which also knocks out USB) on 3 different Pi units. But after rpi-update things stay up for days, no problem.
Posts: 359
Joined: Mon Dec 03, 2012 2:47 am
Location: Elgin, IL USA
by floogle » Fri Jan 25, 2013 10:24 am
efflandt wrote:Check /var/log/syslog for any error messages related to ethernet (whether using ethernet or WiFi).

Have you tried doing running rpi-update to get newer firmware? The script and things need to be installed first https://github.com/Hexxeh/rpi-update

Until I did that I had random network drop outs (which also knocks out USB) on 3 different Pi units. But after rpi-update things stay up for days, no problem.


Thanks for the tip, it's certainly worth a try.

But given my network issues it's basically impossible to do this on the Pi itself. I used my linux machine to install the script onto the Pi memory card, but of course when you run it on the Pi it wants to download the data from github (or where ever). Is there a way to download all the data in advance on a different machine?
Posts: 11
Joined: Tue Jan 08, 2013 11:32 pm
by floogle » Wed Jan 30, 2013 6:31 pm
I'm gonna bump this for some more help before I throw out the Pi :)

Edit: I also wanted to note how odd it is that everything worked just fine to start with. These problems have simply developed, and persisted across OS re-install. Perhaps a hardware problem...
Posts: 11
Joined: Tue Jan 08, 2013 11:32 pm
by newe1344 » Fri Apr 05, 2013 8:17 pm
I had similar problems on ssh via wifi. However, I was powering my pi with a 5v 1.5A wall unit.

As soon as I switched to a 5v 2.1A wall power adapter it worked like a charm.

It makes sense if you think about the environment you're using your pi in.

In the mornings, it would work fine, then as people at work would show up and use wifi, the performance would slowly degrade to the point of being virtually useless.

Once I switched to the higher amperage wall unit, I think the wifi adapter was able to get enough power to handle all the network traffic. It's only a theory but it sounds reasonable.

:?:
Posts: 1
Joined: Fri Apr 05, 2013 8:10 pm
by Superfly88 » Thu May 16, 2013 2:19 pm
I had the same problem with SSH being very unresponsive. Typing a command or editing a file was quite impossible as the cursor stuttered and I wasn't sure what keys had been sent. Eventually, it would time out completely and I had to log back in, waiting several tens of seconds for the login prompt.

Strangely it would still respond to pings and mjpg-streamer was still streaming pictures successfully even when SSH was blocked.

I was already running everything (webcam, Edimax EW-7811Un Wifi) from an old USB hub with a 2.3A power supply. Finally, I had enough of it and changed to a new Arkview 7 Port High Speed USB 2.0 Hub with 2.0A power supply. And so far (just a few hours), everything has been running very well. SSH is smooth and responsive, pictures are streaming nicely.

One thing I discovered was when I plugged in the new USB Hub, the Pi started booting, even before I plugged in the Pi power supply. So it looks like the Pi may be drawing power from the Hub instead of the regular power supply, and perhaps my old USB hub or power supply was just not supplying enough current.

Anyway, do indeed try a new power supply.
Posts: 1
Joined: Thu May 16, 2013 2:08 pm