You don't say what model of Raspberry Pi, what, if anything, you have connected to it, whether the ssh connection is going out from the Pi or inbound to it, and what provides the network connection (Ethernet or WiFi? If the latter, Pi3 built-in, or dongle?).
I have seen similar problems, on occasion, that (if I'm not mixing up stories) were caused by ssh'ing into Pi's (model B's and Zeros) that had WiFi dongles (Edimax EW-7811Un, based on the Realtek RTL8188CUS chip) which were "helpfully"going into powersave mode. Turns out there's an option one can put in a configuration file to disable the powersave mode ("options 8192cu rtw_power_mgnt=0 rtw_enusbss=0" placed in /etc/modprobe.d/8192cu.conf) for that particular chip, which made network connections to those machines more reliable/responsive.
But you haven't provided enough information to determine if you're having that problem or something entirely different.