I was a bit off then, I ran through some recent updates for the Pi and it seems to actually have increased now. Makes things simple.
. I also used UseDNS No command to improve login prompt times for SSH itself. Between those 2 I am in acceptable range for sure now.
I actually get