Sounds to me like a problem with your router.
To connect to your site using an external IP, the router must do what's called "hairpin routing".
Not all routers have this capability.
Maybe your router can do it, but not well.
Can you connect to your site from outside of your LAN?