lrussell wrote:I just want to know if it's safe to continue to use the adapter.
My "no warranty expressed or implied" opinion is "probably". It depends on how "safe" you want to be. It's unlikely that the adapter will damage any of the RasPi ICs, since they use 3.3V so as long as RasPi's main 5V line (connected to TP1) is at least 4.3V they'll be fine. However, as I suggested in my last comment if an HDMI->VGA adapter draws too much current it could cause a weak F3 to fail or cause D1 to burn up. Both of these are easy to fix if one's handy with a soldering iron.
It may be that your problem is simply that your mouse requires close to 5V to operate, while a different mouse may work with 3.3V internally so it really only needs 4.3V to operate. For example, I believe wireless mice generally need the full 5V, and they draw a lot more current than a wired mouse.
Sorry if this is unhelpful, but it depends on your particular circumstances. Knowing the voltages at TP1 and at the input to F3 would be useful. A good rule of thumb (or finger-tip in this case) is that if components like F3 and D1 are getting really hot (i.e., you don't want to leave your finger-tip on them for more than half a second) then expect them to fail. If they're cool, you're probably fine.
For more opinions, you might try entering "HDMI to VGA" in the Google search box at the raspberrypi.org home page.