I recently got my Raspi too and ran into exactly the same Problem.
The first two days, everything worked like a charm. No hangups whatsoever.
Today I got a new (bigger and faster) SD-card and a HDMI to DVI adapter cable and the problems began. I installed Raspbian on the SD and booted it. Still no problems. Then i connected a Mouse to my setup and things started going south. In the end it didn't even recognize the keyboard so I started updating the firmware and kernel.
Nothing got better. So I fired up my oscilloscope and measured the 3V3 and 5V rails. 3V3 seemed completely normal and 5V fluctuated up to 200mV depending on the AC adapter I used. After I flipped it over, i nearly burned my finger on the damn polyfuse! It was boiling hot!
So I measured across it and got 100 to 200mV of voltage drop across this sucker.
I then bridged it by clipping an alligator clip over it and reapplied power to the system.
No problems since then. It's running for about half an hour now and finally does what it's supposed to.
I'm thinking of just bridging it with a piece of wire. Then it will (if it ever will) fail spectacularly instead of quietly
This certainly dampened my first impression on the Raspi, but nonetheless I'm thrilled to finally start working on my first real projects with this little beast!