Using the Debian squeeze image all of a sudden startx doesn't start X unless I'm logged in as root -- logging in as pi and trying it I get two FBDEV(0) FBIOBLANK: Operation not permitted errors, a mouse error (failed to initialize for relative axes), the screen goes blank for a second, and I'm returned to the shell with "waiting for X server to shut down".
The only things I've really fiddled with from the default install are installing some dev tools, compiling FCEUltra (a NES emulator), installing Chromium (Midori didn't want to start), and tweaking config.txt to force HDMI mode. I've restarted several times since doing all those and it was working fine -- as far as I'm aware I changed nothing between it working and not working.