I found this, which could explain some of the trouble.
Currently, the audio drivers for the Raspberry Pi are still in beta, and as such come disabled in the standard release
(debian6-19-04-2012.zip). Here’s a quick way to get it up and running for both the HDMI as well as the analog output. Open up a terminal and install the alsa (Advanced Linux Sound Architecture) utility package:
Now load the sound driver using modprobe:
By default output will be automatic (HDMI if supported, else analog). You can force it to use a specific interface with:
Substituting <n> for the desired mode: 0=auto, 1=headphones, 2=hdmi. For example, to force the Raspberry Pi to use the analog output:
Yah but even after setting this up, I get mangled audio.