Noise problems are always a big problem, because you do not know where the noise voltage that generates a differential input (noise) signal on the (pre-)amplifier comes from, (what the driving source of the noise signal is) it may be the noise is actually driven into the GND (or VCC) of the amplifier, in which case grounding the audio source (PI) makes the problem bigger, instead of smaller. Best you try, and also try to make the ground-connection between PI and amplifier as hefty as possible, (shorting any potential noise signal between the grounds of both), but avoid creating a loop, (dont make a loop from the two grounds) that can induce humm into the GND. Actually the best solution may be an audio isolation transformer between PI audio out and amplifier (I mean two transformers obviously for stereo).
One trick you can try is to actually ground the input of your amplifier to earth, to see what effect that has. If you get enourmous amounts of noise/hum, the noise/hum is coming in via the power supply of the amp.
In any case keep the sensitive (high impedance) connection as short as possible.
Do try if grounding the PI to earth makes a difference, it may well be, the noise is driven into the common (floating GND) of the PI.
I hope this is clear, its a difficult subject.
One thing that is suggested, is that the PWM output itself is noisy, which it is not! It may not be sufficient for a real audiophile, but it's audio signal should NOT drown in noise.
By the way, PWM audio is by nature sensitive for lowering the digital volume, this drastically reduces the signal to noise ratio, so try to always drive the PWM to a reasonable level, the loudest that works without too much clipping. Lower the digital level too much and you throw away the dynamic and sample range.