If you don't need to use Bluetooth audio, you don't really need PulseAudio. You can use ALSA, which is what standard Rasbian with LXDE is using.
I had the same issues with PulseAudio causing crackling and distorted sound as well, but toggling the audio output in the Switcher applet from Analog to multi-channel rebooting, then toggling it back to Analog, and rebooting normally fixed that problem for me.
If you want to continue without PulseAudio, install the pnmixer package from the standard Raspbian repository. It's a systray GUI applet that you can use to control sound with ALSA. In that case, uninstall the pavucontrol package and disable the mate-volume-control applet from loading into systray upon logging in to Mate. You can also uninstall the sound switcher package because it only works with PulseAudio.
Note: The pavucontrol and the mate-volume-control utilities work with PulseAudio but not with ALSA.
Just a little background if you are interested to know. The Raspbian Foundation decided to get rid of PulseAudio in their distribution a while back. The problem is that ALSA has no native mechanism for switching audio output from analog to Bluetooth audio or vice versa. So, the Raspbian developers created a package pimixer that allows you to switch from analog to Bluetooth without having to use PulseAudio. Unfortunately, pimixer installs only in the LXDE system tray, so it can't be used in Mate. So, in Mate, in order to be able to switch from Analog to Bluetooth audio, you have to install PulseAudio and the Sound Switcher Indicator applet. That's why my instructions had those steps in them. So, the necessity of having Bluetooth audio installed in Mate on Raspbian in order to be able to switch the audio output to Bluetooth audio is a weakness of this solution. The crackling and distorted sound issue with PulseAudio installed in Raspbian keeps coming back every now and then.
Perhaps the long awaited Ubuntu version for Raspberry Pi 4b certified by the Raspbian Foundation will solve this issue and the audio output will be able to be switched to Bluetooth audio without requiring PulseAudio. Ubuntu has a Mate desktop version as one of their official forks.