I have a Pi 3B+ running Ubuntu Mate 18.04 (originally upgraded from 16.04 and now the official image - same issue) which is run headless as a Bit Torrent client. Briefly it is configured thus:
On-board NIC manually set to an IP address on my LAN, firewalled to prevent Internet traffic - only ssh and vnc traffic allowed
A USB dongle NIC configured for DHCP and connected to my router for Internet access (firewalled in drop zone)
systemctl set-default is multi-user.target (no gui)
A 240 GB SSD is attached via a USB connection
vnc4server is installed
In operation I:
Boot the Pi
Connect by ssh and load the vncserver "vncserver :1 -geometry 1600x900 -depth 16"
(I have a script loaded by System; Preferences; Personal; Startup Applications; which checks every second for the presence of a vpn tunnel and does a killall transmission-gtk if the tunnel is not found)
Close the ssh connection
Connect to the vnc session with Vinagre (on a CentOS 7 machine)
Open a terminal and run a script to connect to my VPN provider's system
Invoke Transmission to perform such Bit Torrent business as I might wish
Close the vinagre session until I wish to check on progress
The system/process works fine EXCEPT that on occasion I find that essentially all memory and swap has been consumed. I have been monitoring the Pi with glances over an ssh connection. I find that Firefox will consume a lot of memory (expected) but will release it when closed. When the out of memory situation occurs - a constantly glowing greed "disk" access light on the Pi is a clue - I find that two instances of mate-panel are together consuming in excess of 80% of RAM and swap.
If I issue a killall mate-panel I find that the memory is immediately released. If I connect with vinagre with the system running normally (I generally cannot connect and/or the system is unresponsive with the memory condition) and killall mate-panel I observer that the upper and lower panels disappear and immediately reappear. Today I setup a script to do a killall mate-terminal every 10 minutes as a big hammer workaround.
I suspect that this is in some way related to vncserver. I ran the same process with the Pi connected to a monitor with the gui interface loaded and did not experience any memory issue in a couple of days running the vpn and Transmission.
Just wondering if anyone else has had similar experiences or if anyone has some insight to what is going on and perhaps how to fix it.
TIA,
Ken
Re: mate-panel is eating my Pi (at least the memory and swap)
Not on the Pi (as all of mine run without a desktop), or with mate-panel (I don't run mate), but I do have a similar issue on my laptop with the Debian Stretch-based Raspberry Pi Desktop installed.
lxpanel (possibly related to mate-panel? Maybe they use some of the same libraries and the issue is in a library?) has a known memory leak. It eats quite a chunk of my RAM if I let it run for several days without a restart. It's not so bad now I have 16GB, but my old laptop had 4GB and that would get used up in less than a week sometimes. I'm hoping the version of lxpanel in Debian Buster will have fixed that problem.
In the meantime, the command lxpanelctl restart will release the memory for me. I'm thinking of putting it in a cron job to run when I'm asleep. Maybe there is a similar command for mate-panel?
lxpanel (possibly related to mate-panel? Maybe they use some of the same libraries and the issue is in a library?) has a known memory leak. It eats quite a chunk of my RAM if I let it run for several days without a restart. It's not so bad now I have 16GB, but my old laptop had 4GB and that would get used up in less than a week sometimes. I'm hoping the version of lxpanel in Debian Buster will have fixed that problem.
In the meantime, the command lxpanelctl restart will release the memory for me. I'm thinking of putting it in a cron job to run when I'm asleep. Maybe there is a similar command for mate-panel?
Unreadable squiggle
Re: mate-panel is eating my Pi (at least the memory and swap)
Thanks rpdom,
There is a mate-panel --reset. However, that removed all the customization I had done to the top panel
I see there is also a --replace option. That does not seem to disrupt any settings. I guess I might put that in my script instead of killall. I wish the Pi had 4 GB 
Ken
p.s. I think I will NOT. I have been playing with this on my other monitor. --replace did just that. However, the command prompt did not return. When I hit Ctrl+C to regain control I lost both panels. Actually if I add & at the end of the command that might be OK.
There is a mate-panel --reset. However, that removed all the customization I had done to the top panel
I see there is also a --replace option. That does not seem to disrupt any settings. I guess I might put that in my script instead of killall. I wish the Pi had 4 GB Ken
p.s. I think I will NOT. I have been playing with this on my other monitor. --replace did just that. However, the command prompt did not return. When I hit Ctrl+C to regain control I lost both panels. Actually if I add & at the end of the command that might be OK.
Re: mate-panel is eating my Pi (at least the memory and swap)
mate-panel --replace does not seem to be a viable answer. I ran it for a couple of hours and all appeared OK. Until I touched a window on the remote desktop. Then all sorts of mess let loose. Flashing panels, windows etc. I had to ssh to the machine and kill the vncserver session and start over. I am back to killall mate-panel. That does not seem to cause more problems than it solves.
Ken
Ken