Just use Raspbian Lite:
It will only take up a fraction of a modern SD card or USB stick. There will be some stuff in there that you don't need but as it's not running unless you run it it will not be impacting performance or using up RAM.
If something really annoys you in there then there is always "sudo apt-get remove whateverPackage".
I see no point in faffing about trying to make it even smaller.
If I really wanted a minimal sized image for a specific application I would build it myself using Gentoo or Linux From Scratch (LFS).