I don't run Kodi on my Pi4, but I do have it running on a 3B+ and I've noticed the opposite (though this may be tempered by experience). On my desktop, which is an order of magnitude faster than my Pi, it takes a while to exit, whilst on the 3B+ it's surprisingly fast.
If I shutdown the 3B+ from Kodi, rather than exiting, it's even faster. (It is the only Pi I shutdown each day).
Headless PI. OMG, someone cut it's head off. Oh, hang on. it didn't have one to start with.