Are you saying there is no difference in the time it takes to reach 80 degrees with or without the heatsink? That sounds like poor thermal conduction, in comparison to the conduction into the PCB the thermal conduction
Page 13 of this is particularly interesting: Shows only 11% of the heat escapes through the top surface of the die, 87% through the bottom... Until you add a flow of air which still doesn't make a huge difference. Both CM3 and Pi 3 are wire bonded and therefore have no physical connection between the packaging and the top surface of the die suggesting that indeed a heatsink isn't particularly effective at sinking heat!
https://www.idt.com/document/apn/842-th ... -selection