To be fair, they mostly aren't *really* running Linux, they are rather using it as a kind of glue to stick their rather custom GPU and interconnect systems together.
I'm not sure what you mean there.
To be fair (to whom by the way?) those guys don't want to waste cycles running Linux or any other OS. They want to run the massively parallel nuclear physics simulations or climate models or whatever. The less the OS gets in the way the better.
They need an OS on every node to load it up and orchestrate it all. They need the "glue" as you say. Linux is the best glue.
Just the same as Linux on my PC is the glue between the processors in my keyboard, my SSD, my USB devices, my WIFI/ethernet interface, my GPU, etc, etc and my application.