I've successfully built RasPi controlled 8x8x8 LED cube. Currently, I'm doing everything - generatingeffects and LED multiplexing RasPi, which is not ideal: Linux is not a real time OS and even with highest process priority there are various glitches, flashes etc. when something interrupts the multiplexing loop. I'll eventually move multiplexing code to some AVR uC, communicating with RasPi over RS232/I2C.
The whole application is designed as a client-server, client generating effects and sending them in 64B data packets through a named pipe to server. Server periodically refreshes the whole cube from its "video memory (updated with data received from FIFO)", creating desired effect.
I'll release the source code anytime soon
Website with more informations, incl. circuit diagram, board design: http://dev.mrkva.eu/elektro/ledcube/
Demo of first 16 implemented effects https://www.youtube.com/watch?v=S6P3jcxKh-k