The sticky stuff with the licencing is the GPL related stuff. Some of it isn't clear cut, since it all comes down to how separate things need to be before they're not derrived works. I'm not an expert, and if you wanted to know more you could contact the Free Software Foundation. That said, if you close off things that should be opened, someone might just point it out and ask you to open things later.
Just looking over your github, and I'm not sure if it's everything involved at this point, copies-and-fills is derrived from GPL code, so it must be GPL, and anything that includes or links to it must also be GPL. If you didn't actually derrive from the original code and wrote a completely from scratch assembly file, then you can licence it however you choose.
The kernel is GPL as well, so the kernel module stuff must also be GPL.
XFree86 is MIT licence, which only requires you to preserve their copyright notice, you are under no obligation to release source, though if your patches require a modified version of XFree86, you'll have to basically take responsibility for maintaining your proprietary fork if you want people to keep using it, passing along the source to the XFree86 team means you could possibly get it incoporated into the main trunk, ensuring that it stays maintained.
I don't know how your VPU code integrates with the system, if it's a library that talks to GPL software via a strict API that separates the internals (like GL does), then you can probably keep it closed without any real issue. If its operation requires any GPL code to couple tightly to it, shares a lot of internal state information, shared memory, etc... then it's probably going to have to be GPL'ed regardless of if it statically or dynamically links. Dynamic kernel modules are a good example of this.
If it's just uglyness or just requring one to be really on top of things to understand the code or tools, I don't see any reason to keep it closed. You're not obligated to provide easy to use software, and others can potentially learn from viewing the source. If you put it out there, and you're the only one in the world who truly understands it, that's still ok.
I'm a developer of video games, and it's all proprietary stuff, and proprietary software has its place. But personally, anything I develop for the Pi I intend to make GPL (though I may need to change the licence on my xf86-video-rpi to MIT in order to get it adopted into the trunk), so that the device's primary audience, the future hackers and tinkerers of the world, have some really meaty stuff to look at.
Keep in mind that GPL ('Free') software clearly doesn't mean "Must be given away". Its intent is to provide users with all the freedoms you enjoy as the author of the software, except for the right to further restrict those freedoms. You also only have to provide the source upon request if it doesn't come with the binaries, and only to those you have disributed to. So for example, if you sell GPL binaries, then you're only obligated to give the source to someone who buys it from you, and you can wait for them to ask. Anyone you didn't sell to has no right to request the source.
At any rate, the easiest thing to do is probbaly to keep the same licence as whatever the code most tightly couples with. If the code is 100% original and functions relatively independantly of other programs, even if linked, then do what you want, it's your work. Some people will likely fuss about freedom, but then they can try and write their own VPU layer. Broadcom isn't about to open their firmware, and I honestly don't care if they ever do. I'm happy enough to have a graphics stack that works.