Hi,
I have an animating model set up with per-fragment lighting. I would like to know the details about the necessary math operations involved in getting the correct data into the vertex shader for vertex skinning. I need to know about how to calculate the matrix palette and what it represents along with the matrix indices, I also need to know if the matrix weights are just the weights you get from the file to compute vertex positions or if there are additional math operations involved. Thanks in advance for your help.
Re: Explanation of vertex skinning
For vertex skinning you basically need to export the following data:
Per vertex:
A list of bone indices (typically 4).
A list of weights - the weights MUST sum to 1.
Per bone:
The inverse of the bind pose transform - this is because the skin's vertices are stored in the bind pose - ie. if you just draw the skin without doing any skinning you'll see the figure standing as they were when you applied the skin to the model in the modelling package - typically artists will use a "star" position to minimse problems with the armpits and crotch. You need this matrix so that you can "get back to the bind pose" before applying a delta rotation (and possibly translation) to distort the model. Or, in other words you want to subtract off the rotation and translation caused by the bind pose from the animation (but in matrix math this is done by multiplying by the inverse). This is all done with a single matrix in the shader, but you need this information separately to correctly set up the matrices before passing them to the shader.
Then, at runtime, on the CPU you do the following:
Interpolate the animation (using quaternions or however you want to do it). You will probably be using local transforms for the animation so then you'll need to multiply each bone by its parent's matrix to give the full transform.
Next you need to calculate: inverse_bind_pose[bone]*bone_matrix[bone] for each bone to give the final transform that will be used for skinning, and upload these matrices to the GPU in vertex constants.
In the shader you simply calculate:
Alternatively, if you have multiple attributes that you need to skin (eg. position, normal, tangent, binormal for normal mapping) you can weight the matrices instead of the positions:
then transform each of the attributes by final_matrix. This gives the same result as the pseudocode above but is typically much more efficient (only num_attributes matrix multiples + a bunch of multiplies and adds vs num_attributes*num_influences matrix multiplies). It will obviously be about the same, or slightly slower if you are only skinning positions though 
Note that normally you want to upload 3x4 bone matrices to the shader (transposed so that they fit into 3 vertex constants each) - since the transforms will always be affine.
Per vertex:
A list of bone indices (typically 4).
A list of weights - the weights MUST sum to 1.
Per bone:
The inverse of the bind pose transform - this is because the skin's vertices are stored in the bind pose - ie. if you just draw the skin without doing any skinning you'll see the figure standing as they were when you applied the skin to the model in the modelling package - typically artists will use a "star" position to minimse problems with the armpits and crotch. You need this matrix so that you can "get back to the bind pose" before applying a delta rotation (and possibly translation) to distort the model. Or, in other words you want to subtract off the rotation and translation caused by the bind pose from the animation (but in matrix math this is done by multiplying by the inverse). This is all done with a single matrix in the shader, but you need this information separately to correctly set up the matrices before passing them to the shader.
Then, at runtime, on the CPU you do the following:
Interpolate the animation (using quaternions or however you want to do it). You will probably be using local transforms for the animation so then you'll need to multiply each bone by its parent's matrix to give the full transform.
Next you need to calculate: inverse_bind_pose[bone]*bone_matrix[bone] for each bone to give the final transform that will be used for skinning, and upload these matrices to the GPU in vertex constants.
In the shader you simply calculate:
Code: Select all
final_pos=(0,0,0)
for each influence
final_pos+=(pos*bone_matrix[bone_index[influence]]*bone_weight[influence])
Code: Select all
final_matrix=0
for each influence
final_matrix+=bone_matrix[bone_index[influence]]*bone_weight[influence]
Note that normally you want to upload 3x4 bone matrices to the shader (transposed so that they fit into 3 vertex constants each) - since the transforms will always be affine.
Re: Explanation of vertex skinning
Thanks for the great reply! Unfortunately the shader code I am using exceeds the maximum uniform variable count in the Pi so the shader doesn't compile. Hopefully I will still be able to put the knowledge to use. Thanks.
Re: Explanation of vertex skinning
Janq's really concise post on skinning deserves commendation. Just to add the traditional extra .02 to the subject:
Bone transformations usually form a skeletal structure - normally a tree where each node is defined in the space of a single parent, with the sole exception of the root node which has no parent. This is done for the obvious need for modeling real-world skeletal mechanics where most bones have an inner and outer joints, and the movements of bones higher up the chain affect all connected parts down the chain (think how limbs function).
Also, during my own musings with skinning on the RPi I've stepped on an issue where even though the matrix palette would fit in the available uniforms, the vertex shader would run out of registers, causing cryptic failures during the actual draw call. Of course the root of the issue was not clear at the time, but I had some help with the diagnostics from a Broadcom engineer. Back then that issue was pinned on the compiler, with the hope that a future compiler iteration would address it. I don't know if the situation has changed since then as I've been busy with other things. I do know, though, that if I reduced the vertex blending computation from 4 to 3 matrices, the shader worked, with the obligatory awkward results one could expect from doing that to a 4-bones-per-vertex model.
Bone transformations usually form a skeletal structure - normally a tree where each node is defined in the space of a single parent, with the sole exception of the root node which has no parent. This is done for the obvious need for modeling real-world skeletal mechanics where most bones have an inner and outer joints, and the movements of bones higher up the chain affect all connected parts down the chain (think how limbs function).
Also, during my own musings with skinning on the RPi I've stepped on an issue where even though the matrix palette would fit in the available uniforms, the vertex shader would run out of registers, causing cryptic failures during the actual draw call. Of course the root of the issue was not clear at the time, but I had some help with the diagnostics from a Broadcom engineer. Back then that issue was pinned on the compiler, with the hope that a future compiler iteration would address it. I don't know if the situation has changed since then as I've been busy with other things. I do know, though, that if I reduced the vertex blending computation from 4 to 3 matrices, the shader worked, with the obligatory awkward results one could expect from doing that to a 4-bones-per-vertex model.
Re: Explanation of vertex skinning
You can generally drop influences from a skin without too many problems as long as you renormalize the weights afterwards. Going from 3->4 shouldn't really make much difference for most models (more weights are often used on tricky areas like armpits and knees and you'll find that most vertices only really use 1 or 2 weights anyway - this happens often enough that its sometimes worth putting branches in the vertex shader to optimise for it).
To reduce a model by one influence you can just loop through each vertex, find the influence with the lowest weight, remove it, then multiply the other weights by 1/(1-weight_removed).
re: running out of vertex constant space for the bone matrices. There are several ways to deal with this. One way is to fetch the matrices from a texture - this gives decent performance on modern GPUs but maybe not on the Pi (although - its worth measuring just to see what its like).
Another way is to split the mesh into chunks that will fit. You can use a simple greedy algorithm to do this (start with an empty mesh and bone list, progressively add faces that require the least extra bones until the bone list reaches the max. number of bones, then start a new chunk). You'll end up with some bones in more than one chunk, so for each chunk you just record "local bone"->"global bone" mapping list - its a bit less efficient than drawing the skin in one go (you need to upload more bones overall, and make more API calls), but its usually not much worse. This algorithm could be made efficient enough to run at load time, although for a fixed target its best to do it offline.
Lastly: its possible to upload quaternions and vectors instead of matrices (so, if you need translations thats 2 constants per bone, if you only need rotation then you only need 1 constant per bone). It is possible to directly transform a coordinate by a quaternion, or you can convert the quaternions to matrices in the shader (a google search should find you all the maths you need to do this - relatively simple to implement in a vertex shader). This method is generally quite a bit slower on the GPU though (the splitting method is probably going to be best).
To reduce a model by one influence you can just loop through each vertex, find the influence with the lowest weight, remove it, then multiply the other weights by 1/(1-weight_removed).
re: running out of vertex constant space for the bone matrices. There are several ways to deal with this. One way is to fetch the matrices from a texture - this gives decent performance on modern GPUs but maybe not on the Pi (although - its worth measuring just to see what its like).
Another way is to split the mesh into chunks that will fit. You can use a simple greedy algorithm to do this (start with an empty mesh and bone list, progressively add faces that require the least extra bones until the bone list reaches the max. number of bones, then start a new chunk). You'll end up with some bones in more than one chunk, so for each chunk you just record "local bone"->"global bone" mapping list - its a bit less efficient than drawing the skin in one go (you need to upload more bones overall, and make more API calls), but its usually not much worse. This algorithm could be made efficient enough to run at load time, although for a fixed target its best to do it offline.
Lastly: its possible to upload quaternions and vectors instead of matrices (so, if you need translations thats 2 constants per bone, if you only need rotation then you only need 1 constant per bone). It is possible to directly transform a coordinate by a quaternion, or you can convert the quaternions to matrices in the shader (a google search should find you all the maths you need to do this - relatively simple to implement in a vertex shader). This method is generally quite a bit slower on the GPU though (the splitting method is probably going to be best).
-
dom
- Raspberry Pi Engineer & Forum Moderator
- Posts: 5502
- Joined: Wed Aug 17, 2011 7:41 pm
- Location: Cambridge
Re: Explanation of vertex skinning
I do have a patch that may improve things. Where the shader compile previously failed it will now try again with a different register allocation scheme that should handle loops better.blu wrote:...the vertex shader would run out of registers, causing cryptic failures during the actual draw call. Of course the root of the issue was not clear at the time, but I had some help with the diagnostics from a Broadcom engineer. Back then that issue was pinned on the compiler, with the hope that a future compiler iteration would address it. I don't know if the situation has changed since then as I've been busy with other things. I do know, though, that if I reduced the vertex blending computation from 4 to 3 matrices, the shader worked, with the obligatory awkward results one could expect from doing that to a 4-bones-per-vertex model.
I tried it a while back, and it didn't fix the case I was looking at (Mandelbrots).
Because the new scheme only runs if the initial compile fails, it should be harmless, so I'll try merging it in and pushing it out for testing. It could help.
Re: Explanation of vertex skinning
I'm definitely interested in that patch! Once again, thanks, dom, for looking into that obscure issue!dom wrote:I do have a patch that may improve things. Where the shader compile previously failed it will now try again with a different register allocation scheme that should handle loops better. I tried it a while back, and it didn't fix the case I was looking at (Mandelbrots).
Because the new scheme only runs if the initial compile fails, it should be harmless, so I'll try merging it in and pushing it out for testing. It could help.
Janq, your suggestion about bone index reductions on meshes is a perfectly viable approach, useful in many situations. In my particular case though it wasn't.