I'm pretty sure the answer to this question is just: run vncpasswd
Or just find a file somewhere in your home dir named "passwd" and remove it.
P.S. I noticed (while researching this question) that x11vnc doesn't have the same sort of password setup as the other ones (the "normal" ones) do. It seems to take the password on the command line (which is, of course, a Bad Idea).
And some folks need to stop being fanboys and see the forest behind the trees.
(One of the best lines I've seen on this board lately)