Pi as a VOIP server?

Any thoughts on using a Pi as a VOIP chat server? I usually use Ventrilo, with usually no more than 2-5 clients connected. Obviously Vent's out though, as there's no ARM build nor source.

Perhaps the Pi isn't even appropriate?

Any thoughts or advice greatly appreciated!

Ive been thinking about this myself. Here is a forum question on the asterisk website that may help.

http://forums.asterisk.org/viewtopic.php?p=161907

Aaaaand, here's another helpful link.

http://www.raspberry-asterisk.org/

Take a look at IncrediblePBX. I've been using it for some time now. It works well on a 256MB RAM Model B.

[Don't bother with the fax module. The Raspberry Pi has nowhere near the horsepower required to process faxes.]

Great info - Thanks everyone!

If you are looking for something like Ventrilo I could suggest http://mumble.sourceforge.net/ (mumble).
It is quite easy to setup and should work on the Pi. It's less telephonie than the other suggestions, but closer to Ventrilo and in my opinion in nearly all regards better than Ventrilo.