I've ran Moodle on a VPS with 256MB ram (from godaddy.com), I'm not going to lie, it's dog slow, but with a little work, the performance can be acceptable for a small site. Here are some tips (bulleted, because I'm lazy):
*Trim apache as much as possible
*Trim MySQL, but give it enough room to breathe.
*Install the ADODB php PECL module, it makes moodle not use so much RAM.
*Install a PHP accelerator, it will dish out pages MUUUUCHH faster.
*Turn off any unnecessary daemons.
*Turn off any Moodle filters, while they are great, there's a ton of overhead.
*Don't use the chat module...nuff said.
*I've noticed that on my VPS, I waste a bunch of CPU cycles waiting for disk i/o. With a bunch of people on one box, the disks can only serve so many people at one time. Minimize your disk reads/writes as much as humanly possible.
Also, if you can, get more memory.