I see you have a good CPU but the RAM is on the low side. Upgrading to 2 GB is a worthwhile investment I'd say.
BTW, did you go through the other discussions in this formum? This for example: http://moodle.org/mod/forum/discuss.php?d=54123
Hardware and performance
Server Capacity?
This discussion has been locked so you can no longer reply to it.