I will share what made a huge difference on my server. We deployed a new Moodle server last summer on Ubuntu Server 12.04 LTS with Moodle 2.2 installed, after upgrading from Moodle 1.9 installed on an older server. The new server crashed a few times last September due to load when classes resumed. The new server existed in a virtual environment so it was easy to allocate more RAM memory and CPU processors / cores. Adding memory and CPU did not eliminate the crashing because the problem was HOW the server made use of available memory rather than the amount of memory or cpu available.
First huge improvement (and successive smaller improvements) was to run mysqltuner script and follow the suggestions. After some tweaking of mysql memory usage settings in /etc/mysql/my.cnf the server could handle much more load and did not crash.
Next significant performance improvement was to install and configure APC PHP accelerator (php-apc).
My benchmark is to have any Moodle pages load in less than a second. Most are around the 0.5 seconds or less as indicated by enabling Site administration > Development > Debugging > Performance info.
So in summary, tuning the mysql database settings using mysqltuner kept the server from crashing under load. And installing APC PHP cache enabled the pages to load much quicker. Both of these must be tuned to some degree based on your installation.
If your server is based on Ubuntu, both mysqltuner and php-apc are readily available from the repositories.