Generally speaking, all that apply to you.
What is important, is that you are doing something to measure your site performance and can get an idea if changes are making a difference. To oversimplify, there are really two outcomes. Firstly, the basic page load time (i.e. with no-one else logged in how quickly do a variety of pages load for a particular user). Secondly, how is overall server load affected (and how much resource is consumed). I generally recommend running something like 'munin' to monitor system resources and to see if you are reaching limits.
Performance tweaking can get quite obsessive but the thing to avoid is just changing random settings and hoping for the best (do as I say, not as I do!!)
If you are looking for a place to start....
1. Check your hardware is genuinely up to the job. If it isn't you'll never fix it with tweaking.
2. Run a suitable accelerator and consider running a memory cache for the MUC.
3. Configure Apache and MySQL's settings to reflect the amount of RAM you have available. Apache usually comes configured for simple web pages and will eat all your RAM very quickly running Moodle. MySQL usually comes very conservatively configured to use hardly any RAM at all. Giving it some space for its caches can make a big difference.