As I've discovered recently the lang menu (top right corner of the default install of moodle) is a performance hog. It consumes ~ 10MB of RAM and includes ~40 files.
So I've fixed the problem in 1.4.4+ and 1.5-dev by providing a cache. For those running production servers, get any 1.4.4+ newer than this post, and read the config-dist.php file carefully. You'll find instructions to set $CFG->langcache = true; with some notes on how it works and the tradeoffs it has. Once you have it configured, give it about an hour to make sure the lang menu is cached (you can check the dataroot/cache/ directory), and do a full restart of Apache. Memory use should come down significantly.
In my dev sandbox, Moodle 1.5 went from taking 14MB to taking 4MB to serve the homepage. For those tuning Apache, this is good news: now you have more memory for mySQL and for raising Apache MaxClients.
For those playing with 1.5 enable debug and take a good look at the reported memory usage and time. Enable/disable modules, blocks and filters to get a feel for how to enhance Moodle performance by disabling stuff you don't use.
(Edit: If you want to really take advantage of this fix, make sure you are running admin/cron.php from the commandline, _not_ through apache.)