Did you find the cause of the problem?
At first glance, it is an obvious memory leak. As Simon pointed out, the system software (LAMP
) is the first place to check.
But from the way the memory gradually builds up (Munin graph) and because the problem vanishes once you deactivate the cron job, obviously the cron job takes too long to finish or they never end. Look at those very long total CPU times of those processes (column TIME in top)!
Talking of long cron jobs, the Moodle cron job is not meant to be parallelized. If a second cron job starts before the first one is finished, it'll confuse Moodle.
How long does a single cron job take in your machine? Deactivate Moodle cron job frst, leave enough time for those stray php processes to exit or reboot the machine. Then during a quiet hour and a busy hour run cron.php via the command line. You output will be something like this:
# su - OWNER-OF-THE-APACHE-PROCESS
$ /PATH/TO/php /PATH/TO/cron.php
Starting activity modules
Processing module ...
script completed correctly
Execution took N seconds
What is your N?