This may seem like an Apache question, but it seems to only be impacting the Moodle servers, so I thought I would ask here in case anyone has seen this before.

Our setup HAProxy load balancer -> Four moodle web servers -> One SQL server

This setup with Moodle 1.9.9+ seems to be working fine. The problem is that every night
around midnight (sometimes 1am, sometimes 12:30am, sometimes 1:30am, etc) the Moodle servers (all four of them) will start to spawn httpd workers. All servers are running
Apache 2.2.3 under CentOS 5.5.

What is odd is that due to this we started logging all kinds of info to see what the problem
is. We are logging connections on the HAProxy, on the moodle servers, etc. At the time
the spike happens (which lasts typically 10-15mins) there are no connections being made
to the HAProxy (e.g. it goes up during the day, down in the afternoon, up again in the evening, starts to fall about 11pm-4am then goes back up), but there are no excessive
connections. For example, haproxy is showing a few, but the moodle servers are spawning hundreds of httpd workers (to the limit set in httpd.conf).

This happens every night on all servers. It happens when the moodle backups are going on or not, when the cron job is running or not. We just can not figure this one out (at least not
yet). Everything runs fine the rest of the time. We have checked the machines over and
over for some other jobs that would spawn http's but there are none.

If anyone has seen this before, or has an idea, any input would be much appreciated.

Thanks.