Linux load problems at 16:14 each day

Linux load problems at 16:14 each day

by Richard Williamson -
Number of replies: 5

Hi all,
I'm having 'issues' with load on my Fedora 7 server. A spike occurs every day which takes the load up to 2.2 or so. There are no cron jobs scheduled for that time, and as far as I can see, there's nothing else set to do anything at that time. I have tried looking at the memory, processes and I/O to see what's happening, but a) I'm no expert, and b) it all looks normal to me - apart from the HTTPD entries; I thought that each Apache process should take up about 10MB, not 74MB. I've also checked just about every log known to man. If anyone could shed any light on this, I'd be grateful. I've copied in the results of looking at process and memory below:

TIA
Richard


Process ID Owner CPU Command
26500 root 11.5 % /usr/libexec/webmin/proc/index_cpu.cgi
11011 root 4.6 % gnome-system-monitor
2521 root 2.5 % /usr/bin/Xorg :0 -br -audit 0 -auth /var/gdm/:0.Xauth -nolisten tcp vt7
2046 mysql 2.4 % /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-f ...
1748 root 0.5 % klogd -x
1745 root 0.2 % syslogd -m 0
20907 root 0.2 % top
6528 apache 0.1 % /usr/sbin/httpd
6530 apache 0.1 % /usr/sbin/httpd
6531 apache 0.1 % /usr/sbin/httpd
6532 apache 0.1 % /usr/sbin/httpd
6533 apache 0.1 % /usr/sbin/httpd
18781 apache 0.1 % /usr/sbin/httpd
18847 apache 0.1 % /usr/sbin/httpd
22009 apache 0.1 % /usr/sbin/httpd
22010 apache 0.1 % /usr/sbin/httpd
22011 apache 0.1 % /usr/sbin/httpd
22013 apache 0.1 % /usr/sbin/httpd
22014 apache 0.1 % /usr/sbin/httpd
22015 apache 0.1 % /usr/sbin/httpd
1 root 0.0 % init [5] 

> vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  1     36 3033376  78976 355736    0    0   421  1022   26    5 15  3 81  2  0

 
 
Process ID Owner Size Command
2046 mysql 238220 kB /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-f ...
2692 root 149020 kB nautilus --no-default-window --sm-client-id default3
2714 root 131328 kB nm-applet --sm-disable
2727 root 111500 kB /usr/bin/python -E /usr/bin/sealert -s
2690 root 95988 kB gnome-panel --sm-client-id default2
20886 root 94268 kB gnome-terminal
11011 root 93380 kB gnome-system-monitor
2754 root 87596 kB /usr/libexec/trashapplet --oaf-activate-iid=OAFIID:GNOME_Panel_TrashApplet_Facto ...
2851 root 85192 kB /usr/libexec/fast-user-switch-applet --oaf-activate-iid=OAFIID:GNOME_FastUserSwi ...
2749 root 80492 kB /usr/libexec/wnck-applet --oaf-activate-iid=OAFIID:GNOME_Wncklet_Factory --oaf-i ...
2857 root 80048 kB /usr/libexec/mixer_applet2 --oaf-activate-iid=OAFIID:GNOME_MixerApplet_Factory - ...
11398 root 79984 kB file-roller /usr/src/libstdc++-4.1.2-12.i386.rpm
6529 apache 78436 kB /usr/sbin/httpd
6533 apache 78180 kB /usr/sbin/httpd
22009 apache 77884 kB /usr/sbin/httpd
6531 apache 77592 kB /usr/sbin/httpd
6534 apache 77432 kB /usr/sbin/httpd
2723 root 77304 kB gnome-power-manager
22014 apache 77296 kB /usr/sbin/httpd
22015 apache 77216 kB /usr/sbin/httpd
6532 apache 76980 kB /usr/sbin/httpd
6528 apache 75552 kB /usr/sbin/httpd
22013 apache 74868 kB /usr/sbin/httpd
22011 apache 74668 kB /usr/sbin/httpd
6527 apache 74464 kB /usr/sbin/httpd
22010 apache 73708 kB /usr/sbin/httpd
18847 apache 72896 kB /usr/sbin/httpd
6530 apache 72556 kB /usr/sbin/httpd
18781 apache 72248 kB /usr/sbin/httpd
4590 root 71800 kB /usr/libexec/notification-daemon
2103 root 64644 kB /usr/sbin/httpd
2676 root 40052 kB /usr/libexec/gnome-settings-daemon
2681 root 38840 kB /usr/libexec/bonobo-activation-server --ac-activate --ior-output-fd=20
2566 root 33064 kB /usr/bin/gnome-session

Average of ratings: -
In reply to Richard Williamson

Re: Linux load problems at 16:14 each day

by Samuli Karevaara -
What are the http request that cause the spike? Have you checked the Apache access log? Are they normal users, or a web robot, for example?
Average of ratings: -
In reply to Richard Williamson

Re: Linux load problems at 16:14 each day

by Myles Carrick -
hi Richard,

Mate if the spike is only happening at 16:14 each day then it's highly likely that there must be something else other than Apache procs... have you investigated cron jobs for all users?

Also - apache processes for Moodle are always going to be greater than 10MB. - that's more likely if you're just serving out static pages with practically no apache modules loaded (incl. php).
There's another active thread at the moment referring to the documented "rule of thumb" of 1GB per 50 concurrent users... and I think that ratio is rather ambitious (even with a highly tuned system), but looking at your process log you have got a LOT of other junk running (Gnome applets etc) - your top 10 processes are almost all unrelated to Moodle. Are your Linux skills up to running your server without a GUI? It looks like you're installing a package and receiving an alert from an SELinux diagnostic tool also - might not be a good idea while you're trying to analyse your system.

Good luck with solving your problem.

Myles C.


Average of ratings: -
In reply to Myles Carrick

Re: Linux load problems at 16:14 each day

by Richard Williamson -

Thanks for the reply Myles. I've checked *all* cron jobs and there's nothing that's set for that time. There is a performance data gathering routine which runs once per minute, but this spike only happens at 16:14.

I don't think the slowdown is related to memory, as there were only a few users online, and there's 4GB in the server. I'm a little puzzled as to why there are mor Apache processes than users loggen in. Do old processes take a while to 'die'?

You're probably right about not running a GUI, as i do 99% of the admin via webmin, but I'm not that confident just yet.

We've just had a new server with a quad core Xeon and 8GB delivered which I'm in the process of setting up. I'd like to get this problem sorted before I migrate things across, though.

Regards
RW

Average of ratings: -
In reply to Richard Williamson

Re: Linux load problems at 16:14 each day

by Michael Spall -
Could a cron job be running on a different machine that is accessing this one, a backup process or some other IT thing?

Are you running stats? This can cause spikes. And if the stats are not finishing correctly they can start running at times other than the scheduled time. Our stats keep creeping back 5 min. But it is tied to a cron job, so your no cron at 6:14 seems to contraindicate. Look in the Moodle docs for stats and:
http://tracker.moodle.org/browse/MDL-11302

Average of ratings: -