The longer the day goes it seems to get slower. Students on the outside of the network on the internet complain more than the students behind the firewall on the inside network do. I have some information attached that i have gathered to help out. Any help would be greatly appreciated.
(Split from the discussion http://moodle.org/mod/forum/discuss.php?d=57028. Edited by Visvanath Ratnaweera - original submission Wednesday, 1 August 2012, 10:00 PM)
student that are behind the firewall complain less then the one outside of the firewall on the internet.
The longer the day goes it seems to get slower. Students on the outside of the network on the internet complain more than the students behind the firewall on the inside network do.
Base Unit:
PE R515 with up to 8 Hot Swap Hard Drives, LED and AMD Opteron 42XX Procs (225-2372)
Processor:
AMD Opteron 4162EE, 6C 1.7GHz, 2M L2/6M L3, 1333Mhz Max Mem (317-5456)
AMD Opteron 4162EE, 6C 1.7GHz, 2M L2/6M L3, 1333Mhz Max Mem (317-5465)
Memory:
32GB Memory(8x4GB),1333MHz Dual Rank LV UDIMM for 2 Procs,Sparing (317-5503)
Our setup is like this: we have 2 R515's with 32 gig of RAM each
1 Tera of HD in a RAID 5 configuration.
The Servers are configured with ESXi 4.01 and one server with our web on it and the other is the db. Both servers are running Cent OS 5.7, with Zend php and MySQL. We have a Vcenter server running another box.
Running th preformanc.php file gave me these numbers
Disable the record cache (Site Admin -> Server -> Performance) for realistic results!
Processor performance
Function calls 342000 344000
Regular expression replaces over 1KB of text 11700 11700
Disk performance 16KB files read from disk (cache) 32900 34000 16KB files written to disk (cache) 12100 11400
Database performance
Get_record calls on the course table 1810 1800
Insert_record calls on the course table 1260 1250
Update_record calls on the course table 1360 2420
Maximum concurrent users (approx): 112
Mytop on Db
MySQL on localhost (5.5.20) up 19+18:18:22 [10:23:38]
Queries: 9.0 qps: 0 Slow: 0.0 Se/In/Up/De(%): 00/00/00/00
qps now: 0 Slow qps: 0.0 Threads: 2 ( 1/ 0) 00/00/00/00
Key Efficiency: 100.0% Bps in/out: 0.0/ 0.0 Now in/out: 8.4/ 1.7k
Id User Host/IP DB Time Cmd Query or State
-- ---- ------- -- ---- --- ----------
1005524 root localhost moodle 0 Query show full processlist
1005625 moodle inet22a moodle 0 Sleep
Top on DB
top - 10:25:32 up 19 days, 18:21, 2 users, load average: 0.11, 0.07, 0.04
Tasks: 167 total, 1 running, 165 sleeping, 1 stopped, 0 zombie
Cpu(s): 1.2%us, 0.5%sy, 0.0%ni, 97.8%id, 0.1%wa, 0.1%hi, 0.2%si, 0.0%st
Mem: 24685404k total, 3926156k used, 20759248k free, 270620k buffers
Swap: 12025848k total, 0k used, 12025848k free, 2588704k cached
Unknown command - try 'h' for help
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4072 mysql 15 0 591m 195m 6004 S 5.3 0.8 218:49.13 mysqld
4244 root 15 0 60596 20m 6208 S 1.0 0.1 5:31.68 Xvnc
13015 root 15 0 283m 16m 8796 S 0.7 0.1 0:02.99 gnome-terminal
3573 dbus 15 0 21640 1292 732 S 0.3 0.0 1:16.62 dbus-daemon
22957 root 15 0 12760 1148 828 R 0.3 0.0 0:00.15 top
1 root 15 0 10368 688 572 S 0.0 0.0 0:02.18 init
2 root RT -5 0 0 0 S 0.0 0.0 0:00.61 migration/0
3 root 34 19 0 0 0 S 0.0 0.0 0:00.35 ksoftirqd/0
4 root RT -5 0 0 0 S 0.0 0.0 0:00.50 migration/1
5 root 34 19 0 0 0 S 0.0 0.0 0:00.06 ksoftirqd/1
6 root RT -5 0 0 0 S 0.0 0.0 0:00.42 migration/2
7 root 34 19 0 0 0 S 0.0 0.0 0:00.37 ksoftirqd/2
8 root RT -5 0 0 0 S 0.0 0.0 0:00.45 migration/3
9 root 34 19 0 0 0 S 0.0 0.0 0:00.02 ksoftirqd/3
10 root 10 -5 0 0 0 S 0.0 0.0 1:12.45 events/0
11 root 10 -5 0 0 0 S 0.0 0.0 0:00.02 events/1
12 root 10 -5 0 0 0 S 0.0 0.0 0:00.08 events/2
Top on the web server
top - 10:28:46 up 12 days, 2:20, 2 users, load average: 0.42, 0.75, 0.76
Tasks: 190 total, 1 running, 189 sleeping, 0 stopped, 0 zombie
Cpu(s): 16.2%us, 1.0%sy, 0.0%ni, 82.2%id, 0.0%wa, 0.1%hi, 0.5%si, 0.0%st
Mem: 23651168k total, 5446960k used, 18204208k free, 402648k buffers
Swap: 12025848k total, 0k used, 12025848k free, 3545108k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14798 apache 16 0 1669m 87m 50m S 30.6 0.4 0:26.83 httpd
14993 apache 15 0 1670m 86m 48m S 26.6 0.4 0:14.43 httpd
4072 root 15 0 56568 16m 5292 S 1.0 0.1 6:06.19 Xvnc
14751 apache 15 0 1669m 90m 52m S 1.0 0.4 0:51.61 httpd
14845 apache 15 0 1668m 85m 48m S 1.0 0.4 0:22.49 httpd
14924 apache 15 0 1668m 83m 48m S 1.0 0.4 0:22.97 httpd
14951 apache 15 0 1667m 83m 46m S 1.0 0.4 0:21.36 httpd
14978 apache 15 0 1668m 82m 46m S 1.0 0.4 0:15.41 httpd
15044 apache 15 0 1667m 77m 42m S 1.0 0.3 0:05.90 httpd
15104 apache 15 0 1654m 57m 31m S 1.0 0.3 0:01.33 httpd
481 root 15 0 0 0 0 S 0.7 0.0 3:46.01 pdflush
14442 root 15 0 263m 15m 8484 S 0.7 0.1 0:00.72 gnome-terminal
14878 apache 15 0 1668m 85m 49m S 0.7 0.4 0:31.61 httpd
14881 apache 15 0 1669m 84m 48m S 0.7 0.4 0:20.16 httpd
14936 apache 15 0 1669m 86m 50m S 0.3 0.4 0:14.63 httpd
14995 apache 15 0 1667m 85m 49m S 0.3 0.4 0:18.24 httpd
15008 apache 15 0 1667m 82m 46m S 0.3 0.4 0:13.72 httpd
15009 apache 15 0 1668m 84m 48m S 0.3 0.4 0:10.05 httpd
1 root 15 0 10368 684 572 S 0.0 0.0 0:03.57 init
2 root RT -5 0 0 0 S 0.0 0.0 0:01.10 migration/0
3 root 34 19 0 0 0 S 0.0 0.0 0:00.24 ksoftirqd/0
4 root RT -5 0 0 0 S 0.0 0.0 0:01.01 migration/1
5 root 34 19 0 0 0 S 0.0 0.0 0:00.14 ksoftirqd/1
6 root RT -5 0 0 0 S 0.0 0.0 0:00.80 migration/2
7 root 34 19 0 0 0 S 0.0 0.0 0:00.03 ksoftirqd/2
8 root RT -5 0 0 0 S 0.0 0.0 0:00.84 migration/3
Here is my configuration of the eAccelerator.ini file on my web sever
Here is the php info for eAccelerator print out and below is the ini file from the server
Here is my configuration of the eAccelerator.ini file on my web sever
##################################################################### # ############################ ;
You must uncomment one (and only
>> one) line from the following to load ; eAccelerator extension.
>> extension="eaccelerator.so"
>> ;zend_extension="/usr/lib/php/modules/eaccelerator.so"
>> ;zend_extension_ts="/usr/lib/php/modules/eaccelerator.so"
>>
>> ;extension="eaccelerator.dll"
>> ;zend_extension_ts="c:\php4\eaccelerator.dll"
>> ;zend_extension="c:\php4\eaccelerator.dll"
>>
>> ; The amount of shared memory (in megabytes) that eAccelerator will use.
>> ; "0" means OS default. Default value is "0".
>> eaccelerator.shm_size = "1024"
>>
>> ; The directory that is used for disk cache. eAccelerator stores
>> precompiled ; code, session data, content and user entries here. The
>> same data can be ; stored in shared memory also (for more quick
>> access). Default value is ; "/var/cache/php-eaccelerator".
>> eaccelerator.cache_dir = "/var/cache/php-eaccelerator"
>>
>> ; Enables or disables eAccelerator. Should be "1" for enabling or ;
>> "0" for disabling. Default value is "1".
>> eaccelerator.enable = "1"
>>
>> ; Enables or disables internal peephole optimizer which may speed up
>> code ; execution. Should be "1" for enabling or "0" for disabling.
>> ; Default value is "1".
>> eaccelerator.optimizer = "1"
>>
>> ; Enables or disables debug logging. Setting this to 1 will print
>> information ; to the log file about the cach hits of a file.
>> eaccelerator.debug = 0
>>
>> ; Set the log file for eaccelerator. When this option isn't set then
>> the data ; will be logged to stderr eaccelerator.log_file =
>> "/var/log/httpd/eaccelerator_log"
>>
>> ; A string that's prepended to all keys. This allows two applications
>> that use the ; same key names to run on the same host by setting this
>> in .htaccess or in the main ; configuration file for the whole webserver.
>> eaccelerator.name_space = ""
>>
>> ; Enables or disables PHP file modification checking. Should be "1"
>> ; for enabling or "0" for disabling. You should set it to "1" if you
>> want ; to recompile PHP files after modification. Default value is "1".
>> eaccelerator.check_mtime = "1"
>>
>> ; Determine which PHP files must be cached. You may specify the
>> number of ; patterns (for example "*.php *.phtml") which specifies to
>> cache or ; not to cache. If pattern starts with the character "!", it
>> means to ignore ; files which are matched by the following pattern.
>> Default value is "" that ; means - all PHP scripts will be cached.
>> eaccelerator.filter = ""
>>
>> ; When eAccelerator fails to get shared memory for new script it
>> removes ; all scripts which were not accessed at last "shm_ttl"
>> seconds from shared ; memory. Default value is "0" that means - don't
>> remove any files from ; shared memory.
>> eaccelerator.shm_ttl = "60"
>>
>> ; When eAccelerator fails to get shared memory for new script it
>> tries to ; remove old script if the previous try was made more then "shm_prune_period"
>> ; seconds ago. Default value is "0" that means - don't try to remove
>> any ; files from shared memory.
>> eaccelerator.shm_prune_period = "7200"
>>
>> ; Enables or disables caching of compiled scripts on disk. It has no
>> effect ; on session data and content caching.
>> ; Default value is "0" that means - use disk and shared memory for caching.
>> eaccelerator.shm_only = "0"
>>
>> ; The script paths that are allowed to get admin information and do
>> admin ; controls eaccelerator.allowed_admin_path = ""
>> ;sets the content to compress before saving it to memory ;
>> eaccelerator.compress = "1"
>> ;Sets the compress level
>> ;
>> eaccelerator.compress_level = "9"
Pardon me for 'jumping in here', but happen to use CentOS and have had to trouble shoot a similar issue involving only outside users and Moodle.
Towards that endeavor (which turned out to be firewall and iptables issue), learned a lot about server hardware and network. Shared below:
First, qualification for this attempted response … am NOT a network engineer, but happen to assist schools which run Moodles on similar hardware/setups. So the response below is/may be lacking some tools to test the issues.
The issue is speed of connectivity and corresponding response of systems. In following such a path for trouble shooting, one must consider network before application.
The following will help conceptualize;
Running virtuals on X hardware doesn't always mean network is optimal. Obviously, if network is not optimal, application is more than likely not optimal either.
There is a firewall involved in connectivity for users outside the private network.
Those users are coming into the network via the 'front door' and might have connectivity issues with their provider (unless users have a VPN setup). Users outside may have many options for their connectivity. One way to collect such information is to create an HTML block on the home page of the Moodle, that says 'Click here if having connectivity issues'.
That presents a popup window with nothing more than a 'Gracias' image and wording … along with the fact their ip address has been recorded in logs - something web servers do anyway.
To see where and how outside users are coming into the 'front door', one need only to search for the 'Gracias' page in apache access_log's. Using ip address whois, one can collect information on network they are connected to and provider.
And there are firewalls on both servers as well - since CentOS mentioned, more than likely iptables.
Think the best strategy for this would be to work on users inside the private network first. Your entity personnel can 'touch' all that is involved in connectivity between users and the Moodle server.
First … both servers are running a version of CentOS (5.7) that has been updated (5.8).
While throughly describing related areas, there is nothing shared concerning the actual networking. Question: what is the physical connection between the web server(s) and the MySQL server? Is it Fiber Channel or just 'ethernet' via virts?
What's capacity of the NIC's on both web and DB server?
As root user from both the web server and the DB server:
/sbin/ifconfig
will render information and a snap shot of current status of the NIC's.
Example (some values here changed to 'protect' the responder):
[root@www ktask]# /sbin/ifconfig
eth0 Link encap:Ethernet HWaddr 00:22:34:75:96:00
inet add:IP address Bcast:IP address.255 Mask:255.255.255.0
inet6 addr: fe80::222:19dd:fe64:8600/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:63529554 errors:0 dropped:0 overruns:0 frame:0
TX packets:29990015 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:7694649081 (7.1 GiB) TX bytes:5667098831 (5.2 GiB)
Interrupt:185 Memory:ec000000-ec012800
Are there dropped packets between the web server and the DB server?
Example: logged onto a web server constant ping to DB server:
[root@moodle ktask]# ping 192.168.100.18
PING 192.168.100.18 (192.168.100.18) 56(84) bytes of data.
64 bytes from 192.168.100.18: icmp_seq=1 ttl=64 time=0.226 ms
64 bytes from 192.168.100.18: icmp_seq=2 ttl=64 time=0.226 ms
64 bytes from 192.168.100.18: icmp_seq=3 ttl=64 time=0.204 ms
64 bytes from 192.168.100.18: icmp_seq=4 ttl=64 time=0.233 ms
64 bytes from 192.168.100.18: icmp_seq=5 ttl=64 time=0.218 ms
64 bytes from 192.168.100.18: icmp_seq=6 ttl=64 time=0.203 ms
64 bytes from 192.168.100.18: icmp_seq=7 ttl=64 time=0.206 ms
64 bytes from 192.168.100.18: icmp_seq=8 ttl=64 time=0.227 ms
64 bytes from 192.168.100.18: icmp_seq=9 ttl=64 time=0.223 ms
64 bytes from 192.168.100.18: icmp_seq=10 ttl=64 time=0.236 ms
64 bytes from 192.168.100.18: icmp_seq=11 ttl=64 time=0.266 ms
--- 192.168.100.18 ping statistics ---
11 packets transmitted, 11 received, 0% packet loss, time 10000ms
rtt min/avg/max/mdev = 0.203/0.224/0.266/0.021 ms
Also do the reverse … logged on to DB server, constant ping to the Web server.
Are there dropped packets between a user inside the network and the web server?
User workstation/device would have to be used to answer the one above.
Are the times of ping responses acceptable and consistent?
Might look into installing lshw (list hardware). Provides much more technical information about hardware … including NICS and can output into -html for viewing with a browser.
yum -y install lshw
Other tools:
dmesg grep eth0
/sbin/mii-tool -v eth0
/sbin/ethtool eth0
All show eth0 but if you have two NICS configured then use appropriate device designator … ie, eth1
'spirit of sharing', Ken
Thank you ken for jumping in great questions!!
Question: what is the physical connection between the web server(s) and the MySQL server? Is it Fiber Channel or just 'ethernet' via virts?
>gig ethernet post running to a blackdiamond switch with gig ports on both servers.
I have another gig ethernet nic not being used on the server
What's capacity of the NIC's on both web and DB server?
>According to the ethtool 1000m/s full
As root user from both the web server and the DB server:
>both server show no errors,dropped,overruns?
Are there dropped packets between the web server and the DB server?
Example: logged onto a web server constant ping to DB server:
web to DB--- ping statistics ---
57 packets transmitted, 57 received, 0% packet loss, time 56037ms
rtt min/avg/max/mdev = 0.333/0.530/2.373/0.310 ms
DB to Web--- ping statistics ---
46 packets transmitted, 46 received, 0% packet loss, time 45023ms
rtt min/avg/max/mdev = 0.282/0.422/0.801/0.098 ms
Are the times of ping responses acceptable and consistent?
look but i'm wondering the avg time is a little high?
Am still reviewing everything you've shared so far.
Is the operating system of the DB box 64 bit or 32? MySQL can take advantage of it, web server not so much so.
In virtuals, NICs can be bonded together. Think checking into using that free (un-plugged nic) might be in order - anything to increase speed.
Also forgot ... check Apache config. How many daemons are launched on startup? Do server logs ever report an error concerning 'load'?
fgrep load /var/log/httpd/error_log*
Ever do any 'stress testing' at the web server(s) or the DB server?
Here's something to do yourself (using connection outside the private LAN/WAN) AND have a couple of students do same. For that matter, go to 'popular' locations yourself where students inside the network would use tech, plugin/connect, and test from there as well.
Constant ping to your Moodle server and let it run for a minute or two.
On a Mac: traceroute your.moodle.server
On a PC: tracert your.moodle.server
How many hops (routers traffic runs through to get to point B)? (ask the volunteers to copy the route text and paste into an EMail back to you - please, NO MORE 'word doc attached'. ;)
Was there any hop that took longer than others?
'spirit of sharing', Ken
thank you ken for posting
Here is what i came up with to your questions
Am still reviewing everything you've shared so far.
Is the operating system of the DB box 64 bit or 32? MySQL can take advantage of it, web server not so much so.
Both boxes are 64bit Cent OS 5.7
In virtuals, NICs can be bonded together. Think checking into using that free (un-plugged nic) might be in order - anything to increase speed.
Also forgot ... check Apache config. How many daemons are launched on startup? Do server logs ever report an error concerning 'load'?
fgrep load /var/log/httpd/error_log*
[Thu Aug 02 15:16:17 2012] [notice] caught SIGTERM, shutting down
[Thu Aug 02 15:16:17 2012] [notice] core dump file size limit raised to 18446744073709551615 bytes
[Thu Aug 02 15:16:17 2012] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Aug 02 15:16:17 2012] [notice] Digest: generating secret for digest authentication ...
[Thu Aug 02 15:16:17 2012] [notice] Digest: done
[Thu Aug 02 15:16:18 2012] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads.
[Thu Aug 02 15:16:18 2012] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations
[Thu Aug 02 15:19:31 2012] [notice] caught SIGTERM, shutting down
[Thu Aug 02 15:19:32 2012] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Aug 02 15:19:32 2012] [notice] Digest: generating secret for digest authentication ...
[Thu Aug 02 15:19:32 2012] [notice] Digest: done
[Thu Aug 02 15:19:32 2012] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads.
[Thu Aug 02 15:19:32 2012] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations
[Thu Aug 02 15:19:44 2012] [error] [client 127.0.0.1] File does not exist: /usr/moodle/moodle_data
[Thu Aug 02 15:23:26 2012] [notice] caught SIGTERM, shutting down
[Thu Aug 02 15:23:36 2012] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Aug 02 15:23:36 2012] [notice] Digest: generating secret for digest authentication ...
[Thu Aug 02 15:23:36 2012] [notice] Digest: done
[Thu Aug 02 15:23:36 2012] [notice] mod_python: Creating 4 session mutexes based on 512 max processes and 0 max threads.
[Thu Aug 02 15:23:36 2012] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations
[Thu Aug 02 15:59:23 2012] [error] [client 127.0.0.1] File does not exist: /usr/moodle/moodle_data
[Thu Aug 02 16:07:35 2012] [error] [client 127.0.0.1] File does not exist: /usr/moodle/moodle_data
[Thu Aug 02 16:13:05 2012] [error] [client 127.0.0.1] File does not exist: /usr/moodle/moodle_data
[Thu Aug 02 17:02:46 2012] [error] [client 127.0.0.1] File does not exist: /usr/moodle/moodle_data
[Thu Aug 02 20:14:09 2012] [error] [client 127.0.0.1] File does not exist: /usr/moodle/moodle_data
[Thu Aug 02 20:21:13 2012] [error] [client 10.1.4.46] File does not exist: /usr/moodle/favicon.ico
[Thu Aug 02 21:28:37 2012] [error] [client 127.0.0.1] File does not exist: /usr/moodle/moodle_data
[Thu Aug 02 21:33:15 2012] [error] [client 127.0.0.1] File does not exist: /usr/moodle/moodle_data
[Thu Aug 02 21:52:35 2012] [error] [client 127.0.0.1] File does not exist: /usr/moodle/moodle_data
[Fri Aug 03 08:02:20 2012] [error] [client 127.0.0.1] File does not exist: /usr/moodle/moodle_data
[Fri Aug 03 09:42:44 2012] [error] [client 127.0.0.1] File does not exist: /usr/moodle/moodle_data
[Fri Aug 03 10:04:19 2012] [error] [client 10.1.4.46] File does not exist: /usr/moodle/favicon.ico
[Fri Aug 03 10:52:19 2012] [error] [client 10.1.4.46] File does not exist: /usr/moodle/favicon.ico
[Fri Aug 03 11:12:05 2012] [error] [client 127.0.0.1] File does not exist: /usr/moodle/moodle_data
[Fri Aug 03 12:15:20 2012] [error] [client 10.1.4.46] File does not exist: /usr/moodle/favicon.ico
[Fri Aug 03 12:28:36 2012] [error] [client 127.0.0.1] File does not exist: /usr/moodle/moodle_data
[Fri Aug 03 12:53:23 2012] [error] [client 127.0.0.1] File does not exist: /usr/moodle/moodle_data
[Fri Aug 03 14:18:22 2012] [error] [client 127.0.0.1] File does not exist: /usr/moodle/moodle_data
[Fri Aug 03 14:21:20 2012] [error] [client 127.0.0.1] File does not exist: /usr/moodle/moodle_data
[Fri Aug 03 14:32:37 2012] [error] [client 10.1.4.46] File does not exist: /usr/moodle/favicon.ico
[Fri Aug 03 14:33:00 2012] [error] [client 127.0.0.1] File does not exist: /usr/moodle/moodle_data
[Fri Aug 03 15:28:50 2012] [error] [client 127.0.0.1] File does not exist: /usr/moodle/moodle_data
[Fri Aug 03 15:50:25 2012] [error] [client 127.0.0.1] File does not exist: /usr/moodle/moodle_data
[Fri Aug 03 15:57:52 2012] [error] [client 10.1.4.46] File does not exist: /usr/moodle/favicon.ico
[Fri Aug 03 15:58:41 2012] [error] [client 10.1.4.46] File does not exist: /usr/moodle/favicon.ico
[Fri Aug 03 15:59:40 2012] [notice] caught SIGTERM, shutting down
[Fri Aug 03 15:59:41 2012] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Fri Aug 03 15:59:41 2012] [notice] Digest: generating secret for digest authentication ...
[Fri Aug 03 15:59:41 2012] [notice] Digest: done
[Fri Aug 03 15:59:42 2012] [notice] mod_python: Creating 4 session mutexes based on 512 max processes and 0 max threads.
[Fri Aug 03 15:59:42 2012] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations
[Fri Aug 03 16:06:03 2012] [error] [client 127.0.0.1] File does not exist: /usr/moodle/moodle_data
[Fri Aug 03 16:09:12 2012] [error] [client 10.1.4.46] File does not exist: /usr/moodle/favicon.ico
[Fri Aug 03 16:14:33 2012] [error] [client 127.0.0.1] File does not exist: /usr/moodle/moodle_data
[Fri Aug 03 16:24:37 2012] [error] [client 10.1.4.46] File does not exist: /usr/moodle/favicon.ico
[Fri Aug 03 16:25:30 2012] [error] [client 10.1.4.46] File does not exist: /usr/moodle/favicon.ico
[Fri Aug 03 16:25:38 2012] [error] [client 10.1.4.46] File does not exist: /usr/moodle/favicon.ico
[Fri Aug 03 16:27:09 2012] [error] [client 10.1.4.46] File does not exist: /usr/moodle/favicon.ico
[Fri Aug 03 16:29:47 2012] [error] [client 10.1.4.46] File does not exist: /usr/moodle/favicon.ico
[Fri Aug 03 16:34:42 2012] [error] [client 10.1.4.46] File does not exist: /usr/moodle/favicon.ico
(process:1670): libgnomevfs-WARNING **: Unable to create ~/.gnome2 directory: Permission denied
[Fri Aug 03 17:02:17 2012] [error] [client 10.1.4.46] File does not exist: /usr/moodle/favicon.ico
[Fri Aug 03 17:02:54 2012] [error] [client 10.1.4.46] File does not exist: /usr/moodle/favicon.ico
[Fri Aug 03 17:03:22 2012] [error] [client 10.1.4.46] File does not exist: /usr/moodle/favicon.ico
[Fri Aug 03 17:04:11 2012] [error] [client 10.1.4.46] File does not exist: /usr/moodle/favicon.ico
[Fri Aug 03 17:06:57 2012] [error] [client 10.1.4.46] File does not exist: /usr/moodle/favicon.ico
[Fri Aug 03 17:10:14 2012] [error] [client 10.1.4.46] File does not exist: /usr/moodle/favicon.ico
[Fri Aug 03 17:12:03 2012] [error] [client 10.1.4.46] File does not exist: /usr/moodle/favicon.ico
[Fri Aug 03 17:13:15 2012] [error] [client 10.1.4.46] File does not exist: /usr/moodle/favicon.ico
[Fri Aug 03 17:13:56 2012] [error] [client 10.1.4.46] File does not exist: /usr/moodle/favicon.ico
[Fri Aug 03 17:14:02 2012] [error] [client 10.1.4.46] File does not exist: /usr/moodle/favicon.ico
[Fri Aug 03 17:15:55 2012] [error] [client 10.1.4.46] File does not exist: /usr/moodle/favicon.ico
[Fri Aug 03 17:18:15 2012] [error] [client 10.1.4.46] File does not exist: /usr/moodle/favicon.ico
[Fri Aug 03 17:24:01 2012] [error] [client 10.1.4.46] File does not exist: /usr/moodle/favicon.ico
[Fri Aug 03 18:43:49 2012] [error] [client 127.0.0.1] File does not exist: /usr/moodle/moodle_data
[Fri Aug 03 20:59:19 2012] [error] [client 127.0.0.1] File does not exist: /usr/moodle/moodle_data
[Fri Aug 03 21:00:10 2012] [error] [client 127.0.0.1] File does not exist: /usr/moodle/moodle_data
[Fri Aug 03 22:43:24 2012] [error] [client 10.1.4.46] File does not exist: /usr/moodle/favicon.ico
[Sat Aug 04 01:12:25 2012] [error] [client 10.1.4.46] File does not exist: /usr/moodle/favicon.ico
[Sat Aug 04 02:50:26 2012] [error] [client 10.1.4.46] File does not exist: /usr/moodle/favicon.ico
[Sat Aug 04 09:56:47 2012] [error] [client 10.1.4.28] File does not exist: /usr/moodle/favicon.ico
Ever do any 'stress testing' at the web server(s) or the DB server?
>no we havent
Here's something to do yourself (using connection outside the private LAN/WAN) AND have a couple of students do same. For that matter, go to 'popular' locations yourself where students inside the network would use tech, plugin/connect, and test from there as well.
Constant ping to your Moodle server and let it run for a minute or two.
>yes I did that from WKS to Web and WKS to DB avg ping rate was 0.5 sec inside the network
On a Mac: traceroute your.moodle.server
>>havent tried this yet
On a PC: tracert your.moodle.server
How many hops (routers traffic runs through to get to point B)? (ask the volunteers to copy the route text and paste into an EMail back to you - please, NO MORE 'word doc attached'. ;)
>>I have cacti running on a machine in our inside network to monitor the web and db servers now trying to see if that can give me any clues.
I see some things in the apache log that need investing ... also see it shutdown a couple of times (SIGTERM) ... on it's own?
Check apache access log and error log for events just prior to date/time seen on the SIGTERM line.
This reference:
[error] [client 127.0.0.1] File does not exist: /usr/moodle/moodle_data
Something on the server (127.0.0.1) is trying to find /usr/moodle/moodle_data What is it?
Under what user does your httpd daemon run? Typically it's 'apache' user 'apache' group. What's /usr/moodle/moodle_data? Is that the moodle data folder for your site?
Is the disk partitioned?
df
will show
Plus, see it's loading python module. Using anything python? One of the 'standard' practices for web server admin is not to load any modules not used.
/usr/sbin/httpd -M
as root user (not a 'moodle' user) will show modules loaded. One can remove loading of modules via the httpd.conf file.
/usr/sbin/httpd -V
as root user, will show how apache was compiled.
Can you find any .core files? .core files are Core dumps created when server crashes. They are written to the application directory that caused the crash.
At this point, I'd recommend you go through your httpd.conf file ... don't forget to look at any .conf file in /etc/httpd/conf.d
Problem *might be* with Apache config.
'spirit of sharing', Ken
here is the output of the Apache modules
Server version: Apache/2.2.3
Server built: Oct 20 2011 17:00:12
Server's Module Magic Number: 20051115:3
Server loaded: APR 1.2.7, APR-Util 1.2.7
Compiled using: APR 1.2.7, APR-Util 1.2.7
Architecture: 64-bit
Server MPM: Prefork
threaded: no
forked: yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/prefork"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/etc/httpd"
-D SUEXEC_BIN="/usr/sbin/suexec"
-D DEFAULT_PIDLOG="run/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_LOCKFILE="logs/accept.lock"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"
And here is the Http -M output
Loaded Modules:
core_module (static)
mpm_prefork_module (static)
http_module (static)
so_module (static)
auth_basic_module (shared)
auth_digest_module (shared)
authn_file_module (shared)
authn_alias_module (shared)
authn_anon_module (shared)
authn_dbm_module (shared)
authn_default_module (shared)
authz_host_module (shared)
authz_user_module (shared)
authz_owner_module (shared)
authz_groupfile_module (shared)
authz_dbm_module (shared)
authz_default_module (shared)
ldap_module (shared)
authnz_ldap_module (shared)
include_module (shared)
log_config_module (shared)
logio_module (shared)
env_module (shared)
ext_filter_module (shared)
mime_magic_module (shared)
expires_module (shared)
deflate_module (shared)
headers_module (shared)
usertrack_module (shared)
setenvif_module (shared)
mime_module (shared)
dav_module (shared)
status_module (shared)
autoindex_module (shared)
info_module (shared)
dav_fs_module (shared)
vhost_alias_module (shared)
negotiation_module (shared)
dir_module (shared)
actions_module (shared)
speling_module (shared)
userdir_module (shared)
alias_module (shared)
rewrite_module (shared)
proxy_module (shared)
proxy_balancer_module (shared)
proxy_http_module (shared)
proxy_connect_module (shared)
cache_module (shared)
suexec_module (shared)
disk_cache_module (shared)
file_cache_module (shared)
mem_cache_module (shared)
cgi_module (shared)
version_module (shared)
perl_module (shared)
php5_module (shared)
proxy_ajp_module (shared)
python_module (shared)
ssl_module (shared)
Syntax OK
I'm quite sure that i'm unsure of which ones should and shouldnt be there ?
No way anyone could tell you for sure unless they had access to the server itself, but, obviously there are some you need ... php for example. Ok, nothing really looks 'out of place or strange' there, but let's repeat ...
What did the access_log/error_log show for events just prior - up to the date time of that SIGTERM log line?
Is the disk partitioned? df
The core size increase notification is rather large, don't you think? Have there been any .core dumps? Seeing the SIGTERM notification, am guessing there possibliy could be some .core files somewhere.
'spirit of sharing', Ken
Ken I dont see any .core files and here is the disk partition.
thank you for your posts
df output not viewable here. But, that's ok, if you see no partition upon which something moodle (app directory or data diretory) resides that shows it's nearly out of space, then not an issue.
Probably should have asked this a loooong time ago, but what version of Moodle running? Was Moodle installed in the 'traditional' fashion from a download at Moodle.org?
Reason I ask the last question ... /usr/moodle/moodle_data
Ken
I downloaded it from the moodle site and untared it and created the /usr/moodle directory and /usr/moodle_data directory and changed my configuration accordingly.
To the best of my knowledge, the moodle files from download.moodle.org have never created a moodle_data directory upon un-archiving.
So have also to ask ... who installed the CentOS 5.7 operating system? and the apache?
Reason asked:
"Digest: generating secret for digest authentication"
What is purpose of Apache running this?
Ken
I did the install of all the software and services running on the server
How many processes are usual to run on a Web server ? does it depend on the # of users and use?
we have always had it there sine I started back in 1.5 of moodle we have never changed it on other installs 1.7, 1.9, 1.9.4, 2.0, and now 2.2?
this is what i found on Apache sit about that
Are you restarting Apache through /etc/init.d/httpd restart ?
If you are, be aware that 'restart' is in fact a 'stop' followed by a 'start'. The stop is defined as:
stop() {
echo -n $"Stopping $prog: "
killproc -p ${pidfile} -d ${STOP_TIMEOUT} $httpd
RETVAL=$?
echo
[ $RETVAL = 0 ] && rm -f ${lockfile} ${pidfile}
}
and killproc:
killproc sends signals to all processes that use the specified executable. If no signal name is specified, the signal SIGTERM is sent.
Which mean that it is normal for you to see that "caught SIGTERM, shutting down" message
What of the log entries just prior to the SIGTERM line? Yes, am aware that if I do /sbin/service httpd restart I will see SIGTERM, but that's system admin restarting the service manually. Question, I guess, should have been, is there some event which is causing a restart of the apache server?
Does it come backup after restart cleanly ... no errors? Or does it fail to restart and then attempts yet another restart until it builds that digest?
I too have been running Moodles since 1.4 and do know that some work needed to be done with DB when upgrading. I take it that all those have been done as one upgraded over the years. What of the health check in Moodle?
These snaps of logs - which should be watched in realtime in this case - are somewhat helpful, but none (other than yourself) can see/determine if any entry in the log is significant.
Might I suggest installing Mutitail?
http://www.vanheusden.com/multitail/
When having issues, nice to be able to see multiple logs on one screen. Have a script (watchweb) where root user can call up five related Apache logs and watch in realtime - the access_log, the ssl_access_log, the error_log, the ssl_error_log, and the rewrite.log. If I run that in one terminal session window and then in another terminal session window do a /sbin/service httpd restart can SEE what's happening.
'spirit of sharing', Ken