Could anyone recommend a good application monitoring tool/program which would log and alert us if moodle is not responding optimally?
Not so much on networking side but on application and database side.
We are using 1.9.14+ (soon to move up to 2.1). On Redhat 6.2 with php 5.3.3 and MySQL 5.1.45
An open source solution is ideal.
Looked a Nagios briefly but not sure if it can monitor applications and if so, how deep can it monitor? slow queries? backup is slow, which scripts are taking long to execute, etc.
nagios can probably do anything you want if you're willing to configure it.
For the last while, we've been using munin + monit. munin gives nice graphs to watch the system, and monit actively monitors things. (for munin, there are an improved set of mysql agents sitting out on the web that replace the internal ones and make a big difference in tracking systems.)
The apachestatus monitor that comes with monit is a great way to figure out when the server is misperforming in various ways. you can check the status of the apache process to make sure that it isn't spending too much time on dns, logging, or waiting. You can also monitor directories, files, mysql variables, and other running processes.
What do you mean by "application monitoring"? I'm trying to collect the tools for
monitoring Moodle servers here: "How do you monitor your Moodle server?" http://moodle.org/mod/forum/discuss.php?d=192162. There are already subthreads on Munin and Nagios.