Hi,
my moodle site is down. the following message is display. what have happened and how to fix it?
For the love of all things, review your database log file!
While the front-end errors are often very generic the back-end server logs are very specific and detailed. It will likely even tell you which file and while line of code caused the issue.
Far to many recommend simply reboot or restart the service. If something serious is wrong you may lose your database.
I am familiar with my own mysql error log, whose filename takes on the form "/var/lib/mysql/ip_stuff.err." I am not sure if you consider this to be front-end or back-end. Could you give me an example of what the second log filename might look like? Thanks.
ip_stuff.err - is this some kind of windows thing?
I can't say I've ever seen a log file named .err on linux before
The majority of your logs will be in /var/log
/var/log/nginx/error.log
/var/log/mysql/error.log
tail -f 200 /var/log/mysql/error.log
Front-End
This is what is displayed to the user while accessing the website through a browser.
Back-End
Log files residing on the server with significant detail related to the actual error. Typically one can figure out within minutes exactly what is wrong by reviewing the log files associated with the website.
-G
.err logs do exist if DB server configured to use them.
They are DB server boot up/shutdown logs which gives more info than what a services restart command would.
As far as overall logging ... a very busy Moodle site should probably not have logging turned on by default as those log files could grow quite large in a hurry.
IF one needed to watch MySQL logs they can be turned on using some mysql set global commands.
mysql -e 'set global log = 1' -u root -p'password'
mysql -e 'set global log_slow_queries = 1' -u root -p'password'
mysql -e 'set global slow_query_log = 1' -u root -p'password'
Then one could use multitail thusly:
multitail -i /var/log/mysql/mysql-general.log -i /var/log/mysql/mysql-slow.log
To turn off using superuser priv's:
# set global slow_query_log = 0
# set global log_slow_queries = 0
# set global log = 0;
'spirit of sharing', Ken
Thanks Ken.
I am still trying to figure out why my mysql database dies exactly every 31 days, and then I have to restart the database. I am still searching various log files with no luck. The mysql error log never has anything in it suggesting the problem, just that I stopped and restarted the server to fix it.
I have checked all of the crons that I think might be causing this, and haven't found any with a 31 day cycle. Somehow I think it might be connected with a cron job running on the physical server which hosts my VPS.
Cron jobs/task in Moodle do not give reboots/restarts to DB Daemon/service. For that to occur on a regular basis and consistently it has to be operating system/*hosting provider* setup. Just because you can ssh into and see say CentOS 6 or whatever doesn't mean that AMP stack hasn't been changed/customized to meet whatever criteria, etc. for the hosting provider.
On standalone .... not remotely hosted with hosting provider but a physical box or VMWare hosted ... I've seen uptime for mysqld (the DB server) reach 400+ days - *IF* I never restart it nor upgrade it, etc.. and have not rebooted the server for anything else.
One could, however, schedule their own cron job to restart the MySQL server.
Got CPanel? Got WHM? ... could be in there as well.
Suggest checking with hosting provider.
'spirit of sharing', Ken
Yep Ken, I know what you mean. Yes, checked cPanel, WHM, hosting provide, no solution yet. The odd thing is that a mysql stop/ mysql start doesn't fix the problem. I have to go to WHM and restart the mysql service. Then that fixes it.
I plan to experiment with the command "service mysql restart" next time. This problem will occur again on 11/16/16 between 3AM and 5AM, so I have a little time to prepare again.
This has only been happening this year starting in January. Never had this problem before. I really think that it is on the physical server. Or that someone else's VPS on this physical server is causing this problem. Yep, my hosting provider is not finding anything.
Ken,
I may have missed this, but have you tried support at WHM? They have an amazing support team.
What is their website URL that you find gets you to their best support?
If you need a new home for Moodle, let me know I have extra room.
Interesting, however in 5+ years I've yet to read or see a .err file.
I would not be shutting down or restarting just to determine what is wrong.
Logging should be on by default if setup properly and that applies to nginx, php-fpm, mysql, mariadb, etc. Implying someone who knows what they are doing is running the server. Thus you would never find large log.
Plus a lot of database related errors even show up in PHP log files.
Bottom line if your site is crashing, down, or experiencing issues. The first place you should check are your various log files. (nginx, php, mysql)
Also it does not hurt one bit to install something like newrelic (which is free) to monitor resources, disk usage, and notify you of issues. Better to get a warning than have the server crash for some stupid reason that could have been avoided.
What I referred to ip_stuff.err is the same as /var/log/mysql/error.log.
@M - learned something new 10/2016
example clip of .err file that was useful in troubleshooting a troubled server
that ran out of space.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
160924 10:47:39 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
and later ...
160924 11:08:04 InnoDB: ERROR: the age of the last checkpoint is 9435003,
InnoDB: which exceeds the log group capacity 9433498.
InnoDB: If you are using big BLOB or TEXT rows, you must set the
InnoDB: combined size of log files at least 10 times bigger than the
InnoDB: largest such row.
@Rick
ps aux |grep mysqld |grep .err
What do you see?
@Nellie ... you must mean Rick.
And correction ... 'logging should probably not be turned on' ....
should have referred to query log and slow query logging should probably not be
turned on by default.
'spirit of sharing', Ken
Total mess due to running out of diskspace (yet again). Simple free server monitor like newrelic can notify admin or owners before shit hits the fan.
Thank you, friends.
Our server was being attacked. Now we have reset the server data and our site is now normal.
Consider using Cloudflare in front of your server.
Generally schools are pretty geo oriented which means you may even be able to block entire regions from access your server. In addition I've managed to push off a few pretty heavy attacks simply by only allowing CloudFlare traffic to reach the server (meaning no direct connections). Essentially an affordable way to protect / withstand various attacks.