We have a problem with making a backup without a progress bar being shown. The Waiting time (TTFB) is very high, nearly as much as the time of the backup. When the backup is nearly finished a progressbar is shown or the message that the backup is ready. In the meantime the webpage remains unchanged (I can see the request is in progress).
This behavior is not the case in every backup test and on different networks. Could it be a problem with our firewall that blocks the response of this specific backup process?
I would guess that if it is different on different networks it has something to do with the connection between those networks and the server. As such, I would not think it was the firewall on the server itself as you have said it works fine with some other networks but maybe something on the remote networks that is slowing it down or even the end user machines. Are the networks you are connecting from local vs. outside where it is the outside networks that show the problem?
We use a proxy yes, I think the problem is the Javascript calls with Ajax that don't work well via this connection.
We don't see anything happen while debugging until the process on the server is finished (check the printscreen)
We've solved the problem in Bluecoat and progressbar works fine now. But with a large course backup (3.8GB) the backup finishes with a white screen and the backup didn't finish (no new backup in the backup screen). In the temp folder the data of the backup remains. Smaller courses finish without problems.
What could go wrong at this point (the progressbar is at 97,3% at that moment)?
Not sure anyone could respond to the question ... no two courses, unless one is an exact clone of the other, would be alike, thus the process for backing them up is also different. Size is but one factor.
So suggest turning on debugging and attempt backup that 3.8 gig course (which isn't really all that large) again to see if Moodle will report something useful. In the temp directory used for the course backup, BTW, there is a .log file and if a backup fails that log file might give you a clue ... if one could figure out the numbers associated with the stages/plan - ie, backing files, or activities, etc.
Things to check:
PHP setting for memory a script can consume. Think default is 128M .... and the backup process will attempt to use even more. Server apache error logs, might show a line about 'couldn't allocate X bytes' is an indication the memory for a script to consume needs to increase.
PHP setting for time for a script to run ... fact that it gets close to the end of the process could indicate it's just timing out.
As a test, one could take apache out of the loop and run the backup.php script found in moodlecode/admin/cli/ ... just have to give the course ID as a parameter.
That actually uses php cli not the php apache uses.
Without debug or clips from server logs, anyone's 'guess'! ;)
'spirit of sharing', Ken
So suggest turning on debugging and attempt backup that 3.8 gig course (which isn't really all that large) again to see if Moodle will report something useful. In the temp directory used for the course backup, BTW, there is a .log file and if a backup fails that log file might give you a clue ... if one could figure out the numbers associated with the stages/plan - ie, backing files, or activities, etc.
Things to check:
PHP setting for memory a script can consume. Think default is 128M .... and the backup process will attempt to use even more. Server apache error logs, might show a line about 'couldn't allocate X bytes' is an indication the memory for a script to consume needs to increase.
PHP setting for time for a script to run ... fact that it gets close to the end of the process could indicate it's just timing out.
As a test, one could take apache out of the loop and run the backup.php script found in moodlecode/admin/cli/ ... just have to give the course ID as a parameter.
That actually uses php cli not the php apache uses.
Without debug or clips from server logs, anyone's 'guess'! ;)
'spirit of sharing', Ken
Thanks for the tips Ken, I'll try to get some more info via debugging. The log file doesn't show more information, and the folder with the course files stays in the temp folder (if the backup was finished, normally it would have been deleted).
I'll also try to make a backup via the cli script, just to check if that one works.
If a backup is successful, there will be a 0 byte size .log file. If the log file is larger than 0 bytes, it does contain a log of the plans it was executing 100 -> 1000 (1000 means success - which means you'll never see that one cept in one case ... when the built .mbz backup of the course couldn't be copied to the destination directory.
IF that temp directory contains an .mbz file, the backup was successful, but the setup/system has a hickup in using 'copy' to get the .mbz file to designated directory.
Progress bar in this last case might get to 99.x% and then appear to freeze.
But you say the one course is getting to a % less than that, so ???? that's why debug and checking of error logs.
Thought of one other one might see in raw logs .... 'server as gone away'. That is really a MySQL setting for max_allowed_packet ..
https://dev.mysql.com/doc/refman/5.5/en/packet-too-large.html
Then there is also an overall description of your setup ... web server we know has the moodle code (your's has a proxy), but what of DB server ... same server as moodle code or is it a dedicated/shared DB server? How about location of moodledata? That on an NFS .... that also could be the reason for some 'large' backups failing.
So ... rather than guess any more ... debug info please.
'spiirt of sharing', Ken
This is the last debug info in the logfile:
[Tue Jan 02 17:03:17.597625 2018] [:error] [pid 30063] [client 10.132.209.11:46821] setting controller status to 1000, referer: https://academie.belastingdienst.nl/backup/backup.php?id=5928
[Tue Jan 02 17:03:17.607282 2018] [:error] [pid 30063] [client 10.132.209.11:46821] saving controller to db, referer: https://academie.belastingdienst.nl/backup/backup.php?id=5928
The .mbz file is still in the temp backup folder but finally isn't copied to the data server. It seems the copy hasn't succeeded. I don't see it in the backup section screen.
The data (NFS share) and the database is situated on a dedicated server
max_allowed_packet = 16M (on the database MariaDB)
Missed some info requested ... time for a script to run (php setting).
IF it is 30 seconds ... which is default ... increase it. I normally do that to 90 seconds on a new install in anticipation of things such as you are experiencing now. * see below
That change will require restart of apache service.
max_allowed_packet is set for it's default right now.
Think I'd increase it to 32M ... better to have more than enough than not enough - besides, you have a dedicated DB server so you should be able to do that without much concern.
https://mariadb.com/kb/en/library/server-system-variables/#max_allowed_packet
* NFS mount ... am going to assume that moodledata on the web server is the NFS mount. The lions share of what's in moodledata *is* filedir. This to say you might need to look into the config of the NFS mount.
Anything you can do to tweak it's performance?
Think you mentioned CentOS but not what version - below is 6, if you have 7 change the search URL below:
https://www.google.com/search?q=NFS+performance+CentOS+6
*** Not suggesting you need to do the following, but I've had to do it before ...
Set the mount point on the web server for *only* for moodledata/filedir - ie, just filedir
All the other directories in moodledata are physcially on the web server.
Why? If one looks at the entire backup process, moodledata/temp/backup/ is used to either backup a course or restore a course, the faster the code can work with that area (this assumes no issues with communication with DB server) the better. Other things that relate ... the creation of session files is faster, changes to session files is also faster ... the caches are on same disk and faster, etc., etc.
Also something else to *consider* ... set the backup defaults (including autobackups) to a designated directory ... not the 'sea' of files in filedir. That designated directory over on the NFS server and it can still be accessed via a symlink from moodledata/repository/backups/ that is actually another NFS mount.
When/if one needs space ... easier to find what might be the largest files ... those of backups *IF* one knows where they are located and can see them by humanly re-cognizable names - rather than having to do DB query's to find conenthash ... blah, blah, blah.
'spirit of sharing', Ken
Hi Alain,
https://moodle.org/mod/forum/discuss.php?d=310501 could provide food for your NFS thoughts .
HTH,
Matteo
Matteo and Ken, good news (and both thanks for all the help), our backup works fine now!
I changed several settings of the NFS client and server and I have made an enormous performance boost with these new settings! I don't see any broken backups anymore (also not in the temp backup folder) so this seems to be the solution for our backup problems.
First problem was the Bluecoat configuration we solved (the progressbar was visible after that changes).
Next I tuned the database settings with the mysqltuner.pl script and with the tips I found here in the forums.
Last I changed the settings of the NFS client/server.
On the NFS client (in /etc/fstab):
[our_server_url]:/share/moodledata /share/moodledata nfs4 _netdev,relatime,async,hard,rsize=262144,wsize=262144,tcp,timeo=600,intr 0 0
On the NFS server (in /etc/exports):
/share/moodledata [our_server_url](rw,async,anonuid=601,anongid=601)
Thanks Ken, I appreciated your help .
Here are some of the links where I found info about NFS configuration and performance:
http://nfs.sourceforge.net/nfs-howto/ar01s05.html
https://www.slashroot.in/how-do-linux-nfs-performance-tuning-and-optimization
https://access.redhat.com/articles/1172303 (needs access login to the Redhat website, useful article)
For our environment it took some time to figure out the best settings for NFS, the backup/restore/import processes were good use cases to test this performance.
For some monitoring commands om Linux we used this article, but there are better monitoring solutions I think:
https://medium.com/netflix-techblog/linux-performance-analysis-in-60-000-milliseconds-accc10403c55