Hello Moodlers - I have set up a handful of Moodle sites over the last few years but this issue is new for me. Our Moodle site is painfully slow (30+ seconds to even see the first page load) but after that pain it's fast and happy. I've run Apache optimization scripts to ensure the web server is tuned, same for MySQL (it is 3.2 running on Ubuntu BTW with 8 cores and 28 GB or RAM which seems more than adequate to me). It uses SSL. I have run DNS checks and tested on different OS's, browsers inside our school network, outside but same thing. Although I know there could be various factors involved can I ask for some suggestions and guidance? This is a production server and it's causing a lot of grief.
Are you using any kind of caching?
Hi Emma - I might need to ask how I go about finding this out, sorry.
Are you using DB for session information? Initial access is like guest ... moodle server will send a session cookie upon initial access for browsers.
Is the DB server one same server as web server?
Have you been working on theme designs? Is designer mode on/off?
Network comes before application ... so even if you have tweaked everything you can tweak and checked things like DNS response times there could still be something at the network layer causing slow response times upon initial access. Many schools have some sort of server that checks inbound - and tracking from both internal workstation access and external access May not show in things like traceroutes nor in pings.
Might want to talk to your network folks about anything like that.
'spirit of sharing', Ken
Hello Ken - thanks for the suggestions. The MySQL DB is located on a separate server but on the same subnet (it's all in Azure cloud actually). True, some customization has been going on and I do know about the theme designer mode trick - have often had to turn it off but not the case this time I'm afraid.
We host a few other such setups in Azure and no similar issues I'm afraid. What's maddening is that, while the site may not be perfect, it's speedy enough once it's been loaded by at least one client and after that they and subsequent clients (different users) have no speed issues.
Well, now you've shard some details that should be included in your investigation ....
DB on separate server ... even if on same IP segment of the network IT is still remote and it still involves networking. In config of MySQL do you have networking off? In the Moodle config lines for DB server are you using FQDN - assuming Windows shortcut names? or internal IP address?
A bottle neck there would affect anyone hitting the site for the first time .. almost every day cause Moodle tries to log everything ... even before you login.
Uhhh .... I help admin servers that are under VMWare ... so anything that is a layer above the guest OS and Moodle code ... regardless if there are other servers on the same Azure or not ... Azure might be playing a factor. One one of the 'fun ones' I had ... another Guest OS on the same VMWare server where the moodle resided was configured a certain way to grab all the resources of the hardware under certain conditions ... which meant, from time to time, the Moodle was not responding, etc.. No amount of tweaks to Moodlle server wold overcome that.
This to say .. you can't rule out anything just yet .... follow 'paths'. Inside access by workstation -> Switch in segment of the LAN WAN (IDF) -> MDF -> DMZ switch -> Azure -> Moodle. Outside access by anyone ... gateway and other boxen like Barracuda ... then the path above similar. See what I mean?
So .... ?????
'spirit of sharing', Ken
Thanks again Ken. Yes, won't rule out anything yet I agree. Networking is off on the MySQL server and it access the web server using an internal address - we've found the Azure infrastructure to be very quick vm-to-vm as it uses internal VIP's but I do hear you re: networking should always be scrutinized. The config.php file is set such that the database is resolved/connected via this internal only IP, a virtual LAN in effect.
Well then, what have you done to collect any info?
How about FireFox with Web Developer tools turned on. Hit the site to see what's going on ... or not going on ...
You'd have to do that cause so far now one knows the URL to the site.
Uhhhh .... do it from inside and take screen snaps, copy/paste info into something to compare with what you see when doing the same thing from outside.
Dunno what folks think is so secretive about a Moodle URL that's gonna be scanned and indexed already or soon enough that they feal they can't share the URL here.
'spirit of sharing', Ken
Hi Ken - I have just today been using the Benchmark Report/plugin. Made some changes to MySQL query caching, table optimization, etc. The only reason I have not posted the URL is that I have been regularly hitting it with different tools to measure page performance, installing an update, etc. I suspect I will finish these tasks off soon and would be happy to post the URL - cheers and thank you for the comments.
And here is the URL: myilsc.com
I know there are some issues with the javascsript (Fordson theme) and try as I might whenever I upload compressed versions of images (our logo for example) it just doesn't seem to stick and it reverts or re-creates older/larger/non-compressed versions of the logos.
Are you hosting this site on your own network?
What sort of network test have you done from a location outside of your own network?
Traceroute from my location at hop 15 shows only one reponse @78ms but then it seems to stay at the same hop and displays 3 host/ip's ... all on same segment ... @78.2, @70.1 before no responses to the trace. Those are on msn.net.
As far as response times etc. in FireFox web tools network, no issues that I can see ... heck, very little on front page (forcing all to login to see anything).
Have seen slower sites! ;)
Did see Google Analytics is involved. Got anything else like that which might be more 'stealthy'?
Might be interesting to see it from your server .... install iptraf and run it while using a browser that's had cookies/cache/etc. wiped clean.
'spirit of sharing', Ken
Hmm, well that's interesting. First, about the networking yes I know it's weird. In Azure, ICMP echo (ping) is not allowed, well to the public IP anyway) so if it comes back as in-pingable that's normal. When doing Tracert I know it goes through many hops and there are some timeouts etc - once it hits the MS Edge network it goes dark. Not the easiest to troubleshoot from a networking perspective I know.
Have been using various tools like gtmetrix and pagespeed insights form Google and they each point to issues re: large images, too much JS needing to load at the beginning, etc. Now some of these are tied to the theme and others perhaps not. Have setup a job on another webserver to hit the front page every 5-min as our initial thoughts were the webserver is sleeping and the reason the first visit is so slow is because it needs to crank up the whole site from scratch but subsequent visits are fast (from other users, not just the first user).
Will check out iptraf - thanks very much Ken, cheers.
Most networks do block ICMP echos at certain points. Understandable. While not apparently a major factor was reporting that at hop 15 it bounced around a block of IP addresses ... same class C ... three times, before hitting the next hop. Might be a good idea for you to do traceroutes on a regular basis and record them. Why? You nor your school controls networking beyond your boundary router ... things could change.
Still the response times were not in the 100's ... that's good.
Ok, nothing on front page .... google analytics there however. Tools like pagespeed not appropriate for dynamic sites (those back-ended by a DB ... found that out the hard way with an Apache server loading pagespeed extension/mod. So that particular tools recommendations I'd not trust.
So its slow *after* someone logs on. Admin levels are understandable ... profile for that level user is larger ... remember that themes are 'inline' so there's more to render for an admin level.
The other thing that goes on upon login ... establishing a session and logging - and the rendering of what a user sees upon login. So what blocks/other content does one see *after* logging on? How are your students/teachers authenticating? LDAP? Where is that LDAP server ... internal or is it also 'out there' in Azure.
Moodle caches a bunch now. Are you running memcache/d? Will show my ignorance about Azure, but will ask ... how about Azure itself? Does it cache?
Uhhh .... don't think webservers actually 'sleep'. So the checker you've got running am not sure serves the purpose you think. On the server run top and watch. ;) Using swap space?
You might also install a thing called multitail .... allows one to view mutlple logs on one screen ... the access..log, the error.log, and the messages..log might be a place to start with that.
Now that's something you've not mentoned .... how is the Ubuntu box serving Moodle and with what? apache with fastcgi or as mod? You did say you tweaked it some.
Technically, BTW, your school is almost like 'from outside' (cause it is outside msn) with the only exception being what filtering and bandwidth shaper other such network software your school might have. Your server is truely cloud ... 100%.
No easy answer!
'spirit of sharing', Ken
When you say "first" visitor do you mean to a specific page or to the site as a whole?
So do you mean the first access to the site when not logged in? Or is the login slow? Or is a course page slow (either a specific one or all pages?). Once it's "fast and happy" does that persist forever or does the issue start again, e.g. the next day or over some other time period.
Can you walk us through a sequence of events -- e.g. 1. User goes to home page, 2. User logs in, 3. User goes to a course -- and say where the problem occurs?
Have you got the latest Moodle 3.2? There was a problem a few months ago some theme elements took a long time to be generated the first time (https://tracker.moodle.org/browse/MDL-58646 I think).
Thanks for the follow-up Leon. I mean the site as a whole - no login, course access or any other page - just hitting the front page. The site is fast and happy for a while - I can't say how long precisely but maybe a few hours and then it seems to sleep and needs a visitor to wake it up.
The site is running 3.21 (build 20170109) and I see there is a 3.25+ update available. I do know 3.3 is out but our testing with our other Moodle sites has not been great on 3.3 so we are holding off for now. I will look at 3.25+ especially if it has this rolled into it: (https://tracker.moodle.org/browse/MDL-58646 I think).
Try testing with older non Boost theme. It looks like the cache is being purged and first visitor after the purge forces theme cache to be rebuild (boost themes take lots of time compared to non boost ones).
If this is the case, then you need to find out why the cache is being purged, probably designer mode being turned on.
Thanks Grzegorz - I know Boost has some issues and someone kindly pointed me to a patch for that. We have also done some theme customizations and I know there are some slowdowns there too re: image size, JS, etc. have checked already re: designer mode as I know that has pretty devastating effects Thanks again for the Boost insight - I will review that and another member, Ken, had some thoughts about network so these too are ongoing investigations.
I have the same issue, please tell me how did you stop the cron caching job that comes with the theme
Thanks in advance
We just disabled the cron job - it was causing more harm than good.
You disabled the moodle cron job?? That will stop your site from performing a lot of necessary functions...like emailing people, updating completion, etc. etc...
Or was it a separate cron job? And if so, what theme were you using that installed its own cron job?
Sorry, let me back up a bit:
1. Issue was front page would take a very long time to load, but once loaded would then respond properly;
2. Discovered a cron job running every 30 minutes which cleared the Moodle cache;
3. Site was using a Boost-based theme and initial performance hit was due to the theme caching information before returning the page results to the browser;
4. It appeared to the be the front page only as it would be experienced on first page load after the cache was cleared;
5. The standard Moodle cron job was not deleted/disabled however the "purge cache" every 3-min cron job was as this was contributing to the slow page loads mentioned above.
Did you disabled from the administrative panel on moodle?
Or from the cPanel ?
Because I couldn’t find any option to disabled within the moodle theme ?
Can you give me a detailed instructions on how to do that?
Thanks once again
Hi there - this particular cron job (purge caches) was created via command line on the server along with the standard Moodle cron job. Depending on your host you may or may not have SSH access to your server and/or the ability to do this via cPanel. If you're not comfy using the command line on a production server you might want to look at alternatives and the Moodle documentation does provide a few different ways of doing this. If you don't have a 30-min purge caches job then your solution/problem may in fact be different from ours but happy to help in any event.
Thank you Walter