We are in the phase of implementing Moodle 3.2 and doing Load runner test for 500 users but we are getting 500 internal server error and 503 service unavailable for scorm and video (6 minutes) based courses.
Below is the our server resources details.
1) . 3 web front ends/applications server - Windows Server 2012 R2, IIS 8.5, RAM-128GB, CPU Intel Xeon E5 2660 v3 @2.60GHz
2). PHP 5.6.7
3). Dedicated MSSQL 2016 Server.
5) Load balancing is happening through F5 (round robin)
6) Implemented memcache.
7). IIS application pool recycling happens at 2.8GB private and virtual memory utilization.
Enabled the FailedRequestLogs and below is the response for 500.
https://localhostXXXXXX:443/pluginfile.php/2798/mod_video/content/1/Who..mp4, STATUS_CODE 500, 90172 ms, GET
https://localhostXXXXXXX:443/course/?mycourses=1, STATUS_CODE 200, 1031 ms, GET
Please help us to tune the system.
Thanks for your response. We do not find any errors in IIS logs but we find connection_dropped and AppOffline in HTTPerror logs.
HTTP/1.1 GET /pluginfile.php/7535/mod_scorm/content/2/story_content/txt__default_6pTk8hpahL0.swf - 1 Connection_Dropped DefaultAppPool
443 HTTP/1.1 GET /pluginfile.php/7535/mod_scorm/content/2/story_content/txt__default_6pTk8hpahL0.swf - 1 Connection_Dropped DefaultAppPool
We will enable debugging and do the test. May I know where errors will get stored because if it throws errors and warnings on the page then we may not be able to capture the errors as we use HP Load Runner for load testing.
clearly your MP4 content looks like culprit, taking 90 seconds to fetch.
do two things, first confirm you're using NTS (non-thread safe) version of PHP, and secondly try using PHP 7.x.
also tell what kind of drives you have?
with this CPU and this much of RAM, you dont need to put SQL Server on dedicated machine, as well need a load balancer for 500 (assuming concurrnet) users.
Thanks for your response. Yes, we do suspect that video/swf (or any file is big in szie) is the culprit here. Actually we do see errors appearing while application pool gets recycled. Recycling takes longer than 90 secs(w3wp.exe/app pool process stays in STOPPING state for long ). We have set it to 1200 seconds to avoid failures.
We had same issue in Moodle 1.9 (WIndows OS 2008 R2 platform) when we introduced video cources and w3wp.exe was not moving above 3.2GB of RAM and came to know 32 bit application cannot utilize memory > 3GB. So we had set recycling at 2.8GB of memory in IIS and it would work fine and it would recycle in within seconds we did not have any issues.
Coming to your questions.
Yes, we are using PHP ver 5.6.7 NTS version.( Since we have customizations at themes level and do not know whether it supports PHP 7.0.x, I need to check with developers).
We are using VMWARE Virtual disk SCSI Disk device (all our servers are VM).
Yes, 500 count is concurrent users. Our company has around 50,000 employees , so we went for 3 web front ends and one sql server and configuration of each server mentioned in my first post.
I have attached screenshot of IIS app pool settings, fast cgi settings, php.ini and config values in the document. Kindly look at it.
Also, we do use separate content/shared data server(uses same typical VM disk and all servers reside in within the network and ping response between the servers is < 1ms.) and installed memcache service on it. We do get better response times for single memcache server(content server) but not from clustered memcache setup (setting up memcache on each web front end) for video courses.
Your help is highly appreciated. Thank you!!
The limit of 32-bit is 3.5G, and uptil PHP 5.x, 64-bit wasnt fully supported on windows platform, but since PHP 7 it is, so your full memory allocation can be utilized. regarding customized theme support, themes arent dependent upon PHP, but core is, where PHP 7 is suported since Moodle 3.0.1, I dont think it should be an issue on your moodle. I'll suggest giving it a try together with OpCache, it will make a difference in performance.
Also, you did mentioned Xeon E5, are these single CPU servers or multi? set Maximum Worker process value to 0 in Process Model, as yours showing 1, this as well applies to Web gardens (which is essentially your case) even with Single socket servers, as then there will be 1:1 affinity between worker processes and NUMA nodes.
Thank you !! We will definitely try using PHP 7.0 and will share the results.
Yes, we have multi CPU, I had set web garden to 0 and saw only two worker processes were getting triggered. As we did observe issues only during recycling of app pool and with web garden we did not see any performance improvement so disabled it,
We dont have video file size of 3.5GB. We have video files mp4 with 20-50MB when 500 users access the same course videos then we have an issue. We do get 500 and 503 errors when app pool gets recycled. It takes long time to recycle.
I was about to upgrade PHP 7.1 but came across following limitation. https://docs.moodle.org/32/en/Windows_installation
"Windows is not suitable for large Moodle installations because PHP for Windows does not support 64-bit integer numbers, even if it is running on 64-bit Windows in 64-bit mode. One of the problems is that Moodle may fail to access data if there are more than 2 billion records in one database table because the id column is too big for integers. Another limitation is a maximum file size of 2GB, which can cause problems (for example) when making backups of large courses. Please consider a different operating system with full 64-bit support for very large installations."
Can we still go for 64 bit PHP for moodle 3.2 considering we are on Windows 2012 R2 64 bitplatform?