XAMPP on Windows Server 2012 running in production

XAMPP on Windows Server 2012 running in production

by Michael Hankins -
Number of replies: 19

We've been using Moodle to conduct placement tests in our Uni for the past 3 years without issues, but yesterday we had major issues and had to cancel the tests. We kept getting 'network connection lost', 'database is overloaded', 'blank pages' when a student clicks on next etc., 

No one actively manages the VM that Moodle is installed on. I've learned from this forum that XAMPP is not an ideal setup for production. That's what we have! running on a Windows Server 2012 R2. We have bumped up CPU and RAM but are still having issues. 

I've been tasked with finding out why the test failed. Where do I start? Do I tell them that the setup they have is not for production/150 concurrent users? How has this worked in the past? They are definitely going to bring this up!  





Average of ratings: -
In reply to Michael Hankins

Re: XAMPP on Windows Server 2012 running in production

by Ken Task -
Picture of Particularly helpful Moodlers

So you are playing the role of Bob Crachit and want help talking to the Scrooge's at a Uni?  Hmmmmm  ....

Clues ... 'unmanaged VM' ... Windows 2012 R2 ... venturing a guess the Moodle has also been 'unmanaged'.

Why it failed requires more info and inspection of logs of Apache, MySQL, PHP.

Only you can do that ... no one has access to your 'server' and don't believe anyone will ask for access to it (no Vulcan  Mind Meld possible!).   So guess you are asking for someone here in  community to write up justifcation for upgrading?  Uhhh ... don't they call that 'consulting'? smile

So far we know XAMPP (and you already know not an ideal setup) ... but ... for the sake of exercise ... will begin ...

How much memory was allocated?  How much of that memory is being used by: Apache?  by MySQL?
Version of Moodle? Versions of Apache, MySQL, PHP. Can you locate error logs?  Share clips of those logs around the time/date your entity experienced the issues.  (that request says 'clips' ... not entire log files).

For the mysql config, logged onto the client:
mysql> show variables like '%max%';

That should be enough to start the disussion/cussion, me thinks!  Consider this a response from Ghost 1! smile

'spirit of sharing', Ken



Average of ratings: Useful (1)
In reply to Ken Task

Re: XAMPP on Windows Server 2012 running in production

by Michael Hankins -

Thanks for the pointers 

I did have a look at the logs, some are so huge I can't even open them. How do I start managing these logs? I did find this at the time the exam failed:

[Thu Mar 07 08:59:02.919255 2019] [mpm_winnt:error] [pid 2044:tid 10444] AH00326: Server ran out of threads to serve requests. Consider raising the ThreadsPerChild setting

Bellow is the current settings:

# WinNT MPM

# ThreadsPerChild: constant number of worker threads in the server process

# MaxConnectionsPerChild: maximum number of connections a server process serves

<IfModule mpm_winnt_module>

    ThreadsPerChild        300

    MaxConnectionsPerChild   0

</IfModule>

We are running Moodle version: 3.0.4 (they are refusing to upgrade as this is tied to an in-house ERP system) The consultant has also convinced them that Apache is the only way. They also refused to upgrade to 64-bit. At this point, I'm trying to increase memory allocation to Apache, PHP and MySQL.  php.ini memory_limit is set to 128M. Cron is not running. I'm a bit concerned as this path is bound to fail during the next exam. Is there a way I can test the server to see if can handle 100 concurrent users?

In reply to Michael Hankins

Re: XAMPP on Windows Server 2012 running in production

by Leon Stringer -
Picture of Core developers Picture of Particularly helpful Moodlers

That must make me the second ghost.

As you say XAMPP isn't recommended for production use and here's one of the reasons: XAMPP's builds of Apache and PHP have always been 32-bit (64-bit builds were issued for the first time this week!). So regardless of the RAM added, only 4 GB could ever be used by Moodle. The 32-bit build also prevents Moodle from performing file operations on files larger than 2 GB or handling values larger than 2^32 - 1 (e.g. from the database). This could contribute to issues over time as the amount and size of files grows over time, and the number of rows in the database increases.

You don't say what version of Moodle is in use but this could be an older version too?

One thing you could check is that the site cron is running, if it's not you'll see a message on Site administrationNotifications.

You could look into moving to a 64-bit XAMPP build. You'll need to check that your version of Moodle works with the PHP version in the XAMPP release you download. I strongly recommend you do a test migration first to make sure everything works before trying it with the live site.

Generally if you run Moodle on Microsoft Windows I strongly recommend using IIS instead of XAMPP/Apache if that's a way forward for you.

Average of ratings: Useful (1)
In reply to Leon Stringer

Re: XAMPP on Windows Server 2012 running in production

by Michael Hankins -

Is there a way to manage the database, like remove old quizzes, exams etc., Cron is not running. Does a cron job clean out log files? 

In reply to Michael Hankins

Re: XAMPP on Windows Server 2012 running in production

by Leon Stringer -
Picture of Core developers Picture of Particularly helpful Moodlers

If you want to remove quizzes and exams you should probably delete them but normally you want to hide them (or the course) until you're absolutely sure you want to delete them. Depending on your settings these may be initially moved to the Recycle Bin before being later deleted by the cron.

You should get the cron running, it performs housekeeping which, if not done, would certainly lead to a drop in performance over time.

Moodle keeps logs, by default these are retained indefinitely, it makes sense to set the Keep logs for to something appropriate for your organisation, e.g. 365 days. The cron would then clear these logs in line with this setting.

(Does XAMPP rotate Apache's log files? Maybe check these too and rename them if they're huge).

In reply to Michael Hankins

Re: XAMPP on Windows Server 2012 running in production

by Usman Asar -
Picture of Plugin developers Picture of Testers

Michael, you've got one thing right, XAMPP is not for production, XAMPP is usually used when base operating system doesnt comes with a web server of its own (Windows Home editions) so developers can use XAMPP to develop web applications.

anyway, now you have to dig in history, 150 concurrent users isnt very resource demanding but there are plenty of factors playing the role to degrade the service overtime. so you have to find out, what was the initial moodle version installed? as it must have been upgraded overtime and every major moodle update comes with added features at the cost of hardware resource.

secondly, was the test questions remained the same throughout? as they must be changed on yearly, semi-annually basis, this adds to the calls to database.

thirdly, what kind of VM is your Windows server hosted on? as VM's (unless a true hypervisor is used) adds a layer of host operating system beneath, this adds to latency, therefore a direct hardware install is always better than a VM.

Leon already have mentioned 32-Bit, and its limitations of using RAM (3.5GB it is), but have you as well tried allocating more RAM resources to your DB? rather than just allocating to VM?

Lastly, using IIS would be best option, as you will be able to run same tests on same hardware, as sometime ago I had this scenario came up where moodle was setup on XAMPP and though was working, but very slow, so I switched it to IIS on weekend night, and first mail I was sent by the teacher in morning was from student saying what a speed difference, and its been 2 years, no complaints of speed or system getting down ever since.

my advice, switch to IIS and things will improve very very likely.

Average of ratings: Useful (1)
In reply to Usman Asar

Re: XAMPP on Windows Server 2012 running in production

by Michael Hankins -

The question bank changes for every exam. One instructor has 200 another has 125. The VM is hosted on an ESXi host. To increase memory on the db should I edit to my.cnf file? The consultant they called shot down IIS. 

In reply to Michael Hankins

Re: XAMPP on Windows Server 2012 running in production

by Usman Asar -
Picture of Plugin developers Picture of Testers

yes please, but in database optimization, Ken is recommended for consultation, do ask for his input. Also as you mentioned cron hasnt been running, do a manual cron run.

In reply to Usman Asar

Re: XAMPP on Windows Server 2012 running in production

by Ken Task -
Picture of Particularly helpful Moodlers

Not feeling left out here ... don't run Windows.   Besides, OP has been shot down in his attempts to 'improve' the situation.   There does come a point when trying to force square peg into round hole is futile - even if one 'shaves the peg' ... which is what has been proposed now.

Ok, ... so ... tweaking DB ... install mysqltuner.pl ... it's a pearl script.   Run it with super user credentials (db user). and let's see what it says.   Without knowing anything about current settings and data mysql might have on the system performance, etc., any  .... *ANY* recommendation for a tweak to my.cnf would be 100% guess and could actually makes things worse, not better.

As to testing for 100 concurrent users in a Quiz ... only by jmeter ... and it's not recommended one do that on a production server.

Hate to tell ya, but I too, think (from description) one better prepare for a failure.   Just make sure that just prior to the session, make a full site backup ... code, DB, and data directory.

Anyhoo ... let's have a look at mysqltuner output to see what it says ... that will help when the decision is made to move the site to a better environment.   Make the other consultant earn his/her $$$!!!! smile

'Boy Scout' approach ... don't ask superiors, but spin up a LAMP box that's a clone of your current setup but has more memory/etc.   The customizations might not make any diff in taking the quiz.  Use same FQDN and you cheat with your host file to get that server prep'd and ready.   If quiz fails like a whale ... change DNS to point to new server.   Yes, it's a gamble ... you could be 'hero'!   Will superiors ever know?

'spirit of sharing', Ken



In reply to Ken Task

Re: XAMPP on Windows Server 2012 running in production

by Michael Hankins -

I ran MySQL Tuner, please see attachment.

Attachment LNLkNwAbnG.png
In reply to Michael Hankins

Re: XAMPP on Windows Server 2012 running in production

by Michael Hankins -

I noticed that query_cache_type entry missing in my.ini but has a query_cache_size set to 64M. Is it enabled? 

In reply to Michael Hankins

Re: XAMPP on Windows Server 2012 running in production

by Ken Task -
Picture of Particularly helpful Moodlers

WARNNIG .... balancing act ... between apache and MySQL on same server.
With 32 bit PHP you are greatly limited ... period.

Is your MySQL also 32 bit?

User @Linux ... so there is a linux server in the mix here?  Is that other server where the customized code goes?

If that's true, would think your server could run linux ... and then you wouldn't have all of these issues!

MySQL 5.5.36
Check https://dev.mysql.com/doc/refman/5.5/en/


Tuner ...
* 345 Fragmented Tables
* Maximum possible memory usage 1.5G 6% of installed ram ...

**** Highest connection usage 152 or 151
151 is default.  This setting coiuld have been one of the major factors in issues last testing session.
How many users are testing at one time?  Set variable to one over max number
of users taking quiz.  Yeah, that's probably too many but better to have more than enough then not enough! 

*** SEE BALANCING ACT WARNING ABOVE!

**** Innodb data size / buffer pool ... 1.8G/256M
Buffer pool needs to be slightly larger than data size
If you noticed the recommEndations made by tuner at the bottom.
innodb_buffer_pool >= 1G ... I'd make it 2G
Also need 1 buffer pool instance per Gig.

Changes to my.cnf require restart of MySQL ... tuner run again right after restart won't have usage data so you'll have to take a test yourself to get some data.


Where is says run Optimize table ... before doing that make a backup of
the DB (sql dump).

Dunno if it comes with Windows, but ...
mysqlcheck  command using superuser creds.
SYNOPSIS
       mysqlcheck [options] [db_name [tbl_name ...]]

DESCRIPTION
       The mysqlcheck client performs table maintenance: It checks, repairs,
       optimizes, or analyzes tables.

Now with InnoDB it will complain and what it should do is check and rebuild the tables rather than optimize.
which will help the 345 fragmented table issues.

Looks like one had tweaked wait_timeout and interactive_time outs in attempts to fix issue but both can be reduced below current value < 28800

'spirit of sharing', Ken


Average of ratings: Useful (1)
In reply to Ken Task

Re: XAMPP on Windows Server 2012 running in production

by Michael Hankins -

User @Linux ... so there is a linux server in the mix here?  Is that other server where the customized code goes?

- Not sure where that came from. No Linux server in our currently.

**** Highest connection usage 152 or 151

- I've bumped this up max_connections in my.ini correct?

**** innodb_buffer_pool >= 1G ... I'd make it 2G

- MySQL would not start with 2G. 

190313  9:49:35 InnoDB: Initializing buffer pool, size = 2.0G

InnoDB: VirtualAlloc(2175795200 bytes) failed; Windows error 8

190313  9:49:35 InnoDB: Completed initialization of buffer pool

190313  9:49:35 InnoDB: Fatal error: cannot allocate memory for the buffer pool

190313  9:49:35 [ERROR] Plugin 'InnoDB' init function returned error.

190313  9:49:35 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.

190313  9:49:35 [ERROR] Unknown/unsupported storage engine: InnoDB

190313  9:49:35 [ERROR] Aborting


***** Looks like one had tweaked wait_timeout and interactive_time outs in attempts to fix issue but both can be reduced below current value < 28800

- Where are these values? 


***** Where is says run Optimize table ... before doing that make a backup of the DB (sql dump).

- Running optimize all databases. Will run tuner and post the results.





In reply to Michael Hankins

Re: XAMPP on Windows Server 2012 running in production

by Leon Stringer -
Picture of Core developers Picture of Particularly helpful Moodlers

Just a reminder that it's Apache, PHP and MySQL that are 32-bit, i.e. none of these can address more than 4 GB (probably 3.5 GB as Usman points out, possibly less). So you can't allocate more than half of that space to just one MySQL buffer (a quick search supports this). So it's not surprising that 32-bit MySQL wouldn't start with innodb_buffer_pool set to 2 GB.

Average of ratings: Useful (1)
In reply to Leon Stringer

Re: XAMPP on Windows Server 2012 running in production

by Ken Task -
Picture of Particularly helpful Moodlers

Thanks, Leon.    Compiling ... wow ... takes me back to the early 90's.  Been there ... done that ... good experience ... but frustrating at times ... and today, not necessary at all.

Puts the last nail in the coffin.

"placement tests in our Uni" ... "No one actively manages the VM that Moodle is installed on"

This goes deeper than tech, me thinks.  And there might some tech aware students (more than likely are) who think that way as well.  Are these "placement test" free or is there a fee?  Such things that fail for technical reasons that could be addressed ... one might wonder just how good they would have been in 'placement' if they had no technical issues.  Such things (square peg round hole - sledge hammer) are a reflection of the Uni overall.

Think the OP has enough info/proof, etc. that current setup/system is in need of an upgrade and there is no reason that can't happen ... given VM, Linux is free, Moodle is free, etc and all that is required is someone to 'think differently' to get the job done.

Best of luck to Op!

'spirit of sharing', Ken


In reply to Ken Task

Re: XAMPP on Windows Server 2012 running in production

by Michael Hankins -

I met the powers that be and explained our current setup, why it failed, why it will continue to fail and gave them options to mitigate this situation. Got approved for a new server! Will keep this running until the migration is completed. 


I would like to thank the experts here who made this happen. Your input was invaluable. Ken, Leon & Usman.. Hats off to you guys! 


I've made a few enemies in the process! Saving face in front of the students is all I really care about. It's their first experience at the Uni and these tests were making IT look really bad. 


Thanks once again, I've learned a lot with this experience. 

In reply to Michael Hankins

Re: XAMPP on Windows Server 2012 running in production

by Ken Task -
Picture of Particularly helpful Moodlers

Welcome! (am sure I can say that from all who responded/tried to help).

Congrats!   Boy, if it's that hard to improve, pity!

Hmmm .... maybe time to give Linux a try!   Many do ... and never look back! smile

Ken


In reply to Michael Hankins

Re: XAMPP on Windows Server 2012 running in production

by Ken Task -
Picture of Particularly helpful Moodlers

What does mysql -V show from command line?

Ken

In reply to Ken Task

Re: XAMPP on Windows Server 2012 running in production

by Michael Hankins -

# mysql -V

mysql  Ver 14.14 Distrib 5.5.36, for Win32 (x86)

Making a few tweaks as per your recommendations shortly.