[Moodle BenchMark] Disappointing score (1945) for a two VM setup, 12 CPU and 112 GB RAM in total

[Moodle BenchMark] Disappointing score (1945) for a two VM setup, 12 CPU and 112 GB RAM in total

by Daniel Tillman -
Number of replies: 5

Since I didn't want to totally hijack the Moodle BenchMark thread I thought I'd start a new post here.

Original Post https://moodle.org/mod/forum/discuss.php?d=335357#p1359485


Note:  I'm a history teacher by trade, who ran a personal moodle site hosted by professional hosting companies who moved into IT last year when my school adopted iPads for all students.  It was decided that I would run a moodle server for the school and that it would be hosted on one of our servers on campus.  Last year I ran the server on 1 VM with 4 cpu cores and 24 gigs of ram, but since we got some new servers for other services, I was given a whole server for moodle.  

Server Info

HP Proliant DL 360p Gen 8

2x Intel E5-2620v2 (6 cores + hyperthreading on each)

128 Gigs of Ram total

2x 7,200 RPM SAS 1tb drives (Raid 1 configuration) (datastore 1)

2x 12,00 RPM SAS 1.8tb drives (Raid 1 configuration) (datastore 2)

VMware vSphere 6 Hypervisor

- Hardware is split into 2 VM's - VM

VM1 - Moodle Web server / Moodledata on this server.

6 CPU cores, 48 gigs of ram, datastore 1 

OS - Lubuntu 14.04, Apache, mySQL, Php5

VM2 - Moodle Database server

6 CPU cores, 64 gigs of ram, datastore 2

OS - Lubuntu 14.04, Apache, mySQL, Php5


Note: I split the server into 2 VM's because I can't allocate more than 10 CPU cores to 1 VM, and I wanted to use all 12 Cores.  I am also a novice at VMware configuration.  We are using the free version and so have some limitations in configuration options/tools available from what I understand.  I may very well have something misconfigured at the VM level that is causing the issues I'm having.


I am willing to do almost anything to reconfigure this to get the performance that I should be getting - however, I am leading some training sessions on Moodle starting Monday August 1 and so I won't be able to pull the site down for a few days.  I did attempt to upgrade to Moodle 3.1 and upgrade to Ubuntu 16.04 and PhP7 but ran into some problems with Ubuntu 16.04 on the server with the display being locked to 800x600, which isn't an issue most of the time but there are times when the GUI is useful.  After being unable to resolve the issue I reset the server with Ubuntu 14.04 and did a fresh install of everything.  


The moodle site itself handles the school's 1200 students and about 300 classes total.  The most stressful time on the server last year came when we had about 500 students using the Quiz module at the same time.  This year, I expect more teachers to use it and that number to climb to about 900 - 1000 using the Quiz module at the same time for semester/final exams.


Results of the Moodle BenchMark





UnixBench Results

   System: : GNU/Linux

   OS: GNU/Linux -- 3.13.0-92-generic -- #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016

   Machine: x86_64 (x86_64)

   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")

   CPU 0: Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz (4189.9 bogomips)

          Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET

   CPU 1: Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz (4189.9 bogomips)

          Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET

   CPU 2: Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz (4189.9 bogomips)

          Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET

   CPU 3: Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz (4189.9 bogomips)

          Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET

   CPU 4: Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz (4189.9 bogomips)

          Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET

   CPU 5: Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz (4189.9 bogomips)

          Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET

   09:35:29 up 10 days, 23:54,  2 users,  load average: 0.26, 0.10, 0.07; runlevel 2


------------------------------------------------------------------------

Benchmark Run: Sun Jul 31 2016 09:35:29 - 10:03:54

6 CPUs in system; running 1 parallel copy of tests


Dhrystone 2 using register variables           23512033.0 lps   (10.0 s, 7 samples)

Double-Precision Whetstone                              3033.3 MWIPS (9.7 s, 7 samples)

Execl Throughput                                                     1526.6 lps   (29.8 s, 2 samples)

File Copy 1024 bufsize 2000 maxblocks        690641.7 KBps  (30.0 s, 2 samples)

File Copy 256 bufsize 500 maxblocks             204572.2 KBps  (30.0 s, 2 samples)

File Copy 4096 bufsize 8000 maxblocks       1593391.7 KBps  (30.0 s, 2 samples)

Pipe Throughput                                                 1538048.9 lps   (10.0 s, 7 samples)

Pipe-based Context Switching                               30865.1 lps   (10.0 s, 7 samples)

Process Creation                                                         4424.6 lps   (30.0 s, 2 samples)

Shell Scripts (1 concurrent)                                      4324.8 lpm   (60.0 s, 2 samples)

Shell Scripts (8 concurrent)                                       1783.5 lpm   (60.0 s, 2 samples)

System Call Overhead                                           2726923.7 lps   (10.0 s, 7 samples)


System Benchmarks Index Values               BASELINE       RESULT    INDEX

Dhrystone 2 using register variables         116700.0   23512033.0   2014.7

Double-Precision Whetstone                               55.0       3033.3    551.5

Execl Throughput                                                   43.0       1526.6    355.0

File Copy 1024 bufsize 2000 maxblocks        3960.0     690641.7   1744.0

File Copy 256 bufsize 500 maxblocks            1655.0     204572.2   1236.1

File Copy 4096 bufsize 8000 maxblocks        5800.0    1593391.7   2747.2

Pipe Throughput                                             12440.0    1538048.9   1236.4

Pipe-based Context Switching                         4000.0      30865.1     77.2

Process Creation                                                  126.0       4424.6    351.2

Shell Scripts (1 concurrent)                                   42.4       4324.8   1020.0

Shell Scripts (8 concurrent)                                     6.0       1783.5   2972.5

System Call Overhead                                      15000.0    2726923.7   1817.9

                                                                   ========

System Benchmarks Index Score                                         932.3


------------------------------------------------------------------------

Benchmark Run: Sun Jul 31 2016 10:03:54 - 10:32:34

6 CPUs in system; running 6 parallel copies of tests


Dhrystone 2 using register variables          156478170.2 lps   (10.0 s, 7 samples)

Double-Precision Whetstone                             20502.5 MWIPS (9.7 s, 7 samples)

Execl Throughput                                                   15183.4 lps   (29.6 s, 2 samples)

File Copy 1024 bufsize 2000 maxblocks         833217.9 KBps  (30.0 s, 2 samples)

File Copy 256 bufsize 500 maxblocks             239819.3 KBps  (30.0 s, 2 samples)

File Copy 4096 bufsize 8000 maxblocks       2230343.1 KBps  (30.0 s, 2 samples)

Pipe Throughput                                                  9994904.8 lps   (10.0 s, 7 samples)

Pipe-based Context Switching                           1482861.6 lps   (10.0 s, 7 samples)

Process Creation                                                      30072.7 lps   (30.0 s, 2 samples)

Shell Scripts (1 concurrent)                                   20509.8 lpm   (60.0 s, 2 samples)

Shell Scripts (8 concurrent)                                      4632.4 lpm   (60.0 s, 2 samples)

System Call Overhead                                          6365870.6 lps   (10.0 s, 7 samples)


System Benchmarks Index Values               BASELINE       RESULT    INDEX

Dhrystone 2 using register variables         116700.0  156478170.2  13408.6

Double-Precision Whetstone                                55.0      20502.5   3727.7

Execl Throughput                                                    43.0      15183.4   3531.0

File Copy 1024 bufsize 2000 maxblocks          3960.0     833217.9   2104.1

File Copy 256 bufsize 500 maxblocks              1655.0     239819.3   1449.1

File Copy 4096 bufsize 8000 maxblocks          5800.0    2230343.1   3845.4

Pipe Throughput                                               12440.0    9994904.8   8034.5

Pipe-based Context Switching                          4000.0    1482861.6   3707.2

Process Creation                                                   126.0      30072.7   2386.7

Shell Scripts (1 concurrent)                                    42.4      20509.8   4837.2

Shell Scripts (8 concurrent)                                       6.0       4632.4   7720.6

System Call Overhead                                       15000.0    6365870.6   4243.9

                                                                   ========

System Benchmarks Index Score                                        4111.2


Average of ratings: -
In reply to Daniel Tillman

Re: [Moodle BenchMark] Disappointing score (1945) for a two VM setup, 12 CPU and 112 GB RAM in total

by Grant Mucha -

Am I missing something here?

You really need to be using SSD drives to handle the IOPS required by Moodle. You actually have a rather powerful server in terms of CPU and RAM but your putting all that power to waste by not having the proper drive configuration and setup.

Raid 1

This makes no sense given your drive options. You need IOPS to power moodle and have none. I'm looking at your setup an I see a 600HP engine (cpu/ram) connected to transmission (database) that wont shift out of 1st gear.

Reality

You actually need a lot less HP (cpu/ram) and a hell of a lot better transmission (storage).

I will cut to the chase and give you an idea what I mean.

Trash the 7200 drives (Sell or reuse them for internal backups)

Purchase 2x SSD drives

https://www.amazon.com/DISCONTINUED-Samsung-850-PRO-MZ-7KE128BW/dp/B00LMXBOP4

256x2 @ $244 USD (About 512GB minus overhead)

512x2 @ $440 USD (About 1024GB minus overhead)

Whatever suits your size requirements.

Configure the above in RAID0 for performance. I would recommend more drives but to limit cost and get the performance your after RAID0 will suffice provided you have backups in place. If you have additional capital you could take it further and move to 4x drives and RAID 10 and possibly a HW Raid Controller. Or possibly another avenue is something like Samsung NVMe 950 Pro.

"850 PRO drives blew away all of the other drives where it posted 12,837/10,785IOPS, 378/314.87MB/s and an average latency of 0.597/0.711ms for the 1TB and 128GB, respectively."

You indicated being handcuffed by VMware 10 CPU limit. Trash VMware! This entire setup would perform better without running VWware. At least given the hardware you've listed.

Setup.

I would revisit the Ubuntu 16.04 and PHP7 setup. I've upgraded numerous 2.7 and 2.8 installations to 3.1+ on the above without any issue. However, I never use the GUI on a server. I'm some cases I also use HHVM so that might be worth looking into.

Ideally

2x Intel E5-2620v2 (6 cores + hyperthreading on each)

128 Gigs of Ram total

2x 256GB Samsung 850 Pro drives (Raid 0) (datastore 1/2)

Option 1

2x 7,200 RPM SAS 1tb drives (Raid 1) - internal backup

2x 12,00 RPM SAS 1.8tb drives (Raid 1) - everything other than datastore

Option 2

4x SAS drives (Raid 10) - Use for everything other than datastore

*This would be an experiment due to the mix-matching of hardware*

Option 3

Scrape all of your existing storage and rebuild using SSD or NVMe in RAID0 or RAID10 and with proper internal backup / offsite backup (Amazon S3).

Notes

If you are unfamiliar with VMware avoid it unless absolutely necessary. It has its own bottlenecks an quirks that will affect your overall performance.

If you ever upgrade your storage infrastructure I would love to see your performance stats and feedback.


Cheers,

Grant





In reply to Grant Mucha

Re: [Moodle BenchMark] Disappointing score (1945) for a two VM setup, 12 CPU and 112 GB RAM in total

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Daniel must have solved the problem in this sub-thread "Re: [Moodle BenchMark] Need Beta-tester" https://moodle.org/mod/forum/discuss.php?d=335357#p1359462, or at least didn't return after 9 August.
In reply to Visvanath Ratnaweera

Re: [Moodle BenchMark] Disappointing score (1945) for a two VM setup, 12 CPU and 112 GB RAM in total

by Grant Mucha -

You very well may be correct!

But if he does happen to come back, perhaps he will share with us how he solved the problem. wink

In reply to Daniel Tillman

Re: [Moodle BenchMark] Disappointing score (1945) for a two VM setup, 12 CPU and 112 GB RAM in total

by Grant Mucha -

Here is a perfect example

I just did a migration of Moodle 2.7 to Moodle 3.1 and also moved hardware. Notice that the hardware is far less powerful than your own, however the storage is setup in a more effective manner for Moodle.

System Benchmark - 74/75 after repeat tests

Benchmark Score : 74 points


#DescriptionTime in secondsAcceptable limitCritical limit
1Moodle loading time
Run the configuration file «config.php»
0.0050.50.8
2Function called many times
A function is called in a loop to test processor speed
0.1380.50.8
3Reading files
Test the read speed in Moodle's temporary folder
0.0110.50.8
4Creating files
Test the write speed in Moodle's temporary folder
0.02211.25
5Reading course
Test the read speed to read a course
0.1560.751
6Writing course
Test the database speed to write a course
0.01711.25
7Complex request (n°1)
Test the database speed to execute a complex request
0.0840.50.7
8Complex request (n°2)
Test the database speed to execute a complex request
0.1010.30.5

Hardware

CPU - i7-980x (6c/12t)

RAM - 24GB

SDD - 2x 128GB Samsung 850 Pro - Raid 0

HDD - 2x 1.5TB in Raid 1

Amazon S3 offsite storage

System

Ubuntu 16.04

Nginx 1.10

PHP 7.0

MySQL 5.7.13

Ram Disk (4GB)


In reply to Daniel Tillman

Re: [Moodle BenchMark] Disappointing score (1945) for a two VM setup, 12 CPU and 112 GB RAM in total

by Grant Mucha -

Also here is the UnixBench for the system mentioned above.

System Benchmarks Index Score                                        7550.4