Same application,different hardware,No change in performance

Same application,different hardware,No change in performance

by shrikant gakhare -
Number of replies: 25

Please share your observations.

I have moodle on 7 year old server. Users started complaining about performance. Page load very slowly. Take more than a minute or longer. When concurrency goes up, users get frustrated.

I bought very new server, 4 core processor, 32 GB RAM etc. This server is far ahead of old server. I migrated moodle as it is to new server.

In spite of such a robust an best configuration, there is no change in performace, even for single user who is testing the performance. The fact is old server is very fast for one user, even on web. Strange.

Let me know if u want more data.

Experience people please share what could the cause of such pathetic performance and likely solution for same.

regards

Shri

 

Average of ratings: -
In reply to shrikant gakhare

Re: Same application,different hardware,No change in performance

by Ken Task -
Picture of Particularly helpful Moodlers

A little more information might be needed for anyone to be able to give more specific information, I would think.  '7 year old server'.  Does that also mean one has '7 year old' version of Moodle? ;)

Hardware is one thing ... from what you described ... looks like a farily good start, but ... is the OS (Winders, MacOSX, Linux [distro]) of the new server 64 bit?  Run any MySQL tuner to optimize DB performance?  Leave the GUI desktop running with root user logged on running all kinds of 'cool' applets?  How many apache servers are launched on startup?  How many held in reserve?  Has max connections for MySQL been tweaked? etc., etc., etc.

How about the Moodle itself?  *Version?  How many add-ons?  Allowing theme change for every category/course?  Are users allowed to change theme?  What seems to be the most used activities?  What is primary method of authentication?  LDAP?

As you can see, have more questions than answers!

'spirit of sharing', Ken

Average of ratings: Useful (3)
In reply to Ken Task

Re: Same application,different hardware,No change in performance

by shrikant gakhare -

Thanks Ken!

Follows answers to your questions. (To reiterate, we have not changed anything when existing application was migrated. The (old) live MOODLE loads faster than new. New MOODLE is not open for users.We are not using anything than default settings given by MOODLE)

Does that also mean one has '7 year old' version of Moodle? ;)

Answer- MOODLE V 1.9.4 , We customised it over last 3-4 years hence migrated as it is to new server. (As Viswanath Sir suggested, one change at a time)

is the OS (Winders, MacOSX, Linux [distro]) of the new server 64 bit? 

Answer- OS is 64 bit

Run any MySQL tuner to optimize DB performance? 

Following is the result of MySQL tuner but we have not acted on its recommendation.

General Statistics --------------------------------------------------

[--] Skipped version check for MySQLTuner script

[OK] Currently running supported MySQL version 5.1.37-log

[OK] Operating on 64-bit architecture

 

-------- Storage Engine Statistics -------------------------------------------

[--] Status: +Archive -BDB -Federated +InnoDB -ISAM -NDBCluster

[--] Data in MyISAM tables: 2G (Tables: 241)

[!!] InnoDB is enabled but isn't being used

[OK] Total fragmented tables: 0

 

-------- Performance Metrics -------------------------------------------------

[--] Up for: 2d 22h 45m 12s (3K q [0.013 qps], 171 conn, TX: 6M, RX: 526K)

[--] Reads / Writes: 83% / 17%

[--] Total buffers: 442.0M global + 12.3M per thread (151 max threads)

[OK] Maximum possible memory usage: 2.2G (7% of installed RAM)

[OK] Slow queries: 0% (2/3K)

[OK] Highest usage of available connections: 2% (4/151)

[!!] Key buffer size / total MyISAM indexes: 384.0M/2.2G

[!!] Key buffer hit rate: 93.7% (72M cached / 4M reads)

[OK] Query cache efficiency: 79.5% (1K cached / 2K selects)

[OK] Query cache prunes per day: 0

[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 139 sorts)

[OK] Temporary tables created on disk: 9% (19 on disk / 199 total)

[OK] Thread cache hit rate: 97% (4 created / 171 connections)

[!!] Table cache hit rate: 9% (50 open / 548 opened)

[OK] Open file limit used: 8% (97/1K)

[OK] Table locks acquired immediately: 99% (1K immediate / 1K locks)

[!!] Connections aborted: 25%

 

-------- Recommendations -----------------------------------------------------

General recommendations:

    Add skip-innodb to MySQL configuration to disable InnoDB

    Enable the slow query log to troubleshoot bad queries

    Increase table_cache gradually to avoid file descriptor limits

    Your applications are not closing MySQL connections properly

Variables to adjust:

    key_buffer_size (> 2.2G)

    table_cache (> 512)

 

How many apache servers are launched on startup?  How many held in reserve?

Answer- Only one launched

  Has max connections for MySQL been tweaked? etc., etc., etc.

Answer- No

How about the Moodle itself?  *Version?  How many add-ons?  Allowing theme change for every category/course?  Are users allowed to change theme?  What seems to be the most used activities?  What is primary method of authentication?  LDAP?

Answer- V 1.9.4, No theme change for category/course, Users are not allowed to change theme, Add-on – None

Authentication is thru mail to official mail id.

Most used activity

  1. Downloading content (pdf) on to user machine for reading
  2. Assessment /Quiz module

Let me know you want additional info.

Thanks in advance!

Shrikant

Average of ratings: -
In reply to shrikant gakhare

Re: Same application,different hardware,No change in performance

by Ken Task -
Picture of Particularly helpful Moodlers

Thanks for the fully answered response!

The old site is still up and running under fully qualified domain name, right?  The new server Moodle is currently configured (in config.php) with an IP or has there been any changes to the config.php to reflect new server?  Does the front page render slowly or barely at all - all mixed up?

If new server with IP (not a FQDN) then there references in the DB that still point to the real server.  Images, for example, used the config.php variables for the site URL to build img src links.  That would/might cause slow performance.

Something one can do ... install FireBug on a FireFox workstation browser.  It will help ID issues.

Now some points …
Take it you've done some network testing ... the NIC ... switch ports, etc..  Network does come before application.

You've not mentioned which Linux Distro.  So:
uname -an as root user.

How much memory does new server have compared to old?
as root user: top
will show running processes, etc..
At the top of top you'll see memory related stats in real time.

Apache doesn't launch just one.  Check the httpd.conf file for:
# prefork MPM
# worker MPM

ps aux |grep httpd
will show apache processes specifically.

/usr/sbin/httpd -V
will show how apache compiled …

php -v (version of php)

Typically, one will have to tweak php settings/values.  While some below not really required for 1.9.x, they will come into play when moving to 2.x.
May as well do them now.  The following values for a server I run which may/may not be optimal for your.

max_input_vars = 5000
max_execution_time = 120
max_input_time = 150
post_max_size = 210M
upload_max_filesize = 210M
date.timezone = 'your_time_zone'

Tweaks to MySQL as well:
max_allowed_packet=500M
open_files_limit=6000

Any changes to apache or mysqld does require restart of those services and it would be a good idea to check logs after re-starting.

*** And a checking of apache logs - specifically the error_log.

1.9.4 is really old.  Matter of fact, nearing end of life for 1.9.x so before moving to 2 might be a good idea to upgrade to the highest 1.9 (which is 1.9.19 at present)

With version 1.9.6 password salting was introduced.  So the upgrade to 1.9.19 is for a purpose.  Getting password salting worked out first will make the transition to 2 much easier.

Also, you've said customized … what customizations? (basically)

In any/all of the above, one might want to consider exposing too much.
Edit responses not to disclose too much.

'spirit of sharing', Ken

Average of ratings: Useful (5)
In reply to Ken Task

Re: Same application,different hardware,No change in performance

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

Just an addendum to Ken's post.

Since the OP specified he is running Solaris 11 (Nevada) things are a little different. I've tried to block as much of Solaris from my mind as possible, but a few things keep coming back ;)

If I recall correctly, you can't use ps auxw in Sollaris - you'll need ps -ef making the correct command:

ps -ef | grep httpd

Memory allocation in Solaris is also fundementally different to Linux - so for starters there is no sysctl to increase shmmax - unfortunately, I do seem to have forgotten the Solaris way for that.

The new server has 32GB RAM, and is running DB, Apache, and files by the look of it. It's difficult to put a finger in the air and say "Use XMB of RAM for MySQL" because it will depend entirely on workload, the types of queries, etc. That said, it's definately worth increasing the amount of RAM available to MySQL (key_buffer_size) - start by following the advice of the tuning guide you posted output from, though you can probably go for more. I assume that 2.2GB isn't the size of your production database..?

Looking at the perf data, it looks as though MySQL isn't running slowly on the whole (2/3000 connections) but 3000 connections is a tiny number to work with.

The 25% of aborted connections are more concerning... it's probably worth increasing the verbosity of MySQL logs to look at why that might be.

Although Moodle 1.9 is now out of support, it may be worth just sticking where you are whilst you work out these issues. Moodle 2.X is much more resource intensive, and it's probably best to run one change at a time. Once you've got your performance issues nailed though, you really should upgrade. As Ken mentioned, 1.9.6 introduced new password salting and there have been a number of security vulnerabilities discovered since 1.9.4 which are now public.

Ken also suggested checking the apache logs. It's also worth turning on PERFTOLOG (see config-dist.php for this setting). This will log a load of performance data which you can then analyse to help work out where things are going slowly. There's also a couple of tools to analyse these - see http://cvs.moodle.org/contrib/tools/devtools/perf_parse.pl?view=log and http://cvs.moodle.org/contrib/tools/devtools/perf_parse.rb?view=log for starters. It's really worth checking this out, though it's not always clear how this data is useful immediately.

Also, I repeat again, install a PHP accelerator. It really is worth doing. I cannot stress this enough. I don't know which is best suited to Solaris. We used to run eAccelerator on our Debian sparc boxes, but that was a number of years ago.

Andrew

Average of ratings: Useful (2)
In reply to Andrew Lyons

Re: Same application,different hardware,No change in performance

by Mathew Gancarz -
Picture of Core developers

In terms of an accelerator, we're looking at loading it in our environment and we also run on Solaris. For the OP, you just need to go to the PHP bin directory and run the command "sudo pecl install apc" and follow the prompts. Default values are fine. Then you'll just need to configure it in php.ini with the appropriate settings.

Average of ratings: Useful (1)
In reply to Mathew Gancarz

Re: Same application,different hardware,No change in performance

by shrikant gakhare -

We installed APC acceleraror with some improvement. DB size is 5 gb.

Average of ratings: -
In reply to Andrew Lyons

Re: Same application,different hardware,No change in performance

by shrikant gakhare -

Thanks Andrew!

We installed APC Accelerator.Observed some improvement but as users grow as day progresses, site slows down.

For DB access, settings had url https://domain.com, which we changed to "localhost" and response time improved dramatically. But when we tested it for quizzes/assessment, the questions were missing for some courses and for some quesions were displayed and after attempting quiz marks were not shown. strange. what do you say about this ?

WOuld you like to see settings in configuration files? Shall we reduce db size?

We are looking at apache and mysql logs.

Thanks!

 

 

Average of ratings: -
In reply to Ken Task

Re: Same application,different hardware,No change in performance

by shrikant gakhare -

Thanks Ken !

We will as per your advise. Follows information you asked for.

The old site is still up and running under fully qualified domain name, right?

A-     Yes, old site is up and running under FQDN. What can you suggest to improve its performance. We do not have PHP accelerator on this machine. Configuration is given below. You will be amazed that it works much better than new one.

System Configuration:  Sun Microsystems  

System Model : Sun Fire 480R

System clock frequency: 150 MHz

Serial No:0539AM0057

Warranty Expiry:31/07/2010

Release: 5.10

Kernel architecture: sun4u

Application architecture: sparc

Hardware provider: Sun_Microsystems

Kernel version: SunOS 5.10 Generic_139555-08

Memory size: 8192 Megabytes

System :SunOS 10v

Harddisk Size :73.40GB*2

OEM# = 0

Origin# = 1

NumCPU = 2

 

 The new server Moodle is currently configured (in config.php) with an IP or has there been any changes to the config.php to reflect new server?  Does the front page render slowly or barely at all - all mixed up?

A-     Yes, we made necessary changes in config.php as required for new server with IP. New installationis configured with IP only. Front page render slowly. Look at the configuration of new server given below. It is 5-6 times better than old one. We expected lightening performance from new server.

Part Description

Qty

    

Netra SPARC T4-1 server: model family

1

    

Netra SPARC T4-1 server: 2 RU AC base chassis with 1 SPARC T4 4-core processor 2.85GHz

1

Power cord: India, 2.5 meters, IS1293 plug, C13 connector, 10 A (for factory installation)

2

Two 4 GB DDR3-1066 registered DIMMs (for factory installation)

4

One 300 GB 10000 rpm 2.5-inch SAS-2 HDD with mounting bracket (for factory installation)

4

Sun Storage 6 Gb SAS PCIe RAID HBA, Internal: 8 port and 512 MB cache (for factory installation)

1

Netra X4250 SAS Cable Kit

1

Oracle Solaris 11, Oracle VM Server for SPARC, and Oracle Electronic Prognostics preinstall (for factory installation)

1

19-inch 2-post rackmount kit (for factory installation)

1

Hardware Freight Fee

1

 

If new server with IP (not a FQDN) then there references in the DB that still point to the real server. 

A-     References in the DB in new server point to the new server where DB resides. ( No doubt about it. Because page loads but takes more than a minute for single user)

Images, for example, used the config.php variables for the site URL to build img src links.  That would/might cause slow performance.

A-     No 

Take it you've done some network testing ... the NIC ... switch ports, etc..  Network does come before application.

 A- network is fine

You've not mentioned which Linux Distro. 

A-     Solaris 11 (unix)

How much memory does new server have compared to old?

A-     Old -8 GB New- 32 Gb. 

(What apache , MySQL, PHP settings you suggest for optimum utilization and performance)

Also, you've said customized … what customizations? (basically)

A-     Very little. For example, 90 days duration for completion of course,  feedback from students, de-enrollment after 90 days from courses etc. Not much

Average of ratings: -
In reply to shrikant gakhare

Re: Same application,different hardware,No change in performance

by shrikant gakhare -

what is the possibility of missing some file during migration? in that case site would not have worked. Is hardware playing any role in slowing it? We checked statistics, CPU is 99.9 % idle. Ample RAM (around 27 GB is free, we were doing some data selection, just to see how it performs).

What settings you suggest for MySQL based on configuration given above?Do you think apache, config.php should be configured in certain way.? 

(Just doing loud thinking)

Average of ratings: -
In reply to shrikant gakhare

Re: Same application,different hardware,No change in performance

by Ken Task -
Picture of Particularly helpful Moodlers

Could we see a screen shot of the new server front page?

Think we all now know the specs of the box, so let's drop re-posting that information, please.

In all versions of Moodle (that I can remember), there are config lines such as the following (the following from a RedHat flavored machine):

$CFG->wwwroot   = 'http://sos.tcea.org/kmm19';
$CFG->dirroot   = '/var/www/html/kmm19';
$CFG->dataroot  = '/var/www/kmm19data';

(yes, that's a real server)

Notice there are NO trailing slashes.

In the config file of the new server, you've the IP address of the new server.  **BUT, when the database was transferred to the new machine did you edit any references in the sql dump of the current server and change references to http://yourserverbyname/moodle/ to http://newip/moodle/
before importing the DB?

It is necessary to do that.  Moodle uses the wwwroot variable to build links to some things like images and documents (resources).
Wonder what the apache logs on the old server look like?  Are there references in either access or error that show request coming from the IP address of the new server?  Asking for a file/image?

The config lines above come from a copy of a moodle 1.9.x whose original URL is really http://sos.tcea.org/kensmoodle/.  I haven't cleaned up the db for the kmm19 site yet.  There are links to images in the DB for the kmm19 site that still point to the original site.  If I run this query on the database for the kmm19 site:

select * from `mdl_resource` where `alltext` like "%http://sos.tcea.org/kensmoodle%"

I get hits that refer to links/images that really reside in the kensmoodle site.  Thus, some images/links to documents in the kmm19 site are still coming from the kensmoodle site.

If you go to the http://sos.tcea.org/kmm19 site you'll see a 'Wizard' image in an HTML block.  Here's the true image location (spaces added to prevent this forum from showing the image).

http : / / moodle.tcea.org / kensmoodle /file.php/1/fx/wizard_raisehands_sm_wht.gif

That image isn't in /1/fx/ of the kmm19 site yet it displays because the two servers are on the same network and there is nothing blocking communication between them.  In your case, your new server might be using private IP address for some things.  In your case, the slow rendering of images/front page, etc. might be due to the same situation.

Am not saying that this is the only thing that is causing slowness but it could be a factor ... and coupled with other issues in the transfer of the site, they add up to make it very noticiable.

'spirit of sharing', Ken

 

 

Average of ratings: Useful (2)
In reply to Ken Task

Re: Same application,different hardware,No change in performance

by shrikant gakhare -

Dear Ken,

We ran replace.php and it resolved the link issue. This step was taken just after migration as given on moodle.org for site migration.

In fact, on suggestion from one of our friend we replaced ip address with "localhost" for db access and pages were loading fast. But quiz questions were missing or marks were not displayed after attempting quiz.

Surprisingly other additional pages which we designed were loading slowly.

Seeing that new server is responding fast, we added "localhost" to our live server but we did not witness performance improvement.

Thus we concluded this is not the "necessary"  settings we needed.

Here is the screenshot of front page

 

 

 

Average of ratings: -
In reply to shrikant gakhare

Re: Same application,different hardware,No change in performance

by Robert Brenstein -
If you were running your old service for long time and the performance degraded over time (as opposed fro one day to another), one thing to check is whether some of the tables, like log, did not get exceedingly large or corrupted. Presumably, you cloned the database to the new server, so you could have carried over such problems.
Average of ratings: Useful (1)
In reply to Robert Brenstein

Re: Same application,different hardware,No change in performance

by shrikant gakhare -

Thanks Robert!

That is a possibility. Corrupted tables were repaired. Yes, one table has 20 million rows. I can share top ten tables in moodle.

In fact with same data, production server shows excellent response as comapred to new one (under testing). Pls look at the hardware conf in thia forum for 2 servers.

 

Average of ratings: -
In reply to shrikant gakhare

Re: Same application,different hardware,No change in performance

by Robert Brenstein -
Further thing to check is whether indexing is on where needed. I recall reports of admins activating or reactivating indexing with corresponding jumps in performance.
Average of ratings: -
In reply to Robert Brenstein

Re: Same application,different hardware,No change in performance

by shrikant gakhare -
Name Engine Version Row_format Rows
mdl_question_states MyISAM 10  Dynamic 22593197 
mdl_question_sessions MyISAM 10  Dynamic 9201757 
mdl_feedback_value MyISAM 10  Dynamic 2058341 
mdl_grade_grades_history MyISAM 10  Dynamic 712034 
mdl_question_attempts MyISAM 10  Dynamic 481762 
mdl_grade_grades MyISAM 10  Dynamic 325348 
mdl_course_display MyISAM 10  Fixed 266105 
mdl_quiz_grades MyISAM 10  Fixed 252260 
mdl_quiz_attempts MyISAM 10  Dynamic 252552 
mdl_user_lastaccess MyISAM 10  Fixed 247669 
mdl_question_answers MyISAM 10  Dynamic 185889 
mdl_user MyISAM 10  Dynamic 103630 

 These are top 13 tables in DB.

Average of ratings: -
In reply to shrikant gakhare

Re: Same application,different hardware,No change in performance

by Ken Task -
Picture of Particularly helpful Moodlers

Are all your tables MyISAM?  Isn't MyISAM slower than InnoDB for tables that are frequently being inserted to or updated.  Do know that before moving to 2 and if migrating on-site (ie, in place) engine should be InnoDB.  That change will/should require re-running of sql tuner and tweaking values.

'spirit of sharing', Ken

Average of ratings: Useful (1)
In reply to shrikant gakhare

Re: Same application,different hardware,No change in performance

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
It shows that, you haven't identified the bottle neck! You can throw resources anywhere, unless you hit the bottle neck there won't be any effect.
Average of ratings: Useful (2)
In reply to Visvanath Ratnaweera

Re: Same application,different hardware,No change in performance

by shrikant gakhare -

Dear Viswanath/Ken/Andrew

We are going to have web,app and db on three different boxes. 

1. Webservers -2 (apache)

2. App server -1

3. Database server -1(mysql)

How is the configuration here is different than having all three in one?

rgds

shrikant

Average of ratings: -
In reply to shrikant gakhare

Re: Same application,different hardware,No change in performance

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

Hi,

You've already asked exactly this question in https://moodle.org/mod/forum/discuss.php?d=218732.

Rather than wasting peoples time in reading your question twice, it would be best if you only ask it once and give the correct information in the original thread.

Andrew

Average of ratings: Useful (1)
In reply to Andrew Lyons

Re: Same application,different hardware,No change in performance

by shrikant gakhare -

Aplogies Andrew!

How do I delete other threads? I'll try to delete them so that all questions come here.

This server has RAID. Any clues???

Average of ratings: -
In reply to shrikant gakhare

Re: Same application,different hardware,No change in performance

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
> How do I delete other threads?

You can't. You can try sending messages to the moderator of each forum. Their profiles are usually linked to the introduction to that forum.

Or write to Helen as in http://docs.moodle.org/en/Moodle.org_FAQ#How_do_I_ask_for_my_account_on_moodle.org_to_be_deleted.3F.
Average of ratings: Useful (2)
In reply to Visvanath Ratnaweera

Re: Same application,different hardware,No change in performance

by shrikant gakhare -

Thanks All,

Issue resolved and this thread is closed.

regards

Shrikant

Average of ratings: -