Performance woes!

Performance woes!

by Janne Mikkonen -
Number of replies: 33
There has been increasing amount of questions "what kind of settings I need to support x amount of simultanious users?".
Well, WITHOUT clustering the "x" is about 50 or 60 users. Although you can give some CPR to your settings and improve
installation methods and it'll give you aprox 10 more extra user connections:

1. Use *nix OS.
2. Build Apache, MySQL and PHP from source ( you can get some extra peformance out adding proper CFLAGS to gcc )
3. Tweak Apache settings ( although there are some tradeoffs with dynamic pages ).
3. Tweak MySQL settings ( query_cache, query_cache_limit, key_buffer, sort_buffer_size, thread_cache_size,
max_connections ,table_cache ,tmp_table_size, read_buffer_size, read_rnd_buffer_size, myisam_sort_buffer_size, thread_concurrency )
4. Tweak your OS's kernel shared memory settings
5. Tweak your OS's TCP/IP stack settings
6. Tweak your OS's filesystem settings.
7. Buy better processor(s)
8. Buy more memory
9. Use SCSI harddrive(s)
10. Upgrade to Moodle 1.9.x

Ok! Been there, done that. But performance still sucks!!! And I dare to say that one of the biggest
performance killer is ADOdb -library! Let's explore that:

I have a installation where I have category listing in frontpage ( 28 categories and only few course in each one ).
Lets make a script that fetches that category information ( simple $categories = get_records("course_categories") ).
Run the query 1000 and 10000 times and record how long did it take ( mine was 1000 = 2.142, and 10000 = 21.413 ).
Now lets simulate same query but using PHP's native MySQL API ( query is SELECT * FROM mdl_course_categories ORDER BY sortorder ASC )
Run the query again 1000 and 10000 times and record how long did it take ( mine was 1000 = 0.363, and 10000 = 3.631 ).

Run the tests several times that you'll have proper readings.

Moodle 1.6.x and Moodle 1.9.x had the same timing values with internal query cache off
(there were only fraction of a second improvement when I turned internal cache on).

I think it would be wise to write our own multi-db-layer code to replace ADOdb or use some other
already made multi-db-layer ( and yes it'll create again those time and money issues ).

PLEASE DO PROVE ME WRONG HERE! I'd be clad to hear that everything's dandy and a OK!

- Janne -

Average of ratings: Useful (2)
In reply to Janne Mikkonen

Re: Performance woes!

by Eric Merrill -
Picture of Core developers Picture of Moodle HQ Picture of Peer reviewers Picture of Plugin developers Picture of Testers
While there may be a big performance hit there, I think you left out a HUGE place to gain performance... a PHP compiler cache. We use xCache, and have seen a 2-3x boost in performance (page load times), and about a 6x decrease in server loads.

I would do these same benchmarks again with a compiler cache in place and see. We run a system with 200-400 concurrent users w/o much problem (even with the db and apache on the same box).

-eric
In reply to Eric Merrill

Re: Performance woes!

by Janne Mikkonen -
Well, these test are run with APC -cache, compiled with pthread mutex locking type instead of file locking. Can you put some facts on the table? How long your users have to wait if you'll get 200 - 400 concurrent request?

- Janne -
In reply to Eric Merrill

Re: Performance woes!

by Samuli Karevaara -
Eric, are you running 1.8? And are those 200-400 simultaneously running httpd-threads/processes/insertrightterm?

What kind of a box, a quad processor w/ 16 gigs?
In reply to Samuli Karevaara

Re: Performance woes!

by Eric Merrill -
Picture of Core developers Picture of Moodle HQ Picture of Peer reviewers Picture of Plugin developers Picture of Testers
We saw serious performance problems with 1.8, but that was also in part due to a huge load increase we saw at the same time as our version up, so right now we are back at 1.5.3 (which of course changes a bunch of this). We saw basically all of our performance problems were in the roles/context system.

I guess I mainly should qualify that number. I guess it would not be 400 concurent users, we did some numbers based on pages/sec. I have had ~150-200 concurrent httpd processes at once though.

Right now we are doing major load testing of 1.9, and it looks much better (the Moodle gang did a great job with their new performance improvements).

Out box I believe is a 4 way, with 8GB.

-eric
Average of ratings: Useful (1)
In reply to Janne Mikkonen

Re: Performance woes!

by Martín Langhoff -

Hi Janne,

this is discussed more in depth in the Servers and Performance forum. My recommendations are, in order of impact:

  • Yes! Use 1.9 as soon as you can (note that it is in still in beta these days)
  • Use a recent linux distro, targetted to servers, with a recent kernel - such as 2.6.16 or newer. Debian or the server edition of Ubuntu ideally.
  • Disk IO is your main bottleneck, specially for the DB work. Several pairs of SCSI disks setup in RAID-1. Separate the workloads of the RAID-1 pairs - this is a well known practice for DBAs -- what you want are dedicated spindles for each task
  • Lots of memory on the DB server, and tell the DB to use all that memory for caches /& sorting
  • Lots of memory on the webserver, and maxclients set to match
  • Ensure the lowest latency possible between DB and webserver if they are split up
  • eAccelerator or similar on the webserver
  • noatime and other FS tweaks on the webserver
  • Appropriate FS tweaks for your DB
  • 64bit OS & Opteron CPUs on the DB server
  • 32-bit and cheap CPUs on the webserver
  • Run sysstat to collect system load logs -- it is by far the best stat collection package and will capture important stuff noone else does

Things to avoid:

  • 64-bits on the webserver. Unstable, eats lots more memory.
  • MySQL Cluster
  • Windows/IIS
  • SANs
  • Running inside a virtualisation guest
  • Don't build by hand - we spend 99% of the time waiting for the DB, making the 1% slightly faster won't change anything - it is not worth it mixed

HTH

Average of ratings: Useful (2)
In reply to Martín Langhoff

Re: Performance woes!

by Janne Mikkonen -
This exactly why this isn't in servers and performance forum. I'm not talking about tweaking. Yes, we can tweak whole lot of things to improve performance, but how long will it carry out?
In reply to Martín Langhoff

Re: Performance woes!

by Samuli Karevaara -
Have you benchmarked the difference with ADOdb and a bypass-operation-hacked native pgsql drivers? It would not seem to be a lot of code just to test with one decicated DB type, just hack the query execution functions, no need to touch the DDL stuff.
In reply to Samuli Karevaara

Re: Performance woes!

by Janne Mikkonen -
Not as good as with MySQL but still about 60 % faster than with ADOdb ( on my home-server). I've got a graveyard shift again and when I'm done with that I'll be able to test it on my real (at this point testing ) server.
In reply to Martín Langhoff

Re: Performance woes!

by sam marshall -
Picture of Core developers Picture of Peer reviewers Picture of Plugin developers
Other Moodle config tweaks to consider:

1) Turning internal record cache on (set to e.g. 100 records) makes a BIG difference on some inefficient requests. [Maybe less difference in 1.9? We are still on 1.8 for our live system.]

2) Depending on your filter settings, it may be a good idea to turn off text filter caching. CPU = cheap, even in PHP assuming you have some kind of accelerator running, while DB = expensive - so just watch the number of queries on the pages you care most about. Sometimes turning caching on makes it do more queries.

I don't think 'concurrent users' is very useful term for benchmarking, how about hits or better still, pages, per second? I don't have those stats for OU right now though... While OU has a large number of users (400,000 entries in user table) and a fairly large number of active users (85,000 of those users accessed system at least once in past month), most of our course websites are still quite static. For example most do not have forum, because that's still provided by another system. So probably those user accesses don't correspond to very much load and we could probably handle many 'concurrent users' (if all they're doing is downloading a PDF file every 10 minutes) compared to a more active site where users are making many-db-request forum accesses every few seconds.

One other note is that though a fast DB server is important (ours has 16GB RAM, 64-bit, etc), it may actually be the web server that's the limitation. Apparently when we have had performance issues in the past, the maxed-out load has been the web servers, not the DB server. That's cool because you can easily chuck in another load-balanced web server, whereas getting multiple DB servers is harder/slower.

--sam
Average of ratings: Useful (1)
In reply to sam marshall

Re: Performance woes!

by Janne Mikkonen -
"That's cool because you can easily chuck in another load-balanced web server", I totally agree. As a icehockey fan I do like some brute-force sometimes. But this leads us to the original question. Are we happy with it? Just let the snow cover everything we don't wanna see! And then suddenly it's spring again and snow will melt away...

Ok, let's piss off every female forum users. Take one male and one female. Give them a task, which is buy 1 litre of milk. Which one will perform this task quickest? I want my milk fast as I can ( and cold ) wink

- Janne -
In reply to Janne Mikkonen

Re: Performance woes!

by Michael Penney -
I think vehicles are a good metaphor:

If we want 1 litre of milk the fastest, we want a Porsche (or maybe if we are Californian, a Teslasmile.

But if we want 1000 litre's of milk at the same time, we don't want to use the Tesla--then maybe we want to use the Freightliner or the Mack.

By the way, for really cold fast milk, perhaps you'd want a refrigerated truck...or a coolstack: Moodle on SAMPsmile.
Average of ratings: Useful (1)
In reply to Martín Langhoff

Re: Performance woes!

by Penny Leach -

Martin - you wrote:


Things to avoid:

[snip]

  • SANs
[snip]

Why? Surely SANs are usually fast fast?
In reply to Penny Leach

Re: Performance woes!

by Genner Cerna -
we had the same problem with the performance of moodle 1.8, for now we constant monitoring the server to minimize downtime.

dont get me wrong moodle is great but performance is a problem fro now...
In reply to Penny Leach

Re: Performance woes!

by Samuli Karevaara -
We have the data folder on SAN and the db on a local disk. This setup seems just fine.
In reply to Penny Leach

Re: Performance woes!

by Martín Langhoff -

SANs are sold as fast, but unfortunately they rarely are.

There are frequent discussions about performance vs cost of "locally attached disks" vs SANs in the Pg mailing list. They often get religious, but when people actually test comparable HW with different settings and post numbers, the conclusion tends to be that in the low-mid cost range, local storage across several disks wins with a bit of relatively simple tuning (and a linux kernel wink ). In other words, the low and mid tier SANs are slower, and no-one knows how to tune them.

Still, the high end winners are usually very expensive SANs with a lot of tuning. So perhaps I should rephrase it to"stay away from SANs unless they cost more than a jet" wink

And the bit that I don't like about SANs is that they usually come with a proprietary closed binary driver for Linux that only works on a few kernels. So you are stuck with RHEL version X, and unless they ship newer drivers, you could be stuck on that for years. On the client side. Performance improvements on linux kernels have been coming pretty fast, so being stuck on a really old kernel probably negates any speed adv you get from the ssssuper-fast SAN that costs a million.

So the short version still is: stay away from them. Not worth the time and money -- mostly a marketing ploy.

PHBs love them, just as they love mauve databases.

(Note! A well tuned SAN using fibre is way better than NFS. So moodledata on a SAN is a great thing. OTOH, having the root of your OS or pgdata on a SAN is a dicey proposition. Sadly, SAN marketers say you should put everything on the SAN. Use common sense, and whatever you do - remember that at the end of the day, number of spindles and resource contention is where it's at...)

Average of ratings: Useful (1)
In reply to Martín Langhoff

Re: Performance woes!

by Séverin Terrier -
Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Testers Picture of Translators
Thanks Martin for this very usefull message smile

It's simple, and allows to have a rapid and good view to what must be done, or avoid.

Just to be sure, when you say "avoid Don't build by hand", you mean that we have to compile the applications (mysql, php, apache), and not use precompiled ones ?

Séverin
In reply to Séverin Terrier

Re: Performance woes!

by Martín Langhoff -

Apologies if it is not clear... Use the precompiled & packaged versions unless there is a problem with those.

In reply to Martín Langhoff

회신: Re: Performance woes!

by Jong-Dae Park -

Hi! Martin Langhoff,

You mentioned that "running inside a virtualization guest" should be avoided. I found that there are several virtualization schemes and I wonder all the virtualization schemes are not good for moodle. What do you think about using Xen with CentOS?

Thanks in advance.

Jong-Dae Park

In reply to Jong-Dae Park

Re: 회신: Re: Performance woes!

by Martín Langhoff -

No virtualisation scheme is good for Moodle. They may be good for security, and they usually make sysadmins happy. But they don't help Moodle at all.

Xen is specially bad performance wise. At Catalyst we think that the least evil wink is "vserver" (a Solaris Zones lookalike that works on Linux). Still, performance under load won't be good with it -- so we use it where we care more about compartmentalisation than about performance.

In reply to Martín Langhoff

회신: Re: 회신: Re: Performance woes!

by Jong-Dae Park -

Hi! Martin Langhoff,

Thanks for your reply.

In these days, Intel embedded virtualization technology into their multi-core cpus. In certain area such as moodle hosting, it may be necessary to use virtualization technology for easy administration as Moodlerooms does.

I wonder what is the best way to manage the moodle server failure. I was thinking to use server virtualization for the situation where the moodle backup server would replace the original server when the sever was down.

On the otherhand, I wonder what is the best way to increase the number of concurrent users. In our server, we can service 110-170 concurrent users as measured by perspective.php. I want to find out what should I do if I want to increase the concurrent users upto 340, for example.

Thanks in advance.

JD Park 

In reply to Jong-Dae Park

Re: 회신: Re: 회신: Re: Performance woes!

by Marcus Green -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers

"it may be necessary to use virtualization technology for easy administration as Moodlerooms does."

In that case you will be making an easy of admin/performance trade off. Virtualisation will impose a performance overhead, running two O/S's require more resources than one.

In reply to Marcus Green

Re: Performance woes!

by Martín Langhoff -

Indeed. While the overhead on CPU is moderate, the memory overhead is quite significant and the disk IO overhead is murder because disk seeks are still very expensive.

And what's Moodle's bottleneck these days? Disk IO.

The other thing with virtualisation is that you have one more OSs to administer. I can't understand the math of "more work to do is less work to do" unless you take the infrastructures.org approach - but that is a ton of work that only pays off if you have hundreds of hosts.

Virtualisation does let you do some things easier - like upgrade from Sarge to Etch on a complex server running both at the same time, moving one app or moodle install at a time. Not what I'd do, but useful in some cases.

But "easier Moodle administration"? How exactly will I spend less hours on it? IMHO, it's just empty marketing.

OTOH, virtualisation is great for testing. Scenarios:

  • Does moodle run great on Debian, Ubuntu, RHEL, Centos, SuSE and whatnot? Here, I have 10 OSs installed on this machine, and a script to test moodle on all of them.
  • Will our production server upgrade from Sarge to Etch without a hitch? Here, clone the disk and run the upgrade under a virtual server and test the results.
In reply to Martín Langhoff

Re: Performance woes!

by Dan Poltawski -
One of the other points in favour of virtualisation is the independence from hardware, doing migrations between physical hardware in case of failure. etc. A friend of mine has a linux-ha setup with DRDB and a few virtual machines in a university, if one server rooms goes down then the machines pop up in another and continue to provide service.

I suspect most of the push is to find more ways to utilise the current processing power and push it forward.. which like you allude to is becoming less and less relevant. But the consolodation is useful for a lot of purposes (classic case being DNS), though I agree not really moodle.

With regard to the original point about managing server failure - we concentrate on redudancy and reproducability (package management and configuration management are your friends), which of course helps with easy expansion too!
In reply to Jong-Dae Park

Re: 회신: Re: 회신: Re: Performance woes!

by Martín Langhoff -
Hi JD!

Several Moodle Partners (including Catalyst!) offer tuning services and scalability consulting, as well as HA know-how.

There is also quite a bit of good advise in the servers and performance forum.
In reply to Martín Langhoff

Re: Performance woes!

by Geoffrey Rowland -
Picture of Plugin developers
Martin

Just seeking a little clarification of the 32-bit/64-bit issues with Moodle/Webserver performance.

Assuming everything (webserver + MySQL) is running on a single 64-bit architecture server, running 64-bit Linux. Is there any advantage in running the 32-bit version of Apache/PHP software with the 64-bit version of MySQL?

Or can you only effectively manage this with one, or more, 32-bit servers running 32-bit Apache connected to a separate 64-bit server running 64-bit MySQL?

In other words, is it the 64-bit architecture, or the 64-bit software implementation of Apache (or both!) that results in the 'unstable memory-eating' behaviour.

Geoff

P.S. Very much appreciated your contributions (MoodleNet, Mahara, Repositories, OLPC, etc, etc...) at the recent MoodleMoot. All vital stuff!


In reply to Geoffrey Rowland

Re: Performance woes!

by Martín Langhoff -
The 32/64 bit split is mainly for clusters. Cheers!

In reply to Martín Langhoff

Re: Performance woes!

by Geoffrey Rowland -
Picture of Plugin developers
Thanks Martin

I can now sleep soundly in the knowledge that my single, all 64-bit LAMP server is performing reasonably well, whilst dreaming of a new Moodle cluster. sleepy
In reply to Janne Mikkonen

Re: Performance woes!

by Samuli Karevaara -
Loosing ADOdb completely would require a lot of work. But some of the core selects like get_records() could be re-written to use native libraries in some cases, ADOdb in others. There is already quite a lot of DB type specific hacks required + the SQL itself is limited due to the "lowest common" factor.

I don't know the setup of your tests, but your simulation of get_records() with the native API should also do what recordset_to_array() does.

I hacked the accesslib.php a while back to use just the required fields instead of SELECT * which is sometimes used to fetch just an ID, for example. However, I was disapppointed with the results: at first I seemed to gain a lot, but in later tests the speed-up was not there. I didn't do cachegrinding so I don't know if it saved memory or not. Intuition would say that it had to save memory + the abovementioned recordset_to_array() has to be faster when there are less fields to copy.
In reply to Samuli Karevaara

Re: Performance woes!

by Janne Mikkonen -
"I don't know the setup of your tests, but your simulation of get_records() with the native API should also do what recordset_to_array() does."

Yes, the test is looping results into an array of objects, when using native API.
In reply to Janne Mikkonen

Re: Performance woes!

by Janne Mikkonen -
We can add about 10 % more performance by adding a support for ADOdb C-extension ( unfortunately it's not for all databases ). This is a quick an' dirty hack ( just for performance testing... ) to function recordset_to_array:
    if ( $rs && $rs->RecordCount() > 0 ) {
        // Check if adodb_getall exists...
        if ( function_exists('adodb_getall') ) {
            if ( $records = adodb_getall($rs) ) {
                foreach ( $records as $record ) {
                    if ( $CFG->dbfamily == 'oracle' ) {
                        array_walk($record, 'onspace2empty');
                    }
                    $objects[current($record)] = (object) $record;
                }
                return $objects;
            } else {
                return false;
            }
        } else {
            while ( !$rs->EOF ) {
                if ( $CFG->dbfamily == 'oracle' ) {
                    array_walk($rs->fields,'onespace2empty');
                }
                $objects[current($rs->fields)] = (object) $rs->fields;
                $rs->MoveNext();
            }
            return $objects;
        }
    } else {
        return false;
    }
In reply to Janne Mikkonen

Re: Performance woes!

by Ângelo Rigo -
Hi Janne

I was affraid to ask for that :

"I think it would be wise to write our own multi-db-layer code to replace ADOdb or use some other
already made multi-db-layer ( and yes it'll create again those time and money issues )."

I am using a multi-db-layer in my own framework for years, and suspect that can bring more performance.
In reply to Ângelo Rigo

Re: Performance woes!

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Moodle 2.0 dev has already replaced ADODB with a new db layer written just for Moodle.
Average of ratings: Useful (1)
In reply to Tim Hunt

Re: Performance woes!

by Ângelo Rigo -
Great to hear i do use adodb in some projects but become a performance bottleneck using moodle