Visvanath -

It looks like I stirred up a hornet's nest here...

W/re rich function vs not - I think there's very much a place for both (and let the marketplace sort it out). Clearly Google has done well with the simple, lean and mean approach...  but clearly there are more feature rich products which are also doing well. I think the Moodle developers are in a pretty sweet position in that: those that don't want the richer functions, they can stay with 1.9, probably can do so for years and years, and not be crippled in their teaching; conversely, for those that want more features/function, there will be 2.x. Clearly bloatware is a liability, and that's one of the dangers of rich-features, ie, the size and resource demands must not scale faster than the function set. MS with some of its mid-generation Windows clearly added bloat much faster than usable feature...  and the users noticed (shall we say).

W/re optimization - I have never seen anything even approaching the level of packing and cycle optimization that was in the Apollo code, and having said that, never hope to ever again. The *ix kernal is orders of mag removed from that level of opt. I could probably do a multi-hour talk on what we had to do to make the thing work/fit. [ever written 12m lines of assembly code, with an instruction execution timing chart at hand..  let's see: an add fullword (memory to register) runs 1.5usec, but uses up an extra 2 bytes of data memory, where a add halfword runs 1.9usecs, and doesn't use those 2 bytes...  so a visit to the costing committee to see which we have less of in that overlay: cycles or memory, then choose.)

w/re forgiving code: if the code is perfect, and I literally mean that, then one can strip out all the recovery code, all the value checking, all everything that isn't directly related to getting three men in a can home safely. But when one does get a core dump (yes, core), generally there was so little left that one couldn't debug it. Forgiving code is exactly that - the user can do something unexpected, and it doesn't crash and burn, but does something useful/meaningful about it.  I think users today are much less tolerant of a blue-screen system lockups. That comes at a cost.

w/re the armada - we're not talking about that much of a computer, really, more like $2500 +/- (though not big name built, but one that one puts together themselves, ie, no 400% profit margins for Dell or HP etc). And yes, the same bits hopefully will arrive nomatter what... but user time is also valuable, beyond instructor time, beyond IT support people time. But beyond that: by dispatching a transaction as quickly as possible, there is a much higher probability of the hw cache working in one's favor - if it's spread out over time, then the cache probably will contain bit and pieces of multiple transactions, meaning one is running from ram, which for most every modern processor is a huge bottleneck. If one can run from cache, and avoiding processor stalls, one can get by with 1/5th the raw processing capability. I think most people that are designing high-performance transaction processing system have long ago learned this: the longer period of time over which a sequence of instructions are executed, the long time^2 it takes. And given the instruction path lengths within Moodle (at least 1.9) even a 3.0ghz proc (maybe 4 core) allowed to run from cache delivers very impressive performance.

w/re parallel NICs - two approaches: one, yes, bonded, aka shotgun; the other is to have more than one T3s coming in from a backbone. In either case, having one or two cores aggregating incoming traffic and feeding the other 6 -14 cores.

To expand on Tim's comment: in the world of the infinitely-fast server, which I think Greg was positing, there is never any need for more than one server in order to achieve performance, so a single server is definitely the best option.

In the real world we don't have infinitely fast servers. That means we need to look at a system where we delegate all the CPU stuff that Moodle wastes time with (e.g. overhead due to being written in PHP, inefficient coding) to front-end servers. Performance of front-end servers literally doesn't matter because you can have as many as you need. That's where the 'throw hardware at it rather than coding efficiently' approach comes in.

Unfortunately at the moment it is difficult to configure Moodle to work with multiple database servers (master/read-only slave clusters), which means that the critical bottleneck is database performance, which means it's important to reduce unnecessary database queries.

I think our database server currently has 64GB RAM. Maybe the next upgrade will have 256GB; that would be enough to hold the entire DB in RAM again I guess...

Latency over typical internal networks (gigabit ethernet or whatever) is not that bad, incidentally. I don't think it's a big deal if a page makes 100 queries from that perspective. Like Tim indicated we don't really care about tiny degradation in performance for a single user (say the latency adds 100ms to the request, which they won't even notice given how slow the javascript is) if it means we can handle many more users at the same time.

--sam

Tim, sir,

Clearly there is a point at which one must separate, and yes, after one does that there is a much more linear scaling...  and with the number of students at OU you're clearly above that point. But my experience has been that too many shops make that jump way too early (*) and before really optimizing what they have...  as I posted before: 64GB of memory is less than $600 and can go a long ways toward make much better use of the resourcs one already has.  I strongly suspect most 2 year and 4 year schools can get by with a lot less in terms of hardware, if they'd balance the resources, and more imprortantly understand how they're being used. To have all of Moodle in the APC cache, the entire db in memory, the current workingset of data in memorymapped and each transaction processed from end to end in one go, ie, no task swaps, no processor stalls is clearly the optimal way to go. Anything else becomes exponential expensive.

(*) I also see a lot of "mine's bigger than your's" syndrome, where there seems to be a point of pride to split off the the db (and then hang it off a 100mps link :-O ), just to be able to say that they have a separate db server, having zero clue that doing such just cost them 500% in response time, besides a huge hit in total system reliability.