Moodle in English: Chat 1.5.3+ is Unacceptable

We need help. Our school is committed to providing as much interaction between the student and teacher as possible, and we see live chats as being a valuable asset.

BUT IT DOESN'T WORK!! (sorry, for screaming)

If anyone has some advice, here's our situation:

ENVIRONMENT

Windows 2003 Server
IIS
PHP 5.0.5
msql 4.1.12
Moodle 1.5.3+

PROBLEM

Messages do not appear on the screen for users until they post something themselves. This is true for Mac and Windows users, IE Firefox and Safarai users. Further, when it refreshes it often chops of the first few posts, leaving only the latest 3 or 4 posts. It is simply not usable for a learning environment where the string of the discussion needs to be followed.

All suggestions would be welcomed!

~MC~

Average of ratings: -

Re: Chat 1.5.3+ is Unacceptable

by Martín Langhoff - Wednesday, 12 April 2006, 8:13 AM

Matt,

can you give us more detail? -- there is definitely something wrong in your setup or your browsers, server or something. Help us help you.

In any case, if the product is "unacceptable"... We Will Give You Your Money Back

Average of ratings: -

Re: Chat 1.5.3+ is Unacceptable

by Matt Cromwell - Wednesday, 12 April 2006, 7:33 PM

Sorry, "unacceptable" is such a bad word. I guess I mean unusable, or disfunctional.

But, what type of specifics. I'm not our server administrator, but I talk with him everyday, so I can get whatever you need. I tried to be specific about our environment and the behavior of the Chat Module in my first post... but... to no avail.

So, Martin, what specifics? I'm happy to give you anything you need! We don't want our money back! We want to figure out or problems.

Thanks,
Matt C.

Average of ratings: -

Re: Chat 1.5.3+ is Unacceptable

by Dan Stowell - Wednesday, 12 April 2006, 9:03 PM

I'm not familiar with the chat module myself, but one thing that occurs to me is that it might be something to do with firewalls blocking the port(s) which the chat tool may be using. Have you looked into that? It could possibly be an explanation for problems in the push/pull of chat information.

I see from some other messages in this forum that if you're using the chat daemon, it uses port 9111, as opposed to the standard port 80 (which the rest of Moodle uses).

Average of ratings: -

Re: Chat 1.5.3+ is Unacceptable

by Matt Cromwell - Wednesday, 12 April 2006, 9:51 PM

Thanks Dan, but we have the Chat Module set to "Normal Method" because the admin page seems to suggest that you have to be running on a Unix server to use the daemon:

"Using a server daemon requires shell access to Unix"

We're running on a Windows 2003 server, so we figured we have to use the Normal Method.

Here are a few more specifics:

chat_refresh_userlist: "10"

chat_old_ping: "1200" (we set this high to avoid logging people off)

chat_refresh_room: "1" (we set this low because this seemed most relevant to our problem).

Average of ratings: -

Re: Chat 1.5.3+ is Incredible!

by Martín Langhoff - Thursday, 13 April 2006, 11:22 AM

Ah! Now we are talking!

> chat_refresh_room: "1"

that's probably killing you at the database. Set it to 5 and see what happens. Make sure you are using persistent connections. When you have several users on it, ask your server admin to monitor the server and tell you what's up.

> We don't want our money back!

Ok! We'll keep it then

Once it's working, there are a couple of interesting patches I'm about to merge into HEAD that you might want to try... this is a patch series -- you need to apply them all. Use the patch utility with "-p1" to apply them ...

mod/chat: make http-chat more database friendly and change msg insertion to use POST
http://git.catalyst.net.nz/gitweb?p=moodle.git;a=commitdiff;h=711effbf1ab27bf2dff739ae528cc6ca92a541e8

mod/chat - Normal method now support HTTP Keep-Alive
http://git.catalyst.net.nz/gitweb?p=moodle.git;a=commitdiff;h=f6bbe10bb58612440ec95f97e4ac25f8e15d8467

mod/chat - Normal method now supports HTTP Keep-Alive - users pane
http://git.catalyst.net.nz/gitweb?p=moodle.git;a=commitdiff;h=fd8d3febee4a5a5caba4257d38997a588ca0c174

mod/chat - Normal method - users pane only uses keepalive when refreshing fast
http://git.catalyst.net.nz/gitweb?p=moodle.git;a=commitdiff;h=bc29cc8455802146aefbf3a4b97fb6f13ab961ed

mod/chat: Normal method -- faster, lighter DB updates
http://git.catalyst.net.nz/gitweb?p=moodle.git;a=commitdiff;h=1638af20916463fe1df902b9c260d1f45079d60b

Average of ratings: -

Re: Chat 1.5.3+ is (potentially) Incredible!

by Matt Cromwell - Thursday, 13 April 2006, 4:34 PM

Martín! Thanks for the suggestions.

The chat_refresh_room was originally set to "5" and that's when we had problems. To we changed it to "1" hoping that that would help. We're going to do a few more experiments this afternoon (GMT+1), so we'll see how that goes.

We'd love to try your patches out but blush

I have no idea how to apply this patch. We haven't installed CVS Tourtise on our server because we didn't fully know how it worked. But I've installed it on my desktop, I usually make updates to the files on my desktop and then copy them over to the server. I search all over CVS Tourtise help files and options.

If there's a simple way to explain how to make the updates with CVS Tortoise then that would be great. Otherwise, we'll have to wait until they are merged into the HEAD ('cause I know how to do that).

Thanks so much Martín! We'll see what happens.
Matt C.

Average of ratings: -

Re: Chat 1.5.3+ is (potentially) Incredible!

by Wen Hao Chuang - Friday, 14 April 2006, 12:31 AM

Yeah, some instruction about how to apply the patch would be great! By the way, is anyone in the Moodle core development team looking at integrating other open source PHP chat daemon programs (such as the PHPOpenChat, see another thread http://moodle.org/mod/forum/discuss.php?d=42790) or ejabberd?? Here at SFSU we also experienced some problems with the chat module. I noticed that on the Bug report (Bug#2146 - Integrating Jabber/XMPP... see http://moodle.org/bugs/bug.php?op=show&bugid=2146) it was assigned to Martin, but the priority was set to Low. I'm taking a look at the PHPOpenChat and the ejabberd right now but at the same time if anyone else in the core development team is working on this (or maybe consider to raise the priority level on the bug report), that would be wonderful! Thanks!! wide eyes

Average of ratings: -

Re: Chat 1.5.3+ is (potentially) Incredible!

by Martin Dougiamas - Friday, 14 April 2006, 1:43 AM

It's not something we'll be funding in the near term, but if someone else is interested ...

Average of ratings: -

Re: Chat 1.5.3+ is (potentially) Incredible!

by Wen Hao Chuang - Friday, 14 April 2006, 2:46 AM

Hi Matt, after I looked at the patch, I don't think it's that complicated to apply those patches. All you have to do is to locate those files (most of them are under the /mod/chat/ directory) and then comment out (or delete) the lines marked "-" (in red color), and add lines with the "+" mark (in green color), and that's it! I'm setting up a testing server right now to test the performance gain after applying these patches, are you interested in testing it too? Let me know.. wide eyes

Average of ratings: -

Re: Chat 1.5.3+ is (potentially) Incredible!

by Matt Cromwell - Saturday, 15 April 2006, 11:22 PM

I did try to change the files according to the patches as best as I could understand it all. Unfortunately, the results were not good.

As admin and as a student I was unable to see any posts in the chat module. Typing a message was fine, but once the message was sent from the user screen there was no longer any response from the module at all.

Bummer.

So, I'm still hoping to get a few more answers about my original post, and so far the patches seem to only make things worse.

Is there anything else anyone would suggest? ... please ... sad

Average of ratings: -

Re: Chat 1.5.3+ is (potentially) Incredible!

by Martín Langhoff - Tuesday, 18 April 2006, 5:15 AM

> Is there anything else anyone would suggest?

Try with Apache2 on Windows -- instead of IIS.

Average of ratings: -

Re: Chat 1.5.3+ is (potentially) Incredible!

by Martín Langhoff - Sunday, 16 April 2006, 5:24 PM

Don't worry about the patches -- chat should work without them for a small/medium group of users with the straight Moodle code. I don't have any experience with IIS running PHP scripts, unfortunately. I suspect that the problem lies there

Average of ratings: -

Re: Chat 1.5.3+ is Incredible!

by Martin Dougiamas - Thursday, 13 April 2006, 5:05 PM

Good lord man, check them in!

Average of ratings: -

Re: Chat 1.5.3+ is Incredible!

by Martín Langhoff - Tuesday, 18 April 2006, 5:14 AM

> Good lord man, check them in!

Merged those, plus one more. I've figured out we don't really need chatd or even an inetd-based chat. It seems I can do it all from within Apache, and save a whole lotta pain and setup.

There's a very rough version of it now on my laptop, and if I manage to get it working well soon, I'll commit it so that it's on 1.6. If that happens, the Normal method will have an extra option to enable a a non-keepalive "daemon-like" mode that should be as good or better than chatd.

If that works well, I think chatd can very well be retired. But we won't know that until it sees some real world usage.

Average of ratings: -

Re: Chat 1.5.3+ is Incredible!

by Julian Ridden - Tuesday, 18 April 2006, 2:39 PM

If you want help testing, please drop me a line. I just upgraded my box (from mandrake 10.2 to Redora 6) and have a 8 day window where I can do some serious testing without impact on users.

Average of ratings: -

Re: Chat 1.5.3+ is Incredible!

by Martín Langhoff - Wednesday, 19 April 2006, 5:39 AM

Here's a line to you!

All the code for keepalive-with-repeated-fetches is in CVS now, and MartinD is talking about hard-to-pin flakiness on the startup of the chat, which I haven't seen with the latest code.

And if you are interested in testing something more radical, there's the new "Stream" mode for chat updates hanging from here:

http://git.catalyst.net.nz/gitweb?p=moodle.git;a=log;h=streamchat

Average of ratings: -

Re: Some flakiness in chat

by Martin Dougiamas - Tuesday, 18 April 2006, 3:35 PM

That one sounds risky, leave it for 1.7 I think.

Thanks for checking in the other ones.

We did some testing here, and things mostly seem to be working, but there is some flakiness. Not sure why as it's not consistent, but we had two machines with Firefox (taking to LAMP server) just fail to update altogether (panes in chat window go blank), whereas some machines tocked along fine.

Average of ratings: -

Re: Some flakiness in chat

by Martín Langhoff - Wednesday, 19 April 2006, 5:33 AM

> We did some testing here, and things mostly seem to be working,
> but there is some flakiness. Not sure why as it's not consistent,
> but we had two machines with Firefox (taking to LAMP server) just
> fail to update altogether (panes in chat window go blank),
> whereas some machines tocked along fine.

Thanks for trying it! Can't repro here, but will try again later today getting more people involved.

> That one sounds risky, leave it for 1.7 I think.

Ok -- lucky I read this message right now, I was _just_ going to commit it! Anyway, I think it's pretty safe as it retains the standard chat approach. There's a small change in index.php to use jsupdated.php instead of jsupdate.php if the user has selected the "stream" option.

Here's the patch for review (and for testing -- hey Julian!):
http://git.catalyst.net.nz/gitweb?p=moodle.git;a=log;h=streamchat

To download a patch you can apply with "patch":
http://git.catalyst.net.nz/gitweb?p=moodle.git;a=commitdiff_plain;h=3aef5142506cf2f95ea05b868b60dc4e83101539;hp=f639aceb20af4296f9c96b16eef4d6f626e40c40

And here's the description of what it does, and how it does it...

> mod/chat: Normal method - introducing "Stream" updates.
>
> This is an alternative version of jsupdate.php that acts
> as a long-running daemon. It will feed/stall/feed JS updates
> to the client. From the module configuration select "Stream"
> updates.
>
> The client connection is not forever though. Once we reach
> CHAT_MAX_CLIENT_UPDATES (currently 1000), it will force
> the client to re-fetch it.
>
> This buys us all the benefits that chatd has, minus the setup,
> as we are using apache to do the daemon handling.
>
> Chat still defaults to the normal update method, which is now
> optimised to take advantage of keepalives -- so this change is
> safe. The instructions in the config page also indicate that this
> mode may not be well supported everywhere. It hasn't been
> tested on IIS for starters.
>
> In terms of relative cost -- if each hit on jsupdate.php incurs
> on ~20 db queries and delivers one update to the client, each hit
> on jsupdate takes ~20 queries, and then roughly 2~3 queries to
> serve each of the next 1000 updates. On busy sites, the difference
> is huge.
>
> There is still room for enhancements in both keepalive and stream
> update methods. I am pretty sure we can trim DB queries more.

Average of ratings: -

Re: Some flakiness in chat

by Martin Dougiamas - Wednesday, 19 April 2006, 9:44 AM

Doesn't streaming mean that there will still be a whole httpd process running for every person? This would cause a lot of memory usage on busy sites, and is the reason Jon and I build the chatd in the first place ...

Also, a new method should probably be implemented using a separate new chat plugin like gui_stream.

Average of ratings: -

Re: Some flakiness in chat

by Martín Langhoff - Wednesday, 19 April 2006, 11:42 AM

We talked a bit about this on Skype -- summary follows.

> This would cause a lot of memory usage on busy sites

Yes, it will be a bit of a memory hog (but a lot less of a DB hog).

The thing that kills us with chatd is most distributions ship with a PHP that doesn't have pcntl_fork() compiled in. If pcntl_fork() was easily available, I'd do a cleanup of chatd to get it going. Without forking, chatd's scalability is limited by slow clients as it serialises its serving of clients. I may still write a proper forking chatd -- but it will definitely have a very narrow audience.

In terms of gui_stream -- this is _just_ a modified jsupdate.php -- in fact, now that it runs well, I will probably fold it into jsupdate.php.

Anyway -- we've agreed that it's low-risk so I've merged the patch in. TESTERS!!!

Average of ratings: -

Re: Some flakiness in chat

by Martin Dougiamas - Wednesday, 19 April 2006, 3:13 PM

chatd does use forking if it's available. We wrote it knowing that it wasn't a default feature of PHP (it needs to be compiled in) assuming that only people with really big sites would need it (and that they could recompile PHP if they needed to).

Streaming doesn't replace chatd at all, but what you've done IS a very welcome improvement over the standard chat, so thank you!

Testers!!

Average of ratings: -

Re: Some flakiness in chat

by Martín Langhoff - Thursday, 20 April 2006, 5:13 AM

chatd does use forking if it's available.

Surprisingly (and unfortunately) it doesn't! The relevant section of the code is commented out And that's part of the original 1.1 commit. I guess it was meant to be an "we'll enable it when it works better" but it hasn't been enabled.

We wrote it knowing that it wasn't a default feature of PHP

Agreed. Seems to be present in most php4-cli packages out there nowadays. I had been stupidly probing the mod_php4 packages and those don't have it compiled it -- it is probably a big mess to fork off the whole of the apache+php process so packagers don't enable it.

Streaming doesn't replace chatd at all, but what you've done IS a very welcome improvement over the standard chat, so thank you!

You're welcome! As you say, it isn't a replacement -- but if it works as well as I think it works, it will ease the pressure on chatd. (Still very large setups will benefit from a proper chatd, no doubt, as the streaming strategy is memory-heavy.)

Give me a rainy weekend, however, and I'll revamp chatd to actually use forking, and to be simpler. Like 10% of the code it was now. I suspect that we can make it a trivially simple wrapper around the very same scripts that gui_header_js uses.

Average of ratings: -

Re: Some flakiness in chat

by Martin Dougiamas - Thursday, 20 April 2006, 2:42 PM

Goodness, it *is* currently commented out!

Sorry, I hadn't realised that!

Jon must have encountered some last-minute problem with it that he couldn't solve at the time. Pity he's in the Army now so we can't ask him.

/me wishes you rain in Wellington.

Average of ratings: -

Re: Some flakiness in chat -- resolved

by Martín Langhoff - Wednesday, 19 April 2006, 6:46 AM

Hmmm! I know what the problem is with the flakiness -- $USER->lastIP is sometimes not set properly during login. I don't know why, in those cases logout and login again and it will be set correctly.

Fixed in CVS with this patch
http://git.catalyst.net.nz/gitweb?p=moodle.git;a=commitdiff;h=065c5b764e4a437d47a51b870cdcaeed104f561e

Average of ratings: -

Re: Some flakiness in chat -- resolved

by Martin Dougiamas - Wednesday, 19 April 2006, 9:46 AM

Are you sure we want to throw an error there? I don't think get_remote_addr() is always reliable for AOL users etc

Average of ratings: -

Re: Some flakiness in chat -- resolved

by Martín Langhoff - Wednesday, 19 April 2006, 10:30 AM

Hmmm. If that's the case, then IP should be nullable.

I am thinking of the chat_users table...

CREATE TABLE `prefix_chat_users` (
...
`ip` varchar(15) NOT NULL default '',
...
) TYPE=MyISAM COMMENT='Keeps track of which users are in which chat rooms';

I'll get a patch for that tonight.

Average of ratings: -

Re: Some flakiness in chat -- resolved

by Martín Langhoff - Thursday, 20 April 2006, 5:38 AM

Fixed in head -- no DB changes, just set it to '' as MD proposed.

Testers?

Average of ratings: -

Re: Some flakiness in chat -- resolved

by Matt Cromwell - Thursday, 20 April 2006, 6:01 PM

Martín!! You did it! Whoo hoo!

We updated the chat module from moodle.org (CVS didn't have any changes as far as we could see). We used streaming instead of keep-alive and it works great.

Formerly, with keep alive we could only see others' posts if we kept hitting "enter". Plus, the CPU usage was peeking fairly high, 90% or so. Our server admin said that wasn't a major problem for our amount of students, but we thought it was strange.

Now, with streaming set, the CPU usage peeks at about 20%, and messages pop-up in "real" time (wow! what a concept!). The only thing is that when someone posts a message it seems that the other chatters see the message before the poster does. This is only by a couple seconds.

We also experimented with setting the chat_refresh_ room setting at "5" and "1".
at "5" the lag between sending a message and seeing it was longer (I'm sure that's the purpose of the setting)
at "1" the lag was only slightly shorter, and the difference on our CPU was not major. But we were only testing with 2 people.

We have a scheduled chat tomorrow at 18:00 (GMT +1) where we will have around 7-9 chatters. We will be sure to post the results. Unfortunately, I'll won't be able to post again until May sometime (I'll be in Albania for a class). But I'm happy to leave my post here with a lot more peace of mind.

Thanks moodle friends!
Matt C.

Average of ratings: -

Re: Some flakiness in chat -- resolved

by Julian Ridden - Thursday, 20 April 2006, 9:22 PM

Have installed this on my 1.6 and have tested it with 10 concurrent users with no issues to report.

Will need to update the lang file however with the two new strings.

Average of ratings: -

Re: Some flakiness in chat -- resolved

by Martín Langhoff - Friday, 21 April 2006, 4:22 AM

Good to hear! Missing strings??? I'm sure I've committed them to lang/en_utf8/chat.php ...

Average of ratings: -

Re: Some flakiness in chat -- resolved

by Martín Langhoff - Friday, 21 April 2006, 4:32 AM

Excellent! With 'Stream' even high refreshes should be ok (set it to "1" and forget!). Now tell us more about the setup you've been using... webserver? database? clients?

Have fun in Albania!

Average of ratings: -