weird: two different moodle servers stop functioning at the same time

weird: two different moodle servers stop functioning at the same time

by Robert Brenstein -
Number of replies: 8
I posted this in the authentication forum because it affects login but I don't think it has anything to do with authentication functionality per se

http://moodle.org/mod/forum/discuss.php?d=102686

Two different Moodle servers on two separate physical subnets, one running 1.8.3+ and another 1.5.2+. Both running under up-to-date OSX Tiger but one with PHP 5.2.4 and another with 5.1.6. Both have in log "Apache PHP configured -- resuming normal operations" (see other post for full msg) within minutes from each other at 3 am or so in the morning, and both stop communicating with remote pop3 server from that moment on. That goes on for 3 days now - just as the semester ends sad

Administrator can log in as before (uses manual authentication) and both Moodles work just fine for him. Login to pop3 service thru Telnet connection works fine, so the problem must be somehow within each server (PHP? OSX?) but no errors are logged.

Restarting, changing pop3 params, etc is of no help.

Anyone any ideas?
Average of ratings: -
In reply to Robert Brenstein

Re: weird: two different moodle servers stop functioning at the same time

by Dan Marsden -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Plugins guardians Picture of Testers Picture of Translators
are you running scheduled course backups on both at the same time?

smile

Dan
Average of ratings: -
In reply to Dan Marsden

Re: weird: two different moodle servers stop functioning at the same time

by Robert Brenstein -
No. No automatic course backups within either Moodle.
Average of ratings: -
In reply to Robert Brenstein

Re: weird: two different moodle servers stop functioning at the same time

by Martín Langhoff -
- OSX magic updates?
- a networking change of some sort?
- reverse dns broken?
- tighter tcpwrappers policy on the POP3 server?
- magic OS updates on the POP3 server?
- a cronjob running cvs up nightly (a few truly suicidal admins do this!)

The 3am thing is from the rotatelogs script, you'll find it every day. It isn't at midnight on most unixen (surprise!).
Average of ratings: -
In reply to Martín Langhoff

Re: weird: two different moodle servers stop functioning at the same time

by Robert Brenstein -
- OSX magic updates?

No automatic updates for OSX

- a networking change of some sort?

no idea, but shouldn't this affect telnet connection as well?

- reverse dns broken?

didn't look at DNS but one of the Moodle servers uses IP address instead of the hostname

- tighter tcpwrappers policy on the POP3 server?

the server people were doing some changes (certificate update) but on Monday during work hours and the login problem has been going on the whole weekend. I am not using secure access, so this should not affect me anyway.

- magic OS updates on the POP3 server?

again, wouldn't this affect my connecting manually to pop3 as well? The pop3 server is the main mail server for the university and mail access works fine from the campus as well as from outside.

- a cronjob running cvs up nightly (a few truly suicidal admins do this!)

nop, no automatic cvs updates.

There is a backup process running at night (SuperDuper) but it runs outside Moodle and simply makes a copy of new or updated files to another hard drive. Whatever OSX schedules internally runs as well but why log rotation would affect pop3 access through PHP? A malware attack sounds more likely but haven't heard of anything and looking at Moodle security alerts brings nothing.
Average of ratings: -
In reply to Robert Brenstein

Re: weird: two different moodle servers stop functioning at the same time

by Martín Langhoff -
Just to confirm - when you say "connections via telnet" you mean that you tried telnet from one of the servers that runs moodle - right? Instead of from some other host wink

A couple more ideas

- SELinux or other similar security lockdown mechanism blocking outbound network connections from apache-user-owned processes. I am a bit out of touch with server-type OSX and what lockdown tools it has.

- If you are using a PHP accelerator, a corrupt precompiled file. This would explain one host misteriously failing, not two... in case of suspicion, I stop apache, rm -fr all the files, sleep 1, start apache.

- run the telnet under the apache account

- under the apache account, try the php commandline tool, and connect with the pop3/imap tools to your server

- enable "developer debugging" to see all the ugly msgs
Average of ratings: -
In reply to Martín Langhoff

Re: weird: two different moodle servers stop functioning at the same time

by Robert Brenstein -
Thanks for all the ideas.

Yes, I telnet from the Moodle server.

Lockdown could be it but I am not aware of anything running. I definitly did not install anything. These are not new servers and they were not played with lately in terms of configuration.

No PHP accelator.

Apache runs as user www but there is no such a login account. Tried 'sudo www' but no idea what the password can be. Nothing I tried is accepted. Root is not active on this computer.

The SMTP server connection from Moodle works fine. Just posted to a forum and got msg through email.
Average of ratings: -
In reply to Robert Brenstein

Re: weird: two different moodle servers stop functioning at the same time

by Robert Brenstein -
I just remembered that I still have pop3x-based users. pop3x is a pop3 authentication which allows to specify a different pop3 server for each user. In other words, it uses different pop3 servers than the university. I tried two different ones and THEY WORK !!! So the problem is specific to uni pop3 server after all.

I guess I need to talk with central server folks again. Before, they made me try to login to Moodle while they were watching pop3 accesses. They said they did not see any request for me. This was followed by the manually opening a telnet connection to the pop3 service, which worked, so they concluded the problem is at Moodle side.
Average of ratings: -
In reply to Robert Brenstein

Re: weird: two different moodle servers stop functioning at the same time

by Robert Brenstein -
I just had a debugging session with folks at the computer center. It is a real mystery.

Manual pop3 connections work.
Moodle pop3 connections don't work*.
Moodle imap connections work.
Moodle pop3 connections to out-of-campus pop3 servers work.

All connections are from the same computer.

So, the way out of impass is to change all users to imap. I would still love to know why pop3 stopped to work suddenly.

*) By not working, I mean that the sysadmin of the mail server does not see the incoming connection and Moodle reports login failure. No errors are logged even at the highest debugging level except those reported earlier.
Average of ratings: -