CVS is still down

CVS is still down

by Martin Dougiamas -
Number of replies: 28
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
Argh... CVS is still down. sad

https://sourceforge.net/docs/A04/

On 2006-03-30 the developer CVS server had a hardware issue that required us to take the service offline. We are actively working on this problem and hope to have it back up soon. There is not a current estimate for the duration of this outage, but when we get one, it will be posted on the site status page (this page).
Average of ratings: -
In reply to Martin Dougiamas

Re: CVS is still down

by Nick Freear -
Anonymous CVS access seems to be OK, for anyone who wishes to update. Commits using SSH cause a problem for me. No more news from Sourceforge unfortunately.
Average of ratings: -
In reply to Nick Freear

Re: CVS down for at least another 48 hours

by Gustav W Delius -
The Sourceforge CVS status page has been updated to say that: "We currently expect this outage to last 48 hours, at minimum."
Average of ratings: -
In reply to Martin Dougiamas

Re: CVS is still down

by Jason Cole -
Would it be worth considering creating a distributed SCM system to get us off of relying on SourceForge? We were discussing today the idea of creating an svk based system of distributed repositories. The OU could host one, maybe Catalyst and Moodle.org would host additional main repositories. It would be possible then to use svk to synchronize across these repositories if one of them went down (perhaps we'd designate Moodle.com as the primary with OU and Catalyst as secondaries). Local checkouts would be distributed, and commits would happen to the most available repository in order of priority (Moodle.com, Catalyst, OU).  The mirror sites would then replicate the changes.

It might be possible to also do distributed commits, but there would be more work to monitor the code merges and potential conflicts...

As a project manager, the reliance on SourceForge causes me a bit of concern (especially considering how much time we spend trying to work around sf rather than with it).

Jason
Average of ratings: -
In reply to Jason Cole

Re: CVS is still down

by Martin Dougiamas -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
I'm starting to think of alternatives as well. This has been the worst downage ever from Sourceforge, by far, but perhaps it's a good time to leave anyway.

The best thing about Sourceforge is the handling of gosh knows how many thousands of users on anonymous CVS. That's a LOT of bandwidth there. In the first year or so I ran my own CVS server for Moodle but as access increased I was very glad to move it to Sourceforge. These days, phew.

The other good thing about Sourceforge is the management of usernames, SSL certificates etc. People can use one account for multiple projects. This is not such a big deal, though, we could work something out.

To cope with bandwidth (and lag!) using mirroring is a good idea, but I really would not want to start getting into a mess of conflicts, so it would have to be really well structured and planned. I'm sure Martin Langhoff will want to jump in with something about GIT right now. smile
Average of ratings: -
In reply to Martin Dougiamas

Re: CVS is still down

by Jason Cole -
I'm sure he will...

However, Rod and I were discussing a few mechanisms for master and slave repository relationships. If we set up a system where Moodle.com is the master and Catalyst and OU are slave repositories in cascading order, it would resolve the check-in conflicts. Essentially, the current master would be for commits, and would replicate those out to a global network of slaves for anonymous checkout. If, for example, Moodle.com lost connectivity, Catalyst would pop-up as the new master. Once Moodle.com restored connectivity, it would resynch with Catalyst and then resume as master.

We were thinking of some clever ways to do this (including a manual switch with a WAP interface so when the power goes out, you can still reach it).

Once we get the May release out the door, we'll have some bandwidth to actually work on implementing this if we have a plan. We'd also be happy to work with Catalyst as well.
Average of ratings: -
In reply to Martin Dougiamas

Re: CVS is still down

by Martín Langhoff -
smile I knew you'd be thinking of me... have we got a recent cvs tarball? If we do, I can set it up on Eduforge in no time -- it will be a bit labour intensive to setup all the relevant accounts though, and perhaps re-applying missing commits (cvs emails can help here).

In terms of distributed SCMs, we are working, unaffected by this so far, and we'll push our patches into CVS when it's back. This hybrid (CVS+GIT) model is keeping us sane, even though the downtime is a real pain.
Average of ratings: -
In reply to Martín Langhoff

Re: CVS is still down

by Martín Langhoff -
Fetching the latest tarball from http://cvs.sourceforge.net/cvstarballs/moodle-cvsroot.tar.bz2 to see how fresh it is.

I will be more than happy to set it up on Eduforge.org with CVS or GIT wink


Average of ratings: -
In reply to Martín Langhoff

Re: CVS is still down

by Martín Langhoff -

Update: fetched successfully, it untars cleanly and it seems to have up to the last commit that I've seen in the moodle-cvs mailing list, which was

Update of /cvsroot/moodle/moodle/mod/data

In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv1690

Modified Files:

  tabs.php

Log Message:

Fixed tabs logic flow

So we are in good shape -- no data loss so far.

Average of ratings: -
In reply to Martín Langhoff

Re: CVS is still down

by Martin Dougiamas -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
The very latest tarball (I archive it automatically) seems smaller than the one before, which is slightly worrying.
Average of ratings: -
In reply to Martin Dougiamas

Re: CVS is still down

by Gustav W Delius -
There is good news and there is bad news from Sourceforge on their status page.

The bad news is that they don't know how long it will take to restore CVS but it sounds like they are thinking days rather than hours. In addition after they restored developer CVS access they will wait "several days" before this is synchronized with anonymous CVS. So this means that we can probably expect another week's delay in the testing of Moodle 1.6 beta.

The good news is that in future Sourceforge want to avoid having a single point of failure in their CVS service, meaning that we should expect such outages not to happen in the future.

Here is the statement in full:

"On 2006-03-30 the developer CVS server had a substantial system failure. Due to the implementation of the CVS service, there is a single point of failure with multiple points of recovery (there is more than one data source we could potentially recover from if there is any data loss as a result of the failure). This outage currently affects developer CVS access directly, but we have disabled tarball updates and data syncs from the developer CVS server to the anonymous pserver/ViewCVS hosts as an additional level of precaution. Our main focus since the outage was detected has been to safegaurd all data on the developer CVS server as well as possible. We are currently attempting to backup the data on the host, which is taking longer than we initially anticipated it would, but is a necessary step to fully safegaurd the host's data. Next, we are going to perform some data validation to ensure the data set appears valid. Pending successful completion of those steps, we'll reenable developer CVS access. A few days after, we'll reenable CVS tarballs and syncs to anonymous CVS. In the mean time, we're currently advancing plans for a CVS architecture change based upon the knowledge we gained during Subversion deployment to eliminate the single point of failure that developer CVS currently has, add horizontal scalability and overall service resiliance. However, we still do not have an estimate on when developer CVS services will be restored, but we have been, and currently are actively working to restore access to CVS. We appreciate your patience with us while we work to properly resolve this major outage."
Average of ratings: -
In reply to Gustav W Delius

Re: CVS is still down

by Martín Langhoff -

We are currently attempting to backup the data on the host, which is taking longer than we initially anticipated it would, but is a necessary step to fully safegaurd the host's data. Next, we are going to perform some data validation to ensure the data set appears valid.

Ouch! I smell a disk crash or FS corruption.

Average of ratings: -
In reply to Gustav W Delius

Re: CVS is still down

by Martin Dougiamas -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
Yes, that is good news. Perhaps this outage is just what we (the sourceforge communtity) all needed.

I'll play wait and see for a little while longer to see how this plays out before considering any major moves to a new system.

The thing that makes this worst is just the coincidence that it happened exactly as we went Beta and were entering our most intensive bug fixing period.
Average of ratings: -
In reply to Martin Dougiamas

Re: CVS is still down

by Gustav W Delius -
And another update:

( 2006-04-03 14:04:56 - Project CVS Service )   As an update to the 2006-03-30 CVS outage, our current estimate is that CVS services will be back online (developer access) late Tuesday or early Wednesday (Pacific Timezone).
Average of ratings: -
In reply to Martin Dougiamas

CVS UP! ?

by Dan Marsden -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Plugins guardians Picture of Testers Picture of Translators
Looks like developer cvs might be back online!

smile

Dan
Average of ratings: -
In reply to Dan Marsden

Re: CVS UP! ?

by Dan Marsden -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Plugins guardians Picture of Testers Picture of Translators
here's the update:

Recent Updates (2006-04-04 14:19:55 - Project CVS Service ) On 2006-04-04 at 14:15 Pacific developer CVS services were re-enabled after replacement of the failed hardware, successful completion of a full host backup, and a cursory data validity check on the host. As a precaution, we have not re-enabled the sync between anonymous pserver/ViewCVS and the developer CVS host. We anticipate that the sync's along with tarball generation will be re-enabled on Friday.
Average of ratings: Useful (1)
In reply to Dan Marsden

Re: CVS UP! ?

by Martin Dougiamas -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
Let the bug fix check-in fest begin! approve
Average of ratings: -
In reply to Martin Dougiamas

Re: is anonymous CVS working?

by Grigory Rubtsov -
Is anonymous cvs following? I can see no updates in HEAD since Mar 30.

Sincerely yours,
Grigory Rubtsov
Average of ratings: -
In reply to Grigory Rubtsov

Re: is anonymous CVS working?

by Anthony Borrow -
Picture of Core developers Picture of Plugin developers Picture of Testers

It appears that anonymous CVS is being worked on but not yet read. The latest from the SourceForge status page is:

( 2006-04-06 13:04:54 - Project CVS Service )   As result of the 2006-03-30 CVS outage, and previous CVS outages, some CVS repositories do not exist on the anonymous/ViewCVS CVS hosts. This will result in proxy errors and other various unusual results when trying to access those hosts. Once we get the CVS services into better order, we'll start the synchronization processes again and these problems will go away. Our primary concern right now is for the security and reliability of developer CVS access, after which we'll restore anonymous access.

Average of ratings: -
In reply to Anthony Borrow

Re: is anonymous CVS working?

by Grigory Rubtsov -
For testing of the new code I wish to always have a moodle taken from cvs HEAD. Can I get a read-only developer cvs access? My sf.net id is "rgbeast".
Average of ratings: -
In reply to Grigory Rubtsov

Re: is anonymous CVS working?

by Just H -
Hi Grigory

If you just want to checkout the latest, don't you just get it using anonymous access? (Forgive the following if it's a bit obvious - it's just a cut and paste from a text file I use in case I forget it, guessing you already know as you have a sf id).

To connect and login for the first time to the CVS server, you can use this command:

cvs -d:pserver:anonymous@cvs.sourceforge.net:/cvsroot/moodle login

There is no password - when asked for one, just hit Enter.

To checkout (download) the entire Moodle code for the first time, use this command to get the latest STABLE version:

cvs -z3 -d:pserver:anonymous@cvs.sourceforge.net:/cvsroot/moodle co -r MOODLE_15_STABLE moodle

Or the latest development version (head):

cvs -z3 -d:pserver:anonymous@cvs.sourceforge.net:/cvsroot/moodle co moodle

or contributed modules/blocks

cvs -z3 -d:pserver:anonymous@cvs.sourceforge.net:/cvsroot/moodle co contrib

Did a fresh install of 1.6 beta earlier this evening, anonymous cvs still seems to be a few days old due to the problems sourceforge where having as mentioned above.

Regards
H
Average of ratings: -
In reply to Just H

Re: is anonymous CVS working?

by Grigory Rubtsov -
Dear Harry,

I am using anonymous cvs you mentioned for 11 months already and often having a problems with the delay, and now I can only update to the version of Mar 31. The delay often leads to reporting bugs already fixed in cvs.

Sincerely yours,
Grigory Rubtsov.
Average of ratings: -
In reply to Grigory Rubtsov

Re: is anonymous CVS working?

by Just H -
Hello again Grigory

Thought you already new about it but wasn't sure hence posting, no offence meant smile

The delay has been bad recently, especially as the developers where pushing out 1.6 sad  On the plus side, your version seems to be 2 days newer than mine?

Hopefully source forge will be up and running again soon for anonymous. Not being a developer, can't say how easy/hard it is to get full cvs acces to Moodle.

Regards
Harry
Average of ratings: -
In reply to Just H

Re: is anonymous CVS working?

by Grigory Rubtsov -
Dear Harry,

I am not so active to request full developer access, as the more people have access the larger is the management overhead put into the heads of moodle core team. I just wonder if it's possible to get read-only developer access.

Hope the problems with anon cvs will disappear soon. Sf.net state that with svn they have no delay at all - probably moodle shall migrate to svn some day in the future.

Best wishes,
Grigory Rubtsov
Average of ratings: -
In reply to Grigory Rubtsov

Re: is anonymous CVS working?

by Martin Dougiamas -
Picture of Core developers Picture of Documentation writers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
No, sorry. If I start giving out read-only access to the developer CVS to some people then I have to do it for everyone.

I'm sure SF will get their anonymous CVS server working again soon - the developer server has been going fine since it came back up.

Note that the versions on the Moodle download page is built from the developer server if you want more recent testing versions.
Average of ratings: -
In reply to Martin Dougiamas

Re: CVS UP! ?

by Ludo (Marc Alier) -
HI Martin,
UTF now works in the new wiki, along with lots of points in the list of changes,  bugs and improvements you and Eloi suggested. Are we still on time to squeeze it on  1.6 beta?
Regards
Ludo
ps. last wiki bundle is on dfwiki home
Average of ratings: -