A 15 year old student put Moodle down

A 15 year old student put Moodle down

by Bernat Martinez -
Number of replies: 23

Hi, 

One week ago our Moodle (1.9.19) went down, we informed our provider (Hostgator) and they said us that the cause was a lot of connections from a specific IP. 

Some days later a student boosted with their friends  that he had caused it. We checked his activity log and find that both IP coincided. 

So we ask ourselves is it so easy for a 15 year old student to put Moodle down? It could happen the same in Moodle 2? 

Please have a look at the images,  we are not techie-server people so we cannot understand that IP that Hostgator informed us, was the student IP not for this day, but 5 days later!!

Many Thanks in advance for your help

PD: We have not yet talked with the student about the issue, we want to know something more before doing so.

Attachment Captura de pantalla 2014-05-22 a les 20.17.48.png
Attachment Captura de pantalla 2014-05-23 a les 11.35.17.png
Average of ratings: -
In reply to Bernat Martinez

Re: A 15 year old student put Moodle down

by Nigel Irwin -

Hi

The dates do not match anyway. In the first graphic they are for May 15 and in the second they are for March 20 and June 15.

Average of ratings: -
In reply to Bernat Martinez

Re: A 15 year old student put Moodle down

by Richard Oelmann -
Picture of Core developers Picture of Plugin developers Picture of Testers

Besides which that would really not be a case of Moodle itself crashing but of your hosting provider's rules on load limits for its servers. Particularly on a shared hosting service (you don't say what level of hosting you have with the providers) this can be quite low limits.

Average of ratings: Useful (2)
In reply to Richard Oelmann

Re: A 15 year old student put Moodle down

by Bernat Martinez -

Thanks Richard, sorry I forgot to say we are using a VPS hosting with  2G RAM (used 1.7) and 120 disk space (only 70% used)

Average of ratings: -
In reply to Bernat Martinez

Re: A 15 year old student put Moodle down

by Ken Task -
Picture of Particularly helpful Moodlers

Googling for: "php invoked oom-killer in ub"

Finds several links.  One for MySQL.  One for OpenVZ.  Even one for Facebook.

oom = out of memory and has to do with the systems memory management.

Seems to me IF the ip address happened to be accessing Moodle at the time it's actions (whatever the student was doing at the time) could have been using something Moodle that required memory and thus it was recorded.   It may not, however, indicate the person using the IP address was responsible.   Students do sometimes claim to have done something ('cool factor?'), but didn't purposely.

Does server keep apache access logs?   Check those for that IP and the date of the error (if your logs go back far):

fgrep 'IPADDRESS' access_log

Should show accesses by that IP address.

fgrep '[19/May/2014' access_log

Should show all hits on that day.

And something related ... I see vps### in the messages log clip provided ... that's a virtual box.  Had issues with a Moodle installation one time that was really a VMWare problem.   Another guest OS on physical server had been configured to use all available resources (ie, memory, etc) when it ran a certain process.   Moodle was in-accessible for that time period.   Not supposed to happen!

Both PHP and MySQL have memory limits settings.  A VPS package is purchased with memory caps.  Is PHP and/or MySQL (or combo) configured to use more than the VPS package can provide?

Above, of course, is a guess, given the information shared.

'spirit of sharing', Ken

Average of ratings: Useful (2)
In reply to Ken Task

Re: A 15 year old student put Moodle down

by Bernat Martinez -

Thanks a lot Ken for your detailed response

Yes we are using a VPS hosting with  2G RAM (used 1.7) and 120 disk space (only 70% used). 

Maybe you are right and it was a VMWare problem, and the provider is trying to isguise it.

Average of ratings: -
In reply to Bernat Martinez

Re: A 15 year old student put Moodle down

by Guillermo Madero -

Hi Bernat,

While the IP address could be useful, you have to take into account that it probably is a shared IP.

You also need to take into account that youngsters will swank about anything they can. What the student said might actually be just fake.

The provider's report seems to be a process log. How about getting the web server access and error logs instead?

You should check the Moodle logs for the day in question. There's no use in checking other date.

Average of ratings: Useful (1)
In reply to Guillermo Madero

Re: A 15 year old student put Moodle down

by Bernat Martinez -

Hi Guillermo, 

We have checked Moodle logs for that day and everything seems to be normal. Maybe you are right, the boy was pretending to be cool with friends and it was just a coincidence 

Average of ratings: -
In reply to Guillermo Madero

Re: A 15 year old student put Moodle down

by Bernat Martinez -

Hi Guillermo, 


Following your advice we have checked Moodle logs for that day and have found a lot of loggings using guest access from same IP (but different from student's), see below

 this could be the cause?

Attachment Captura de pantalla 2014-05-24 a les 11.23.14.jpg
Average of ratings: -
In reply to Bernat Martinez

Re: A 15 year old student put Moodle down

by Guillermo Madero -

Hi Bernat,

Three or four clics each minute, nothing extraordinary here. You definitely need the server access and error logs.

Average of ratings: -
In reply to Bernat Martinez

Re: A 15 year old student put Moodle down

by Ken Task -
Picture of Particularly helpful Moodlers

Ok, think you are fixated on using Moodle logs/reports for something that is really a virtual server/operating system issue.   The guest access is a setting in Moodle which you can turn off.

The messages log shared with you by tech support you might be able to investigate yourself in a VPS with command line access.   Although it looks like a log for the VPS systems setup on a virtual hosting server(s).

Would help to know what operating system your VPS is .... is it Ubuntu?  is it CentOS?  is it 'other'?  Makes a difference in where to find logs and how the log files are named.

Example is for CentOS - typically configured ... which may NOT be the same way your provider setup the VPS):

cd /var/log/

ls (will show all the logs and folders for other logs)

The messages log (I think because I don't have any 'kernel' lines in mine) is what the tech shared with you.  In the listing above one might see a messages file and other messages.1, messages.2, messages.4 files.   The 'dot' number files are old logs.   The currently used log is the messages file.   There is also, in CentOS, a 'secure' log.

Only by searching apache logs (access and error) and or other system logs ... messages, secure, etc. will one *might* be able to determine if the activity was malicious.   But, according what was shared, it was a memory management issue.

BTW, a 2 Gig box isn't much for newer versions of Moodle - think minimal for the version you are running now ... 1.9.19 (which is no longer getting security updates/fixes/patches).

Think I'd inquire with provider about what one of those lines means ... in humanly understandable form ... not virtual server speak.

'spirit of sharing', Ken

Average of ratings: -
In reply to Bernat Martinez

Re: A 15 year old student put Moodle down

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

I'm not entirely convinced that protecting your server from DOS attacks is a Moodle issue per se. 

I've definitely been here on several occasions myself. You can try running mod-evasive on apache (but it breaks Theme Developer mode as it loads so many pages it looks like an attack!) or you can simply block IPs that cause trouble. Probably both. 

It's another curse of being a system administrator. 

Average of ratings: -
In reply to Howard Miller

Re: A 15 year old student put Moodle down

by Dale Davies -

Got to agree here, Moodle is (of course) as secure / performs as well as it can be made to, plus the community of developers will always be striving to make it better.  This is the same as any self hosted CMS, LMS or any other web site you host on your own servers. 

There will be limits as to how far you can push the site before you really need to dig deep into the server configuration, but this requires an in-depth understanding of how things like the OS, web server, php, and database interact with each other. Sometimes it can be easy to shoot yourself in the foot.

Have you considered that this may not have been an "attack", but just a traffic spike (an unusually high amount of normal/legitimate traffic)?

As Howard suggests, to mitigate against this type of problem you could install mod-evasive, IMHO you should probably not be using Theme Developer mode on a production site anyway.

You could also use a service like Cloudflare, this is a CDN and basic web application firewall so should help reduce the load on your server while speeding static assets, Cloudflare will block a lot of spam traffic and give you an easy way to block problematic IP addresses too.




Average of ratings: -
In reply to Dale Davies

Re: A 15 year old student put Moodle down

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Reading this thread, it occurs to me that there are similar problems throughout society. I sufficently disenfranchised 15-year-old could spray grafitti on the school wall, or even try to burn down the school, or throw a brick of a bridge an cause a car or train crash. The reason this does not happen is more to do with the fact with live in a civilised socitey than due to technical protections.

Average of ratings: -
In reply to Tim Hunt

Re: A 15 year old student put Moodle down

by Dale Davies -

Hmm, I don't know!  It depends where in the world you're from I suppose.  

In my experience having an actual person attempt to bring down the VLE is unlikely, but it is definitely far more likely to happen than burning the college down.  A simple "lets all hold down the refresh button and see what happens" is all it takes and would usually have less severe repercussions for the students involved, if any at all.

We are dealing with the law of diminishing returns here though, e.g. putting lots of time and money into making sure the server stands up against an event that not likely to happen often.  But it does illustrate the point that although Moodle is open source and costs nothing to download it is not exactly "free" unless you happen to have powerful servers and/or lots of expertise to hand 24/7. You could hand it all off to an ISP by using a managed server solution, but that's not exactly cheap either.

Average of ratings: -
In reply to Dale Davies

Re: A 15 year old student put Moodle down

by Marcus Green -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers

"it is not exactly "free" unless you happen to have powerful servers and/or lots of expertise to hand 24/7"

But big servers and lots of experience are far from free. Part of the problem is in the ambiguity of the word Free in the English language, it can mean free of financial cost or free as in freedom.

(Imagine Nelson Mandala had not been free but simply had a 10% discount).

In my experience the Liberty offered by free software is at least as important as the lack of financial cost.



Average of ratings: Useful (1)
In reply to Dale Davies

Re: A 15 year old student put Moodle down

by Richard Oelmann -
Picture of Core developers Picture of Plugin developers Picture of Testers
Not sure it illustrates any point at all about Moodle being free/open source, Dale - you try hosting one of the commecial alternatives such as Bb on those servers and I will guarantee the same/similar results to a DOS attack, but you'll have also paid a massive licence cost on top of your server infrastructure.
You are right though - Moodle is not a zero cost alternative to run at any significant scale, but neither is any other system.
The point I think it makes is that if any system is on the web, it needs to be properly installed and provisioned, including hardware and support (and with consideration of the size/risks involved in any particular instance).


Average of ratings: Useful (3)
In reply to Richard Oelmann

Re: A 15 year old student put Moodle down

by Dale Davies -

I perhaps should not have referenced "open source" as such.  I was actually trying to make your second and third points. 

You made them much more clearly!

Average of ratings: -
In reply to Dale Davies

Re: A 15 year old student put Moodle down

by Matt Bury -
Picture of Plugin developers

Hi Dale,

Re: "You could hand it all off to an ISP by using a managed server solution, but that's not exactly cheap either." -- What's your definition of cheap?

I've been looking around for a better hosting provider for my demo Moodle because most shared hosting services don't support Moodle at all or not very well.

Dedicated servers, managed 24/7 by providers start at around $800 per year. That'll easily be sufficient for a few hundred students at a time or, if usage is light, many more. Many ISPs will set up your server and even install software and web apps for you for a minimal charge (They usually have images or installer scripts already set up).

You can get virtual private servers (VPS) set up with Moodle and ready to run (which is free) on AWS that'll cost as little as $200 per year for a micro-instance (good for experimental, personal demo, and small-scale teaching sites). I think a full instance works out at around $1,000 per year. The cost is per hour of what you actually use and no more (you don't pay for high latency like you do with dedicated servers).

Compare that to what it costs to buy content or hire a content developer and hire and train admin and teaching staff and the cost is hosting and support is negligible. I frequently see sums of $10,000 - $50,000 being thrown around by proprietary LMS service providers, and how likely do you think it'll be that you'll have a lot less choice, fewer options, not as many features, and have to pay through the nose to have anything done on top of that?

Prices of VPS' seem to have come down significantly in the last couple of years. Is there a price war on between hosting providers? Anyway, it looks like now's a good time to shop around... something you couldn't do with most proprietary LMS providers.

Average of ratings: -
In reply to Matt Bury

Re: A 15 year old student put Moodle down

by Matt Bury -
Picture of Plugin developers

Another point worth making that often gets left out of the equation is that when you use a FOSS project like Moodle, you can join a huge community of practice. You have access to the good will, generosity, and expertise of thousands of other Moodlers with a diverse range of interests, skills, knowledge, specialisations, etc. Help often comes from multiple views with multiple solutions that you can choose from and you can also learn more about using Moodle by helping others; so called "writing for thinking" encourages you to organise your knowledge and thoughts on a given topic thereby increasing your understanding of it and maybe doing some additional research to fill in some missing details, all while helping a fellow Moodler solve a problem/find a solution. smile

Average of ratings: -
In reply to Matt Bury

Re: A 15 year old student put Moodle down

by Fred Roller -

A very good solution that is inexpensive is BlueHost.  I have used them for years.  $180 covers your domain for three years and you get lots with that price.  As an IT I love that you essentially get a sever that is yours to do with as you like; especially activating ssh.  Moodle is a one click install and easily updated, backed up, and maintained.  Behind the scenes you have tools to secure as you see fit.  Another $100 will get your ssl certificates for https.  If the shared server starts to bog then you can upgrade to a package that fits your needs.  If your organization has another division that needs a web presence then just purchase the domain name and install, say, Wordpress.  You get the idea.  I don't know if it is faux pas to mention a host by name but I work with non-profits on nearly non-existant budgets to develop domains for there projects.  www.womenofconnections.org is a good example.  Lady is struggling to keep her house but she doesn't have monthly fees for the site so her work continues.  I am not a reseller just found this fits alot of solutions similar to yours. - HIH

Average of ratings: -
In reply to Matt Bury

Re: A 15 year old student put Moodle down

by Dale Davies -

Would an $800 per year dedicated server be sufficiently reliable in a large college with students using it daily, without some serious time and effort spent managing the configuration?

Lets look at a descent VPS solution, say £1000 a year. 

You'll need a member of staff to dedicate some time to setting it up, installing Moodle, working out if it can be connected to the domain (via LDAP for automatic authentication), MIS system (for automatic enrolment), then coming up with a plan for how the system will be structured and rolled out across the organisation. If you have a member of staff who has prior knowledge of Moodle and experience in this field then great, otherwise you'll need to hire.

Then you need to look at customising your theme a little bit because it doesn't quite fit in and feedback from staff and students is not great, it doesn't do x, y and z when <insert name here> proprietary system does.  So again you've got to pay for staff time,

Maybe you just racked up another £5-8k in IT staff time? So we're totalling maybe £8k ($13k) so far.

Then, as in the OP's original post, things start to break as you scale up and someone has to work out how to fix it. Now everyone in the organisation is suggesting that you ditch Moodle and move to a proprietary LMS.

This is of course not a failing of Moodle in any way, my point was just (as Richard put it above) that Moodle is not a zero cost alternative to run at any significant scale. 

I see so many people blaming Moodle for being insecure, patchy, having poor performance and being buggy, when it is actually that the installation has not been properly planned and/or is under resourced.

Average of ratings: -
In reply to Dale Davies

Re: A 15 year old student put Moodle down

by Dave Perry -
Picture of Testers

We self host internally, and the biggest problem we have is our internal/DMZ network structure. It's a horribly complex beast, and usually quicker to access moodle off site! Having said that, it has got a lot better since we installed a speed tester on it and discovered the server had a driver installed that twinned the network cards on it (which was doing more harm than good) and rejigged it. And IT have installed faster links between certain infrastructure points.

In terms of manpower here we have the equivalent of 2 FTE support staff (I'm one, the other is split between two people hours wise) and where we need expert Windows/technical help (or co-operation) IT are usually willing to chip in where they can (and actually paid for our server, as they were convinced all the speed complaints were due to the age of the old server - which wasn't quite the case).

 

Average of ratings: -