Moving moodle.org to kubernetes

Moving moodle.org to kubernetes

by Eduard Cercós -
Number of replies: 13
Picture of Moodle HQ

Hi everybody!


On Thursday we migrated this site moodle.org to a new hosting environment based on kubernetes+docker. We had some reasons to do so:

- Current site was hosted on an standalone instance that prevents it from scale horizontally.
- We hitted a bug/configuration error that has been a pain for us for the last 3 months.
- Need to provide more availability and better performance when possible
- Standardize processes with other sites
- Reduce downtime (scheduled or not)

So we prepared the new environment that would give us
- more capacity, dynamically increase when needed
- reduced downtime
- analyze Moodle to improve its integration with docker/kubernetes

New installation is based on a variable number of pods running nginx + php-fpm (php 7.3) as frontend with Redis as cache, session and lock backend. We have two redis instances to keep isolated session and caching/locking.

We're continuously working on improving this install, so any feedback would be appreciated.

Average of ratings: Useful (10)
In reply to Eduard Cercós

Re: Moving moodle.org to kubernetes

by Tasos Koutoumanos -
I'm also very interested in how this is turning out? What are the benefits and what are the ... pains!
Average of ratings: -
In reply to Eduard Cercós

Re: Moving moodle.org to kubernetes

by Jeff White -
Would it be possible to release a paper on how the system is built out and most of the relevant settings? I would love to learn more about building out Moodle in Kubernetes.
Average of ratings: -
In reply to Eduard Cercós

Re: Moving moodle.org to kubernetes

by Gabriel Lauter -

Hello everybody... maybe it can help...

https://wiki.geant.org/display/NE/Managing+and+running+large+scale+Moodle+system%2C+for+IT

There you will find a presentation made by Eduard Cercós about this topic.

Average of ratings: Useful (4)
In reply to Gabriel Lauter

Re: Moving moodle.org to kubernetes

by Alain Raap -
Picture of Particularly helpful Moodlers
Thanks for sharing this interesting information and the presentations! Will there be organized a next event? I see several flavours with / without Cloud solutions and also different solutions with docker / kubernetes and AWS, very interesting.
Average of ratings: -
In reply to Eduard Cercós

Re: Moving moodle.org to kubernetes

by Eduard Cercós -
Picture of Moodle HQ
Hi everyboy and sorry for the really late reply, for some reason I missed all the updates sad
I would like to thank all your comments and linking to my presentation. Although you may have work it out, I'd try to answer your questions to help any one else coming to this forum.

- After over a year running in kubernetes we're very happy with this changes. As we stated main benefits are stability and scalability. The upgrades are integrated in our CI/CD pipelines and any change without database upgrading is deployed without downtime.
- Major pains are external to kubernetes and shared with any large scale installation. One we hit was the interference with other services (solved using resource limiting) and another one was the usage of some storage classes (like EBS) that can take some time to attach/detach (solved using shared filesystems like EFS).
- We recently included a ReadinessProbe which prevents faulting pods to be used. It has reduced the downtime in stress situations and helped to solve some DoS attacks we faced recently. But it may lead to some 'There something wrong with internet' temporary errors. We're still working on it
- What we really love about k8s is how easily you can add services with very little setup. For example we use a standard image of Redis without any special changes. Its pod has a memory limit so when it's passed, the instance is silently restarted and all its content flushed. This is just for the cache instance, as session instance could lead to some errors while restarting (as it happened recently)
- Related with https://tracker.moodle.org/browse/MDL-63770, we have no problem there as external and internal ports where all standard (443 & 80) All intermediate ports used inside the cluster by its elements (proxies, services, etc) are never seen by Moodle. Also, k8s uses a virtual network so you can have multiple services using the same port.

Hope it helps! If any other question I'll try to answer as soon as possible
Average of ratings: Useful (4)
In reply to Eduard Cercós

Re: Moving moodle.org to kubernetes

by Sachin maurya -

Hii Eduard, thank you for all the comment . It was really helpful.

I wanted to know more about the docker images that you are using . As I already say you video and presentation and there you have mention that you are building your own image and not using the bitnami one .We were using  bitnami image and it works fine but when we are going in the HA(high availability ) mode and trying to run multiple pods  we faced some issue with moodle version 3.9  and older(this was the issue) , so in the issue you can see that there was some problem with the session directory when we try to launch moodle in HA mode by specifying ENV variable   MOODLE_SKIP_BOOTSTRAP="yes".

So that I will skip the population of database as it was already population when we launch the single moodle instances.

Now bitnami image version 3.10 and 3.11 works fine as it was fix after that issue was raised, but as we are looking to use other older version (2.7 +) so we will not be able to support HA using bitnami image . We need to create our own image (i might be wrong here).

Can you shared some light how can we do that ?

Also I did check out this 2 issue:

https://moodle.org/mod/forum/discuss.php?d=402632

https://moodle.org/mod/forum/discuss.php?d=402425

Average of ratings: -
In reply to Sachin maurya

Re: Moving moodle.org to kubernetes

by Eduard Cercós -
Picture of Moodle HQ
Hi Sachin,
You can create a docker image easily following any Docker file, or docker-compose, etc. If you never create one before I'd recommend to take a look to official Docker documentation here (https://docs.docker.com/get-started/). Here it is one Dockerfile similar to the ones we use to create a moodle site:
FROM ubuntu/nginx:latest
COPY /build/setup.sh /tmp/setup.sh
COPY /build/scripts /opt/scripts/
COPY /build/deploy_key.pri /root/.ssh/id_rsa
COPY /build/cron.json /opt/crons/cron.json
COPY /build/default.conf /etc/nginx/includes/default.conf
RUN bash /tmp/setup.sh
VOLUME ["/opt/app"]
VOLUME ["/opt/data"]
VOLUME ["/opt/logs"]
EXPOSE 80
CMD ["/bin/bash", "/opt/scripts/run.sh"]

Two key steps:
- setup.sh script contains anything you need to add to the base image (eg, packages, moodle code, config files, etc)
- run.sh script contains the entry point of the container. It wakes up the web server daemon, etc

Other recommendations:
- Make sure you have mounted the /opt/data filesystem in a shared volume
- Make sure the code directory (/opt/app here) is in a local volume. We usually add the code to the image so it is loaded in a local docker volume.
Average of ratings: -
In reply to Eduard Cercós

Re: Moving moodle.org to kubernetes

by Karim Ayari -
Hello Eduard,

I'm currently working on a custom bitnami moodle image. why bitnami? because they have all the scripts needed for deployment and I don't really want to reinvent the wheel smile and they have a helm chart that I slightly modifiedand deploy with argocd.
However, I have questions regarding your kubernetes deployment:

How do you handle the upgrade? I would like to be able to do it by updating the image and not manually inside the container and especially how do you manage the reinstallation of the plugins?

thank you for your reply smile
Karim
Average of ratings: -
In reply to Karim Ayari

Re: Moving moodle.org to kubernetes

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi

It is time somebody carefully analyze the Bitnami Docker image for Moodle. It pops up sporadically in the forums and do not get useful answers, because it is so much different from the native installation of Moodle on LAMP and as a result the advice the regulars here do not apply.

In a recent workshop the Moodle Bitnami Docker and the plain vanilla LAMP installation on VPS came up site-by-side. Yes, the Docker image was installed and running in minutes. But as the workshop progressed, as people started upgrading, installing plug-ins, taking backups, etc., on the Bitname-Docker front there were always discussions about Docker and Bitnami whereas on the VPS front, there was nothing to discuss. Halfway though we swapped the Bitnami with a hand-made Docker image starting from Ubuntu Linux Server, but it came too late to undergo the same scrutiny.

At another occasion I came across a sophisticated clustered Docker/Kubernetes/Galera/.. environment for Moodle deployment. The sysops have gone in to great lengths on _deployment_, which means for them spawn a Moodle instance and forget it. They are obsessed with it. What they forgot was Moodle is a tiger by the tail. It can not be forgotten, it makes the users to make the noise. In plain text, they haven't looked in to the simplest thing about _running_ Moodle, like upgrading, "downgrading" ;(, the moodledata, like additional plug-ins, like backups, restore,... BTW, I don't even know whether the Docker images they took were Bitnami or something else, that was highly secret.

What we need is full documentation, the counterpart of, say https://docs.moodle.org/en/Installing_Moodle_on_Debian_based_distributions. I know, the Docker philosophy is one click and it runs. You don't have to look in to it. But if one doesn't have a clue, What and Where, he'll fall flat at the first user problem.

P.S. @Moderator: I propose to move this whole discussion to Installing and Upgrading help forum. This has nothing to do with performance. (I know, there are claims that this system is efficient and scalable. But this thread has no evidence, no load tests nor numbers supporting it. It is still about (easily) _installing_ Moodle.)
Average of ratings: Useful (1)
In reply to Visvanath Ratnaweera

Re: Moving moodle.org to kubernetes

by Karim Ayari -
I agree with you, you shouldn't do kubernetes because it's fashionable. I try to imagine a scenario in order to be able to offer a flexible and robust hosting service. I am in a test phase while remaining pragmatic 😉
Average of ratings: -
In reply to Visvanath Ratnaweera

Re: Moving moodle.org to kubernetes

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
I use Docker with Moodle for my development and testing work (not in production). I have a bunch of Docker Compose scripts and, yes, it makes Moodle a "one click" installation. But it's *my* one-click installation.

That's the problem - everybody who creates Docker (or Bitnami or whatever) installations for Moodle is building it for their own requirements - or at least some specific requirement - and therein lies the problem. So if you use my Docker install (help yourself) and it all goes wrong then you need to ask me, not these forums. And don't bother, because I'm too lazy to answer big grin

For full disclosure... https://github.com/thepurpleblob/docker_moodle11
(there's a branch for Moodle 4.0/PHP 8)
Average of ratings: Useful (3)
In reply to Karim Ayari

Re: Moving moodle.org to kubernetes

by Eduard Cercós -
Picture of Moodle HQ
Hi Karim,

I wouldn't say using your own docker image is reinventing the wheel as using a customized SO installation isn't, is it? I found really useful adapt your containers to your needs and simplify the overall maintenance and upgrading process.

As we add the code to the image, we have a pipeline process that once the code is upgraded (including the desired plugins) the docker image is rebuilt. Once deployed, we launch the upgrade process (either manually or automatically) to finish the changes in the database.
Average of ratings: -