Hardware and performance

Moodle-data storage advice needed for SAN

 
 
You either love it or you hate it
Moodle-data storage advice needed for SAN
 

We are setting up a Load Balanced 2 web server node environment and I have been told that the organisations SAN doesnt support NFS.

I was simply going to mount a Ubuntu share to an NFS share!

Are there any alternatives, Samba? or some other way

Thanks

 
Average of ratings: -
You either love it or you hate it
Re: Moodle-data storage advice needed for SAN
 

I have some more information on this. The organisation uses a iscsi SANs which will not allow or support NFS.  All I need is a share on a on the SAN for our Moodle data folder, however I think that myself and the Windows Network team are talking different languages.

In the past I have requested a storage area on the network and mounted this area to the web server using an NFS mount

thanks in advance for any help

 
Average of ratings: -
Picture of Emma Richardson
Re: Moodle-data storage advice needed for SAN
Group Particularly helpful Moodlers

I use a share on a SAN and mount it with samba from a linux box.  Seems to work perfectly.

 
Average of ratings:Useful (1)
You either love it or you hate it
Re: Moodle-data storage advice needed for SAN
 

Thanks for that, yes this is my thoughts. A SAN doesnt have to support NFS or SAMBA or CIFS. All I need is a share on the SAN and then I can use a variety of methods to mount that share on our Ubuntu machines

I am now being told that the SAN really presents itself as a server connected to the SAN rather than a share.  Well I dont care if it presents itself as a dustbin as long as I get a shared folder on it. smile

Thanks for the heads up on this

Albert

 
Average of ratings: -
Picture of jason everling
Re: Moodle-data storage advice needed for SAN
Group Particularly helpful Moodlers

Since you say it is an iSCSI SAN just have them create a LUN for you and mount using the iSCSI in Ubuntu..You will need to install the iSCSI packages then provide your IT dept with your iSCSI id then the volume will show up in Ubuntu then you can mount it and your web servers. iSCSI in Ubuntu, go down to the Initiator section since you do not need to setup the target; http://www.howtoforge.com/using-iscsi-on-ubuntu-10.04-initiator-and-target .

You can then use OCFS2 as the clustered file system for all the servers, For Ubuntu great guide here http://www.hbyconsultancy.com/blog/shared-ocfs2-partition-on-ubuntu-server-10-04-x64.html

Jason

 
Average of ratings:Useful (3)
You either love it or you hate it
Re: Moodle-data storage advice needed for SAN
 

Wonderful many many thanks

Great

Albert

 
Average of ratings: -
You either love it or you hate it
Re: Moodle-data storage advice needed for SAN
 

Ummmm

I have now been told if its a LUN then its not Iscsi rather a fibre channel?

Some further digging me thinks

 

Cheers

 
Average of ratings: -
Picture of Peter Dorman
Re: Moodle-data storage advice needed for SAN
 

If it's a LUN served of fibre channel then it appears as a local scsi disk on your Linux server. 

You might need to do something to rescan for the new disk then run a partitioning tool such as gparted to format the disk & mount it. 

I don't know how you scan for disks on ubuntu. The command varies between distributions.

 
Average of ratings:Useful (1)
You either love it or you hate it
Re: Moodle-data storage advice needed for SAN
 

I think I have two choices here.  Firstly I could get this iscsi Lun working and stick the Moodle data folder in it. And then have two sets of application code bases on my two web servers (these can be synchronized using a cron based rsync command periodically.

Or I could use OCFS2 as a clustered file system for both the Moodle-data and the Moodle application code base.

Although the later sounds rather tricky.

Too many choices

 
Average of ratings: -
You either love it or you hate it
Re: Moodle-data storage advice needed for SAN
 

Me again.  Is it OK to connect two web servers moodle-data folders to a single ISCSI Lun?

 

Cheers

 
Average of ratings: -
Picture of J S
Re: Moodle-data storage advice needed for SAN
 
No.
 
Average of ratings: -
Picture of jason everling
Re: Moodle-data storage advice needed for SAN
Group Particularly helpful Moodlers

I don't know where that last response came from, Even if it is Fibre Channel you can still mount on both web servers at the same time. Just use OCFS2 as the shared cluster file system. It will work perfectly fine and what it is meant for, concurrent read/write to a single lun shared by many servers. Up to 32 I believe. Here is full http://www.oracle.com/us/technologies/linux/025995.htm

You DO NOT put the /moodle core on the shared OCFS2 partition, only /moodledata. We use a main repo for the Moodle core files that is updated via git then rsync'd to the rest of the web servers. It rarely changes.

Also since it is a shared LUN on OCFS2 you DO NOT need to worry about putting MUC ( Moodle Universal Cache ) into memcache. Just use the standard disk cache in /moodledata. It will be as fast as you disks can spin.

Jason

 
Average of ratings:Useful (3)
You either love it or you hate it
Re: Moodle-data storage advice needed for SAN
 

OK this is still ongoing.  Apparently a fibre channel  LUN isnt available so my options are getting thin. Its an ISCSI San but isnt on a linux network but a windows network, does this make any difference?

I have details on how to create an iscsi lvm target on a sans but LVM is a linux technology and i am sure that the network team use some form of windows software to create iscsi targets.  Does this make a difference/

 

Cheers

 
Average of ratings: -
Picture of jason everling
Re: Moodle-data storage advice needed for SAN
Group Particularly helpful Moodlers

No it doesn't matter, can be any type of ISCSI Target, Your Linux OS will not care which it runs on.

All you need to do is give them your iSCSI iqn name like iqn.some.host.or.numbers to them to allow your host to access that LUN then add the IP Address of the iSCSI Target to your config like so;

sudo iscsiadm -m discovery -t st -p  10.10.10.1

You would then see the available LUNS, 10.10.10.1 would be your targets IP Address.

You then follow the rest of the initiator setup here;

http://www.howtoforge.com/using-iscsi-on-ubuntu-10.04-initiator-and-target

Jason

 
Average of ratings:Useful (1)
You either love it or you hate it
Re: Moodle-data storage advice needed for SAN
 

Hi Jason

Many Many thanks for your help on this matter, its the first time that I havent just been able to request an NFS share to do the job.  I have found out this morning that the reason I cant discover the SAN using iSCSI initator is the fact that that the SAN is on a protected, isolated network.

So the only way of accessing a volume is to create a VM in the ESXi and mount the volume from there!

Which means another Ubuntu VM in the mix, which is just another VM that could crash.  So in terms of reliability is not as good as a share or accessing an ISCSI LUN, and it also used valuable VW Sphere resources, processor and RAM.  Too me this daft but may be the only way forward

I am currently looking at an Openfiler which may be the appliance that I need.  However I think it is something that the network team would need to install and manage so im not sure that they would go for this

Many thanks indeed

 
Average of ratings: -
Picture of jason everling
Re: Moodle-data storage advice needed for SAN
Group Particularly helpful Moodlers

Are your servers that you are going to host Moodle also on vSphere? First time that you had mentioned it. If so than you do not need to worry about a SAN.

Create 2 New VMs for Web Servers with basic 20GB Disks for the OS then add a new hard disk to one of the VMs in vSphere BUT must be configured correctly for sharing across multiple VMs.

So, Edit Settings > Add > Hard Disk > Create a new virtual disk > Choose size and choose "Thick Provision Eager Zeroed" > under "Virtual Device Node" choose a "SCSI (1:0)" then finish.

After the disk has been zero'd out go back into "Edit Settings" > click the new SCSI controller 1 and choose "Physical" Virtual disks can be shared between virtual machines on any server.

Edit the settings for the 2nd VM and choose add hard disk, choose existing, browse to where you created the disk for VM 1 , make sure you assign it the same SCSI (1:0).

You now have a virtual hard disk shared between your machines that you can then setup OCFS2 on.

If not, let me know what is available to you, like your setup so I can get a clear picture.

Jason

 
Average of ratings:Useful (2)
You either love it or you hate it
Re: Moodle-data storage advice needed for SAN
 
Well thanks for that and yes they VMs on V sphere, sorry I thought I had mentioned that before. I have a lot of experience with multi node set-ups, physical and virtual, I have no experience with using SANs as the Data Root. I have always been able to have an NFS share somewhere on the network and mounted this share to the two web servers. I assume that because this type of shared disk is ISCSI then I have to set up OCFS2 clustered file system. This is of course different than an NFS share where one does not have set up OCFS2 Cheers
 
Average of ratings: -
You either love it or you hate it
Re: Moodle-data storage advice needed for SAN
 

The more I think about this, the more I think that maybe the easiest option here would be to create 2 Vm web servers, have them as you suggested with 20GB OS disks and have one with a second disk for the moodledata folder and just share this disk using NFS. Would this work? I mean in terms of data consistency or is it simply that using any shared drives on VSphere would require a clustered file system.

 

Cheers

 
Average of ratings: -
Picture of jason everling
Re: Moodle-data storage advice needed for SAN
Group Particularly helpful Moodlers

The easiest route since you are on Vsphere is a shared disk. You don't really want to go NFS route, you would degrade your I/O performance especially for the new versions of Moodle. Just as I mentioned before setup a 2nd disk to be shared between the two and setup OCFS2. It really is a quick and painless setup. We use it throughout for both Windows and Linux application servers. If you go NFS you would then have to setup a shared cache using memcache or similar requiring allot more work and configuration.

You can use this guide which is targeted for Windows Failover on Vsphere but is the same process for setting up the shared disk for any OS. http://pubs.vmware.com/vsphere-51/topic/com.vmware.ICbase/PDF/vsphere-esxi-vcenter-server-51-setup-mscs.pdf

Jason

 
Average of ratings:Useful (2)
You either love it or you hate it
Re: Moodle-data storage advice needed for SAN
 

Thanks for all your help but I think there is a little confusion here as the VM team have told me that this can only be achieved using a fibre channel, we only have ISCSI SAN.

And the doc in your link specifically mentions that "

Disks that are shared between nodes. A shared disk is required as a quorum disk. In a cluster of virtual
machines across physical hosts, the shared disk must be on a Fibre Channel (FC) SAN."

I think NFS is the only way forward for this sad

 

cheers for assistance, most helpful

 
Average of ratings: -
Picture of Visvanath Ratnaweera
Re: Moodle-data storage advice needed for SAN
Group Particularly helpful Moodlers
Hi Albert

A dumb question: What is the aim of this exercise? To run Moodle in a load balanced setup or to get the maximum Moodle throughput of the available hardware?
 
Average of ratings: -
You either love it or you hate it
Re: Moodle-data storage advice needed for SAN
 

Load balanced setup with a shared moodledata root.   Always either had an NFS or GFS share provided for the moodledata root.

I think I might just ask for a 4th VM with a large disk, install Ubuntu on it and configure it as an NFS server and put the moodle data root on that.  I was just trying to avoid having the 4th machine as this could go down so would be less reliable.

2 web servers

1 MySQL DB

1 NFS server

 

Cheers

 
Average of ratings: -
You either love it or you hate it
Re: Moodle-data storage advice needed for SAN
 

More thought!

I dont think that will work either as I am still trying to share an ISCSI drive with two web servers! Doh smile

Albert

 
Average of ratings: -
Picture of jason everling
Re: Moodle-data storage advice needed for SAN
Group Particularly helpful Moodlers

Even through the DOCS say that, you can still use it! We DO NOT have Fibre Chanel and still use a virtual shared disk in Vsphere. Our Vsphere environment is mainly a DAS Switched Environment with some 10GB ISCSI and it works just fine.

Here is a link to setting up OCFS2 using ISCSI to connect multiple hosts for concurrent read/write.

http://knowledgelayer.softlayer.com/procedure/connect-multiple-servers-single-iscsi-lun

Thanks,

Jason

 
Average of ratings:Useful (1)
You either love it or you hate it
Re: Moodle-data storage advice needed for SAN
 

Hi

Yes i have been doing some research and although both microsoft and VMware say no your solution does work and has done for other people as well. I already have OCFS2 set up on my servers, so just need a shared ISSCI disk configured and a target given to me smile

Many Many thanks for your help

Its much appreciated

Albert

 
Average of ratings: -
You either love it or you hate it
Re: Moodle-data storage advice needed for SAN
 

Well we have tried to do this but we get an error:

An error was received from the ESX host while powering on VM MWeb2.
Cannot open the disk '/vmfs/volumes/MWeb1/MWeb1.vmdk' or one of the snapshot disks it depends on.
Failed to lock the file

 

Does the second VM need to have the SCSCI controller 1 set to "Physical, virtual disks can be shared between virtual machines on any server"

Cheers for everyones help

 
Average of ratings: -
You either love it or you hate it
Re: Moodle-data storage advice needed for SAN
 

To answer my own question Yes IT Does

Thanks for everything Jason and Co

 

Albert

 
Average of ratings: -
You either love it or you hate it
Re: Moodle-data storage advice needed for SAN
 

Ok Managed to get the shared disk setup on VSphere without any errors.  With an additional step added to Jasons instructions i.e. making sure both VMs ISCSI controllers have set to share disks.

Both now Debian machines can see the drive if i fdisk -l

The drive is formatted to OCFS2 and OCFS2 and O2CB both start and work with no errors.

I can mount the first or the second machine to this drive but cannot mount them both as I get the error:

mount.ocfs2: I/O error on channel while trying to join the group

What have a done wrong here? The only issue i can think off is that following jasons instructions further up this post didnt work.  The second VM throw a error on startup stating that there was a lock error.

So I altered the second VM's ISCSI contoller settings to also allow the sharing of disks and this worked with no errors in VSphere. Although I dont understand why as its both VM's sharing the same disk on the first VM. I do not share disks on the second VM.

I have folloed these instructions :http://www.hbyconsultancy.com/blog/shared-ocfs2-partition-on-ubuntu-server-10-04-x64.html

However there is no mention of ISCSI targets? Do I need this step as well?

Cheers

 
Average of ratings: -
You either love it or you hate it
Re: Moodle-data storage advice needed for SAN
 

It works !!!!

The cluster.conf has be the same for each server, this sounds obvious but the only difference that i had was each server had its own cluster details at the top of the cluster.conf and the second server under it.  But no, just copy the first servers conf and paste it into the second

 

Thanks

Going to write a blog on this one

 
Average of ratings:Useful (1)
Picture of jason everling
Re: Moodle-data storage advice needed for SAN
Group Particularly helpful Moodlers

Yay! Im glad you have a cluster up and running now. I have done the shared disk so many times in Vsphere setting the second disk to shared must have slipped past me.

We actually use the shared disk on non-clustered Vsphere hosts also, just as long as each ESXi host has access to the same LUN.

Jason

 
Average of ratings:Useful (1)