Lonewolff
Posts: 143
Joined: Fri Dec 28, 2012 11:13 pm

How to boot into RAID array (mdadm)

Wed Jan 16, 2013 11:03 pm

Hi Guys,

I am wondering how I can tell the kernel to boot into a small RAID array?

I have had success in tellinge the kernel to boot from a USB stick (which is awesome), but I want to take it to the next level and into a RAID array (/dev/md1).

The only thing stopping this I guess is having the mdadm service running prior to when it calls up /dev/md1 in cmdline.txt.

Any ideas on how I can get mdadm to run as part of the kernel or have tthe kernel call this up on a separate partition?

Any help would be awesome 8-)
Loving my peice of the Pi

gragib
Posts: 99
Joined: Sat Dec 22, 2012 8:09 pm
Location: Calgary, AB, Canada

Re: How to boot into RAID array (mdadm)

Thu Jan 17, 2013 6:41 am

The Raspberry Pi does not have any connectors for directly connecting to hard drives, only through USB or over a network connection. Are you thinking of RAIDing USB drives?

Lonewolff
Posts: 143
Joined: Fri Dec 28, 2012 11:13 pm

Re: How to boot into RAID array (mdadm)

Thu Jan 17, 2013 7:27 am

That is exactly it. ;)

I have no problems making the RAID array. I have the array working right now. But, what I want is to be able to shift the OS load over to the array.

I have made the bulk OS load from the USB devices recently with no problem. But, to replicate this for the RAID array the mdadm service must be running. So, we can't move the OS partition to the raid array yet as the kernel and stuff hasn't been told to use mdadm yet.

That is the dilema I have. ;)
Loving my peice of the Pi

gragib
Posts: 99
Joined: Sat Dec 22, 2012 8:09 pm
Location: Calgary, AB, Canada

Re: How to boot into RAID array (mdadm)

Thu Jan 17, 2013 7:36 am

I'm guessing you want RAID for resiliency/disaster recovery and not for speeding up drive access. Am I right?

For many USB drives, the speed bottleneck is in the low speed of USB 2.0, not how fast the platter can spit out bits. So RAID will be of little use there if you want RAID for speeding up access.

As far as setting up the OS on RAID goes, you could flash the OS to SD, boot from SD, copy the image file to SD, set up your RAID and then flash the image file to the RAID from within your SD-booted OS. Then on the SD card, modify cmdline.txt so that it boots using the RAID device and not the OS on the SD card. AFAIK the Raspberry Pi is hard-coded to boot from SD, so you'll need the bootloader on the SD card.

Lonewolff
Posts: 143
Joined: Fri Dec 28, 2012 11:13 pm

Re: How to boot into RAID array (mdadm)

Thu Jan 17, 2013 7:50 am

It is actually for speed and I am aware of all I/O bottlenecks. :lol:

I have done extensive testing lately (on the Pi itself) and have made a mix up of devices and had excellent results.

Previously, the SD card was providing read speeds of ~18 MB/s. I managed to make an array that provides a consisitent ~55 MB/s. 8-)

I know the kernel partition needs to stay on the SD card and I am fine with that. I just want to be able to offload the OS partition to the RAID array.

For three times the transfer speeds, it has got to be worth a shot doesn't it? ;)
Loving my peice of the Pi

gragib
Posts: 99
Joined: Sat Dec 22, 2012 8:09 pm
Location: Calgary, AB, Canada

Re: How to boot into RAID array (mdadm)

Thu Jan 17, 2013 7:53 am

Have you done similar tests with a good USB flash drive with high-quality NAND chips? How much storage space are you looking for?

Lonewolff
Posts: 143
Joined: Fri Dec 28, 2012 11:13 pm

Re: How to boot into RAID array (mdadm)

Thu Jan 17, 2013 8:18 am

Yep, I even tested with a WD velociraptor and still tops out at just over 20 MB/s because of the USB 2.0 transfer cap.

At this stage, size doesn't matter. I am more focused on getting the OS to boot from it (kernel and 1st partition of the SD card aside)
Loving my peice of the Pi

Lonewolff
Posts: 143
Joined: Fri Dec 28, 2012 11:13 pm

Re: How to boot into RAID array (mdadm)

Thu Jan 17, 2013 9:49 am

I did it 8-)

I'll do some more tests and maybe write up a how to. :D
Loving my peice of the Pi

gf42
Posts: 22
Joined: Fri Jul 20, 2012 10:31 am
Location: Australia

Re: How to boot into RAID array (mdadm)

Fri Jan 18, 2013 7:31 am

How did you do it in the end?

User avatar
Jim Manley
Posts: 1593
Joined: Thu Feb 23, 2012 8:41 pm
Location: SillyCon Valley, California, and Powell, Wyoming, USA, plus The Universe
Contact: Website

Re: How to boot into RAID array (mdadm)

Fri Jan 18, 2013 7:53 am

Lonewolff wrote:I know the kernel partition needs to stay on the SD card and I am fine with that. I just want to be able to offload the OS partition to the RAID array.
Only the /boot directory needs to stay on the SD card, and once control has been transferred to whichever USB storage device is identified as the root path in /boot/cmdline.txt (i.e., not root=/dev/mmcblk0p2, the SD card, but rather root=/dev/sda1, etc.), you can remove the SD card during operation if you want (it would have to be reinserted to perform a reboot, though).
The best things in life aren't things ... but, a Pi comes pretty darned close! :D
"Education is not the filling of a pail, but the lighting of a fire." -- W.B. Yeats
In theory, theory & practice are the same - in practice, they aren't!!!

Lonewolff
Posts: 143
Joined: Fri Dec 28, 2012 11:13 pm

Re: How to boot into RAID array (mdadm)

Fri Jan 18, 2013 8:13 am

Yeah, but what if the device is a raid array controlled by mdadm? mdadm would have to be running before cmdline.txt is called up.

So, the reference to the /dev/md1 wont exist yet.
Loving my peice of the Pi

Lonewolff
Posts: 143
Joined: Fri Dec 28, 2012 11:13 pm

Re: How to boot into RAID array (mdadm)

Sat Jan 19, 2013 4:31 am

Is there a way to get cmdline.txt (or some other method) to get mdadm running before the USB device detect stage? If so, I believe this would solve the all of the problems.

Any help or tips would be awesome as I am running out of ideas on this one.
Loving my peice of the Pi

User avatar
Jim Manley
Posts: 1593
Joined: Thu Feb 23, 2012 8:41 pm
Location: SillyCon Valley, California, and Powell, Wyoming, USA, plus The Universe
Contact: Website

Re: How to boot into RAID array (mdadm)

Sat Jan 19, 2013 5:23 am

Lonewolff wrote:Yeah, but what if the device is a raid array controlled by mdadm? mdadm would have to be running before cmdline.txt is called up. So, the reference to the /dev/md1 wont exist yet.
I'll have to wait until I get home to my Pi and notes later tonight, but my (admittedly incomplete) understanding is that you can't do anything until the configuration of the GPU blob is completed through cmdline.txt and config.txt, no matter what OS or even bare metal code you're booting. Some tend to not know or forget that the Pi is really a relatively powerful GPU (for the electricity consumed) that boots first and foremost and only deigns to bring up the ARM CPU as an afterthought to act as an I/O traffic cop for the USB and Ethernet ports. A lot of people are beating themselves over the head trying to make the CPU do a bunch of stuff it was never meant to do and are then disappointed with the results. There's no apparent reason why a RAID array that's connected via USB (that's not available until after the GPU boots) can, much less needs, to have mdadm running first (but, I need to refresh my wetware cache on the precise sequence that's executed during boot through the OS completing its configuration, loading, and execution). Maybe the two-step GPU and then CPU boot process is confusing either/both of us.

In any case, I want to try this out myself - 55 MB/sec ain't nothing to sneeze at, especially given the normally bare-board state of my Pii (insert "EWWWWW!!!" here). Thankfully, I'll have time this long weekend to work on it as Monday is a federal holiday, although our hallowed San Francisco 49ers are going to be visiting the lion's den ... errr ... First People's coliseum in Atlanta on Sunday. Although the 49ers are 3.5 point favorites, they're on the road on artificial turf in a warm enclosed stadium where visiting teams can make mistakes due to the noise of the 12th Man - the home team fans in an indoor echo chamber.
The best things in life aren't things ... but, a Pi comes pretty darned close! :D
"Education is not the filling of a pail, but the lighting of a fire." -- W.B. Yeats
In theory, theory & practice are the same - in practice, they aren't!!!

Lonewolff
Posts: 143
Joined: Fri Dec 28, 2012 11:13 pm

Re: How to boot into RAID array (mdadm)

Sat Jan 19, 2013 5:36 am

Awesome Jim. I look forward to seeing what you find out.

As, long as the CPU can push around the 55 MB/s happily, things like X, Firefox, and the likes would be almost 'daily use' usable.

I have even tried to mount over the top of / with /dev/md1 but this doesn't seem to do anything.

It would seem that the mount of / with /dev/mmcblk0p2 (the default in fstab) doesn't do anything, anyway. Because, if I comment that line out, the OS boots all the way up as per normal anyway. :?
Loving my peice of the Pi

User avatar
jojopi
Posts: 3000
Joined: Tue Oct 11, 2011 8:38 pm

Re: How to boot into RAID array (mdadm)

Sat Jan 19, 2013 5:45 am

The foundation's kernel binaries have raid1 as a module. So you would need to have loaded that module and assembled /dev/md1 before you could mount it.

The normal way to do that is with an initramfs image containing mdadm, raid1.ko, and necessary scripts. Upstream debian builds the initramfs for you, but I do not think anyone has made kernel packages for raspbian that work the same way.

Alternatively, if you build your own kernel and compile raid in, you might get away with a boot time assembly cmdline like "md=1,1,/dev/sda1,/dev/sdb1". I have never tried this.

Lonewolff
Posts: 143
Joined: Fri Dec 28, 2012 11:13 pm

Re: How to boot into RAID array (mdadm)

Sat Jan 19, 2013 6:03 am

It would appear that the current Pi kernel supports RAID 0 & 5 out of the box. Mdadm didn't work with RAID 1, and online forums suggested this was due to the lack of RAID 1 support in the kernel.

Does this help with your suggestion?

What are the parameters for "md=1,1,/dev/sda1,/dev/sdb1" in your example? Is this something you made up or is it a valid line for cmdline.txt?

Thanks for your help BTW 8-)
Loving my peice of the Pi

User avatar
jojopi
Posts: 3000
Joined: Tue Oct 11, 2011 8:38 pm

Re: How to boot into RAID array (mdadm)

Sat Jan 19, 2013 6:58 am

Lonewolff wrote:It would appear that the current Pi kernel supports RAID 0 & 5 out of the box. Mdadm didn't work with RAID 1, and online forums suggested this was due to the lack of RAID 1 support in the kernel.
I am not sure what kernel you consider current, but I see:

Code: Select all

pi@delta ~ $ zcat /proc/config.gz |grep MD_
# CONFIG_MD_LINEAR is not set
# CONFIG_MD_RAID0 is not set
CONFIG_MD_RAID1=m
# CONFIG_MD_RAID10 is not set
CONFIG_MD_RAID456=m
# CONFIG_MD_MULTIPATH is not set
# CONFIG_MD_FAULTY is not set
So raid levels 1,4,5,6 are supported, as modules.
What are the parameters for "md=1,1,/dev/sda1,/dev/sdb1" in your example?
I have a couple of missing commas there. See: https://github.com/raspberrypi/linux/bl ... ion/md.txt

Lonewolff
Posts: 143
Joined: Fri Dec 28, 2012 11:13 pm

Re: How to boot into RAID array (mdadm)

Sat Jan 19, 2013 11:44 am

My bad, I meant that RAID 0 wasn't supported (Striping). As, your text above confirms.
Loving my peice of the Pi

Lonewolff
Posts: 143
Joined: Fri Dec 28, 2012 11:13 pm

Re: How to boot into RAID array (mdadm)

Sat Jan 19, 2013 9:49 pm

Hi jojopi,

I have tried adding this to cmdline.txt

md=1,/dev/sda1,/dev/sdb1,/dev/sdc1 md_mod.start_dirty_degraded=1

and also (for older style arrays)

md=1,5,0,0,/dev/sda1,/dev/sdb1,/dev/sdc1 md_mod.start_dirty_degraded=1

Still hangs straight after USB detection and there is nothing on the screen during the boot process that suggests that it is trying to re-assemble the array. :?

I am confident that your suggestions are on the right track though :D

Any more suggestions would be awesome!
Loving my peice of the Pi

User avatar
jojopi
Posts: 3000
Joined: Tue Oct 11, 2011 8:38 pm

Re: How to boot into RAID array (mdadm)

Sat Jan 19, 2013 11:45 pm

Are you using a kernel with the necessary raid options compiled in, not as modules? That is "CONFIG_MD_RAID1=y", not "CONFIG_MD_RAID1=m". If modules are required then you must use an initramfs to load them before mounting the root filesystem.

I cannot guarantee that the boot time assembly cmdline will work, even with raid compiled into the kernel, because I have never tried it. But I do think that compiling a custom kernel to test it will be easier than building a custom initramfs.

Either way you may need to log the serial console output for debugging purposes.

Lonewolff
Posts: 143
Joined: Fri Dec 28, 2012 11:13 pm

Re: How to boot into RAID array (mdadm)

Sun Jan 20, 2013 3:05 am

Looks like you are right, it appears to be a module.

I might try a custom kernel like you suggest.

I'll have to google that one though, as this will be a first for me. :lol:

Thanks for your awesome help so far, though 8-)
Loving my peice of the Pi

Lonewolff
Posts: 143
Joined: Fri Dec 28, 2012 11:13 pm

Re: How to boot into RAID array (mdadm)

Sun Jan 20, 2013 6:31 am

Hi jojopi,

I have managed to get my Ubuntu rig to build up a kernel from scratch. 8-)

I am struggling to find out how to build RAID directly into the kernel. Could you please give me some advice on how I can add this in?
Loving my peice of the Pi

dawithers
Posts: 9
Joined: Thu Nov 28, 2013 2:14 am
Contact: Website

Success!

Thu Nov 28, 2013 2:28 am

I was able to accomplish this without modifying the kernel. The trick is installing the linux-image-rpi-rpfv kernel from the Raspbian repository. It includes an initramfs in which you can load the raid1 module during boot which will allow you to use /dev/md0 as your root. I am in the process of writing a blog post detailing the steps I took. I will post it shortly.


Lonewolff
Posts: 143
Joined: Fri Dec 28, 2012 11:13 pm

Re: How to boot into RAID array (mdadm)

Fri Nov 29, 2013 3:38 am

Your article refers to RAID 1 (which is available out of the box).

This post is about getting RAID 0 working, which isn't out of the box and does require a kernel recompile to add the missing module.

I did mention RAID 1 earlier on in the post, but corrected myself in a subseqent post that I was actually meaning RAID 0 (striping).

So your article you have posted does not relate to this thread.
Loving my peice of the Pi

Return to “General discussion”

Who is online

Users browsing this forum: Heater and 50 guests