Starting from scratch: RAID + LVM
by Mez on Nov.03, 2008, under Personal
Ok, so working with RAID and LVM is… well… not easy.
I have three hard drives in this machine, all of different sizes, (160Gb, 200Gb, and 300Gb), and I’m gong to be setting these up so that I have a RAID-5 made up of three 160Gb Partitions, (with LVM on top) a RAID-1 Partition of 40Gb (no LVM) and the rest split between boot, swap, and misc.
However, the ubuntu alternative CD doesn’t make this easy, and, one of my favourite annoyances is that once you configure RAID, you cannot make changes to the partitions on the disk where you have RAID partitions 
(hidden away in the middle of a block of text that people aren’t likely to read… I only spotted it on the 4th/5th attempt)
Getting RAID setup is fine, getting RAID+LVM has taken me… 6 attempts so far, and I think that I’ve managed to say the right prayers to the right gods this time… or maybe not…
“The kernel was unable to re-read the partititon table on /dev/md1 (Invalid argument). This means Linux won’t know anything about the modifications you made until you reboot. you should reboot your computer before doing anything with /dev/md1
Maybe I should do this manually *clicks reboot*
November 3rd, 2008 on 7:01 pm
Well, you should not (never!) partition a md-device. Very bad things will happen, if you try (which you just experienced).
Trust me, your setup is not very exotic or special, you just need to follow a strict order:
1) setup _all_ partitions
2) setup every md-device
2a) never partition a md-device.
2b) no, don’t do it, really!
3) configure LVM
4) profit
I’ve done similar setups using the installer, worked like a charm.
November 3rd, 2008 on 8:09 pm
I can’t even get raid to work. Click F6 in windows and loading a driver is FAR EASIER than raid with Ubuntu. Alan Pope did some videos about install raid a while back and I tried it with no success. It would be great to get updated screencast/videos up explaining this with 8.10 from scratch. (FYI this is from a user’s perspective).
November 3rd, 2008 on 8:19 pm
I’ve had quite bad experiences with LVM, also in the combination with software raids (raid not starting after reboot, magically changing uuids, unable to assemble array although the devices are there, etc.)
I don’t know if i’ve been unlucky, but I stopped using LVM completely. IMHO it’s just not worth it.
November 4th, 2008 on 12:57 am
I think LVM is the best invention since pre-sliced bread.
I used it since the old 2.2.x days, when it was an addon patch to the kernel.
Right now all my servers (several mail-storage setups in the multi-TB size, Xen-Hosts, multi-purpose systems, etc.) are setup using MD-RAID with LVM on top.
Most people which had severe problems using LVM had a) faulty hardware, b) an error in their way of thinking or c) tried to use it in a way or for a purpose it was not designed for.
My setup normally looks like this:
————————
root@XYZ-1:~# fdisk -l /dev/sda
Disk /dev/sda: 80 GB, 80023749120 bytes
255 heads, 63 sectors/track, 9729 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 * 1 243 1951866 fd Lnx RAID auto
/dev/sda2 244 486 1943865 82 Linux swap
/dev/sda3 487 9729 74236365 fd Lnx RAID auto
root@XYZ-1:~# fdisk -l /dev/sdb
Disk /dev/sdb: 80 GB, 80023749120 bytes
255 heads, 63 sectors/track, 9729 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 * 1 243 1951866 fd Lnx RAID auto
/dev/sdb2 244 486 1943865 82 Linux swap
/dev/sdb3 487 9729 74236365 fd Lnx RAID auto
————————
On the first partition is /dev/md0, which is / (root), while the third partition formes /dev/md1, where my LVM resides:
root@XYZ-1:~# lvscan
ACTIVE ‘/dev/vg03/backup_lv’ [1.09 TB] inherit
ACTIVE ‘/dev/vg02/home_lv’ [1.00 TB] inherit
ACTIVE ‘/dev/vg01/usr_lv’ [4.00 GB] inherit
ACTIVE ‘/dev/vg01/var_lv’ [4.00 GB] inherit
ACTIVE ‘/dev/vg01/tmp_lv’ [4.00 GB] inherit
ACTIVE ‘/dev/vg01/vartmp_lv’ [4.00 GB] inherit
ACTIVE ‘/dev/vg01/spool_lv’ [8.00 GB] inherit
ACTIVE ‘/dev/vg01/log_lv’ [6.00 GB] inherit
(vg02 and vg03 are on a FibreChannel storage array.)
As you can see, I not even using the full 72GB of vg01. But if I ever need a bigger /var/log or some other filesystem, I can resize or create one, without the need of fiddling with partitions. And, this is the best part: I don’t need to shutdown or reboot the machine. This is why I use LVM.
November 4th, 2008 on 10:12 pm
You can give the recent openSUSE release (11.1) a try, complex stuff like raid+lvm should be supported out of the box and the partitioning configuration has been much improved lately.
February 25th, 2009 on 6:37 pm
“Well, you should not (never!) partition a md-device. Very bad things will happen, if you try (which you just experienced).”
if you don’t setup partition on the md0,md1,md2 then the install won’t continue. Can you elaborate about what we are supposed to do after that?
March 12th, 2009 on 4:11 am
You should either partition your drives and then setup raids across the partitions for each mountpoint. Meaning you format /dev/md0 etc directly as ext3 or whatever you want.
OR
Use LVM to “partition” your raid device. Meaning you format the logical volumes and mount those.
I don’t know if I express that correctly,
Check out http://tldp.org/HOWTO/Software-RAID-HOWTO.html