Online Capacity Expansion (Gone South) with Openfiler and a 3Ware RAID card
February 20, 2007 by Jason · 5 Comments
If you've been reading my blog then you'd know that I use OpenFiler for my home storage needs. When I originally provisioned the system it was with a pretty beefy box. One thing I went short on though was storage. I purchased a high-end RAID card (3Ware 9590SE-16ML) but then proceeded to only install 3 drives (Western Digital RE2 500 GB). I did this for a bunch of reasons, but the most important are that, 1. hard drives cost money and they wern't currently needed, 2. hard drive prices go down, and 3. the raid controller I purchased supports Online Capacity Expansion (OCE).
A few days ago I noticed that my storage array was about 80% full. As such, I decided to order another 500GB RE2 (for $40 less than the original three, proving point #2) and test out 3Ware's OCE.
The steps below are what I had planned to do to get the system up and working with the new, larger volume.
- Install the extra drive (turn computer off if you don't have a hot-swap drive cage).
- Open 3DM2 in your web browser and log in as 'admin'.
- Click "Management > Maintenance".
- Check the check box next to your existing array and click "Migrate".
- In the pseudo-popup, select the new drive that you just added, verify the RAID level & stripe size, and then click "OK".
At this point I waited almost 37 hours for the migration to complete. I was expecting to be able to simply use the lvm tools ('pvresize', etc.) to enlarge the physical volume, volume group, and logical volume. Alternatively, I assumed that I could just create a second physical volume, etc. and add it to the existing volume group to accomplish basically the same thing (although this would have limited me in the long run since you can only have 4 primary partitions).
This is where I ran into trouble.
When I installed 2.2 a few months back I went with the 64-bit version and selected a GPT partition table so that I could eventually use a larger-than-2TB volume by taking advantage of OCE and only adding a drive when needed. As it turns out, this makes increasing the physical volume size all but impossible because fdisk can't see it to start (GPT uses a forged MSDOS partition table) and parted can't resize it becuase of the use of LVM (whch means that the physical volume that parted can see doesn't have a file system).
After 2 days of tinkering with the partition table and various versions of LVM & parted, I managed to completely break my GPT partition table. I've managed to "hotwire" the partiton table by rewriting it to think that the logical volume was actually the physical volume. Unfortunately, this means that the OpenFiler GUI is now all-but-useless. As such, I'll be reinstalling sometime over the next couple days. I've ordered another pair of 500GB drives that I'm going to add to the array so hopefully this will all be worked out before I need to expand again.
Oh, and here's a link to a thread on the OpenFiler forums with more information about this issue and what I did to break my system.
"3Ware's Online Capacity Expansion" on OpenFiler Forums
I was going to leave you a message on the openfiler forum, but their site seems to be having issues with registration of users.
The issue is if you use the system-config-lvm python scripts to setup the partitions. If you manually partition, set the type to 0x8e for LVM, I think you can resize.
One thing no one else seems to have noticed
Oops, hit submit mid sentence
One thing no one else seems to have noticed is that the start and end of the partition when partitioned by the scripts are invalid. The end of my partition was after the end of the drive, and consequently when I tried to expand it, everything was broken.
I have not actually tested this doing the real 3ware expansion yet, but I'll post again when I have to let everyone know if it works.
I was able to reproduce the same situation as everyone else by playing games and doing things a bit manually, except that pvresize worked after I deleted the partition and remade it with fdisk (reboot after making the new partition of course...kernel feature required to fix this part).
I have now manually partitioned, and I am running the capacity expansion right now to determine if it is going to work after performing a real OCE.
As I said, I will post here again whether it works or not.
I actually came to the conclusion that it was the GPT partition flag that did it. I tried this out with a few VMWare machines and when the GPT flag was enabled everything went south after I enlarged the disk. I didn't have any problems when using an MSDOS flag.
That could very well be the case. I am not familiar with the differences between GPT and MSDOS; however, I have just now confirmed that OCE on a 3ware (or ARECA) with LVM does work, provided a person manually partitions using fdisk. As for the true root cause, I suspect you are correct that it has something to do with GPT.
I was going to look at the source for LVM and the system-config-lvm scripts, but I got lost somewhere in dm_mod trying to follow what it was doing. Really a shame that most OpenSource stuff doesn't follow minimum commenting standards which are traditional in the business/commercial world. If they did, I suspect that a significant number of folks could assist with debugging (that, and Linux might actually be capable of being evaluated under Common Criteria).
I just expanded my raid5 GPT partition table from 4 750gbs to five (I have the 3ware 9650se-16ml). I have 2k3 installed and used the diskpart command, it only took about 10 seconds to expand the partition table (although my migration took almost 72 hours to add a 5th drive - if you use the bios utility it goes quicker (30-50%), but then you can't use the server).
Here's the site that confirmed the diskpart utility for me (before I made the plunge).
http://www.carltonbale.com/2006/04/expanding-the-capacity-of-my-media-server/#comment-2960
Hope this helps,
Nate
ps - I thought corrupting GPT was extremely hard since it has the table information written twice (in the beginning and the end)?