On 12/02/2009 11:49 PM, Ben M. wrote:

Thanks. The portability bonus is a big one. Just two other questions I think.

• Raid1 entirely in dom0?

that's what I do, for simplicity sake. I do all raid in Dom0, usually also LVM (sometime I do use simple/plain/old partitions in dom0 and LVM is domU, but very rarely). after that it's fairly easy to create a new LV (or simply a file in /var/lib/xen/images -- even if it's just bind-mounted from another disk) and use it for the new VMs

On Wed, Dec 2, 2009 at 2:44 PM, Ben M. centos@rivint.com wrote:

Thanks for sharing Grant. Your point about hardware raid is well taken. However, the discussion is about Fake-Raid vs. Software RAID1 and controller/chipset dependence and portability. The portability of a software RAID1 hard drive to an entirely different box is, I have learned, much higher and less time consuming.

Portability is no different with a RAID controller as long as you've standardized on controllers. Our situation is not standard but hundreds of drives are swapped between machines throughout the month without an issue. We use RAID sets to transfer large amounts of data (TBs) between machines. All systems use 3ware 9550 or 9650 controllers though so you throw the set in and rescan an it's there.

Grant McWilliams

On Wed, Dec 2, 2009 at 9:51 PM, Christopher G. Stach II cgs@ldsys.netwrote:

----- "Grant McWilliams" grantmasterflash@gmail.com wrote:

...

Portability is no different with a RAID controller as long as you've standardized on controllers.

For this to be true, it would have to be absolute. Since many people have evidence that it is not true, it's not absolute. Controllers of the same revision and firmware version have had portability problems.

-- Christopher G. Stach II

Like I said in our environment we have hundreds of drives moving between controllers *every month and have had zero problems with it. Not all machines use the same controller (some are 3ware 9550SX and some are 3ware 9650SE) nor do they use the same firmware version. We've been doing this for 3 years now and have never had a drive have problems. All drives are initialized, partitioned, formatted and populated with content in 3 locations around the world and then multiple sets are shipped to 75 machines in various geographical zones and used for one month. At this point we've done close to 10,000 swaps (300/month) and we've never had a controller not see a drive and recognize it without anything more than a tw_cli rescan.

You can talk theoretics but I can tell you my real world experience. I cannot speak for other vendors but for 3ware this DOES work and is working so far with 100% success. I have a bunch of Areca controllers too but the drives are never moved between them so I can say how they'd act in that circumstance.

Grant McWilliams

Grant McWilliams grantmasterflash@gmail.com writes:

I don't use software RAID in any sort of production environment unless it's RAID 0 and I don't care about the data at all. I've also tested the speed between Hardware and Software RAID 5 and no matter how many CPUs you throw at it the hardware will win. Even in the case when a 3ware RAID controller only has one drive plugged in it will beat a single drive plugged into the motherboard if applications are requesting dissimilar data. One stream from an MD0 RAID 0 will be as fast as one stream from a Hardware RAID 0. Multiple streams of dissimilar data will be much faster on the Hardware RAID controller due to controller caching.

Personally, I never touch raid5, but then, I'm on sata. I do agree that there are benifits to hardware raid with battery backed cache if you do use raid5 (but I think raid5 is usually a mistake, unless it's all read only, in which case you are better off using main memory for cache. you are trading away small write performance to get space; with disk, space is cheap and performance is expensive, so personally, if I'm going to trade I will trade in the other direction.)

However, with mirroring and striped mirrors (I mirror everything; even if I don't care about the data, mirrors save me time that is worth more than the disk.) my bonnie tests showed that md was faster than a $400 pcie 3ware.

As far as I can tell, there's not much advantage to hardware raid on SATA; if you want to spend more money, get SAS. The spinning disks are going to be the slowest thing in your storage system by far.

battery backed cache is cool for writes, but most raid controllers have such puny caches, it doesn't really help much at all except in the case of small writes to raid5.

On 12/03/2009 03:08 AM, Grant McWilliams wrote:

Personally, I never touch raid5, but then, I'm on sata.   I do agree
that there are benifits to hardware raid with battery backed cache if
you do use raid5 (but I think raid5 is usually a mistake, unless it's
all read only, in which case you are better off using main memory for
cache.  you are trading away small write performance to get space;
 with
disk, space is cheap and performance is expensive, so personally, if
I'm going to trade I will trade in the other direction.)

Interesting thoughts on raid5 although I doubt many would agree. I don't see how the drive type has ANYTHING to do with the RAID level. There are different RAID levels for different situations I guess but a RAID 10 (or 0+1) will never reach the write or read performance http://www.tomshardware.com/reviews/external-raid-storage,1922-9.html of a RAID-5. The disk space waste isn't too much of a problem anymore because as you say drives are getting much cheaper. Although on that subject I'll mention that enterprise drives and desktop drives are NOT the same thing. We deal in hundreds of drives and see about a 3% failure on desktop drives and only a fraction of that on enterprise drives.

I will say though that in my opinion the one really important thing to consider is the price. These controllers aren't cheap and if you skimp you will pay. For sequential single reads (streaming one stream) I'd consider using a software "RAID" 0. For a mirror I'd consider Software RAID but once I get serious and go for RAID5 or RAID6 I'd only use Hardware RAID.

and none of this options is the answer to the problem "FakeRaid or Software Raid" :))

Grant McWilliams grantmasterflash@gmail.com writes:

Interesting thoughts on raid5 although I doubt many would agree. I don't see how the drive type has ANYTHING to do with the RAID level.

raid5 tends to suck on small random writes; SATA sucks on small random anything, so your worst-case (and with my use case, and most 'virtualization' use cases, I you spend almost all your time in the worst-case) is much worse than raid5 on SAS, which has reasonable random performance.

The other reason why I think SATA vs SAS matters when thinking about your raid card is that the port-cost on a good raid card is so high that you could almost double your spindles at sata prices, and without battery backed cache, the suckage of RAID5 is magnified, so soft-raid5 is generally a bad idea.

There are different RAID levels for different situations I guess but a RAID 10 (or 0+1) will never reach the write or read performancehttp://www.tomshardware.com/reviews/external-raid-storage,1922-9.htmlof a RAID-5.

I understand raid5 is great for sequential, but I think my initial OS install is the last sequential write any of my servers ever see. My use-case, you see, is putting 32GiB worth of VPSs on one mirror (or a stripe of two mirrors) It is all very random, just 'cause you have 30+ VMs on a box.

If you do lots of sequential stuff, you will see very different results, but the vitalization use case is generally pretty random, because multiple VMs, even if they are each writing or reading sequentially, make for random disk access.

(now, why do my customers tolerate this slow sata disk? because it's cheap. Also, I understand my competitors use similar configurations.)