Nick Bryant wrote:
Just wondering if anyone out there has any opinions or better still experience with Coraid's AoE products in a centos environment?
I have been bombarded with CORAID marketing in various Linux groups over the last few months. This is because CORAID smartly hit the Linux tradeshow circuit, which is definitely the best way to get free "word of mouth" advertising.
I have seen their products. I have read their technical specifications. And I have come to one conclusion.
AoE is vendor marketing. It is NOT SAN of any sort. It relies 100% on server-wide (e.g., GFS) coherency. Or you merely use it as dedicated storage (even if you slice for different servers).
- The "efficiency" argument
Sure, it has less overhead than iSCSI. That means if you put in a cheap, $50 GbE card or use the on-motherboard GbE NIC, you're going to get better performance.
But anyone who is serious about a SAN puts in a $500 iSCSI HBA GbE, which are very affordable now.
- The "feature" reality
AoE has virtually _no_ features. It's "dumb storage" and that's that. iSCSI is intelligent. How intelligent your target acts is up to you (and the cost). It's multi-vendor and there are complete standards, from host to target -- especially intelligent targets. AoE just a simplistic block interface that relies on 100% software coherency.
We're looking at developing a very basic active/standby 2 node cluster which will need a shared storage component.
AoE is _not_ it then. AoE does _not_ allow shared storage. You must slice the array for each system so they are _independent_. It is _not_ a SAN. It does not have multi-targetting features, only segmented target capability.
They talk GFS. It's GFS in the worst absolute setup -- 100% software, 0% intelligent target. ;->
It would be a huge bonus if other servers could also use the storage device.
Now it can do that. You just slice your array for whatever servers.
Originally I was looking at the Dell/EMC iSCSI solution as it's a cheaper solution than fibre channel.
Have you considered Serial Attached SCSI (SAS)? Most people haven't heard of it, so be sure to read my blog from 3 months back: http://thebs413.blogspot.com/2005/08/serial-storage-is-future.html
12-24Gibps (1.2-2.4GiBps after 10/8 encoding) using a 4 or 8 channel trunk, which is what external target solutions are using. SAS is SCSI-2 over Serial (physically, basically same as SATA, with twisted pair which SATA-IO also requires). It's very scalable and flexable and _damn_fast_. In a nutshell, SAS can use ... - Internal SATA drives - Internal SAS drives - External SAS drives - External SAS enclosures (with SAS drives) - External SAS subsystems (with SATA or SAS drives) - External SAS hubs (intelligent multi-targetting)
So what's the catch of SAS? Same as SCSI: 1. Few people go for the multi-target options 2. Shorter distance (8m ~ 25')
#2 is not an issue if you're providing storage for the closet. That was always the bonus with multi-target, intelligent target, SCSI, before higher-speed FC-AL was available.
#1 is where SAS is still "getting off the ground." It leverages existing SCSI-2, so the multi-targetting is there. But the vendor products are still coming out.
So the verdict is still out on SAS as a SAN solution merely because of the limited products available right now. But it's definitely far more affordable than FC-AL, leverages everything learned with multi-target SCSI-2 and is a heck of a lot better for the closet than iSCSI (where distance isn't a factor).
However, the performance issues without using a TCP offload HBA are a bit of a concern.
If you go iSCSI, you _must_ go with a HBA. Heck, I would argue that you would probably want a HBA for layer-2 Ethernet too, although the layer-3/4 traffic is far worse.
But HBAs start at $500 these days. That's chump change. There are some outstanding iSCSI HBAs under $1,000, so that shouldn't deter you.
An intelligent, multi-targetted subsystem is where the cost is going to be. And that's where multi-target SAS devices, once they become more commonplace, should be significantly cheaper than iSCSI (before figuring disk cost).
Then I found the Coraid (www.coraid.com) products based on the open standard AoE protocol.
It's _their_ standard. And its _empty_. It has _no_ SAN-level code. There is _no_ multi-targeting logic. You have to slice the array -- only 1 connection per end-user volume.
It's got a number of benefits including: price,
So does SAS, and it uses the proven SCSI-2 protocol for multi-targetting.
less protocol overhead for the server
SCSI-2 is better than layer-2 Ethernet, let alone designed for storage. ;->
and the ability to use any disks where as "enterprise" approved products form the likes of Dell/Sun etc only support 250gb sata disks at the moment.
Not true! There are 400 and 500GB near-line 24x7 disks from Seagate and Hitachi, respectively.
[ Sounds like someone fed you marketing. ;-]
In fact, one of Hitachi's big partners is Copan Systems, and they have the 500GB drives in their VTL solution.
I guess my concern is that it's a new technology that's not been widely adopted so far and all the issues that go along with that.
That's the _least_ of your concerns. AoE has _nothing_ in it from a SAN perspective.
I sure wish CORAID was more forthcoming on that.
At the same time, whenever I mention multi-targetting the same volume, it finally shuts up the marketeers. It's all hype, 0 SAN substance.
Dave Hornford OSD@HornfordAssociates.com wrote:
What are you planning on running over the shared connection? Database, eMail, File Shares? How many users? How much data? What is your I/O profile?
Agreed. If you can afford the latency and lower DTR, iSCSI will do. If you need maximum performance, really investigate SAS.
I've worked with 'enterprise' storage most of my career either as a consumer, adviser or provider - can't comment on AoE other than to suggest you look at what are the business & technical goals, how they solve it and what is your risk profile against the business need of the system. (I'm currently working as an adviser)
AoE is _not_ a SAN solution. That 1 statement remove any consideration.
At the same time, I haven't deployed a multi-targetted SAS solution yet. The boards are out, the drives are out, the enclosures are out, and some subsystem products exist. Just because it leverages existing SCSI-2 multi-targetting doesn't mean someone has a well-designed, well-respected, intelligent SAS multi-targettable/sharable solution yet.
When you have an opportunity to chase into "enterprise support" of new disk is usually comes down to insufficient time for testing, or known problems (heat, torque, variability in sample, failure rate & edge-case incompatibility with previously certified products are normal)
AoE is designed for 1 thing -- centralized, segmented storage. It does that well. However, it is _not_ a SAN standard. It does _not_ support multi-targetting of the same volume.
So that means it's no different than if you had local storage using 100% software (e.g., via GFS) to synchronize. It is "dumb".
You have hinted with the concern over TCP offload that you may have higher-end performance needs and that this system carries a high business value and needs a lower risk solution.
Agreed. If you only need storage in the same closet (same 25'), see what intelligent, multi-target SAS solutions are available.
Remember risk is a cost.
Hmmm, I don't know why I haven't formulated that statement before. I talk at length with clients about "risk analysis," yet that simple statement says it all.