Hey folks,
I had some general questions and when reading through the list archives I came across an iSCSI discussion back in February where a couple of individuals were going back and forth about drafting up a "best practices" doc and putting it into a wiki. Did that ever happen? And if so, where is it?
Now my questions : We are not using iSCIS yet at work but I see a few places where it would be useful e.g. a number of heavy-use NFS mounts (from my ZFS appliance) that I believe would be slightly more efficient if I converted them to iSCSI. I also want to introduce some virtual machines which I think would work out best if I created iSCSI drives for them back on my Oracle/Sun ZFS appliance.
I mentioned iSCSI to the guy whose work I have taken over here so that he can concentrate on his real job, and when I mentioned that we should have a separate switch so that all iSCSI traffic is on it's own switch, he balked and said something like "it is a switched network, it should not matter". But that does not sit right with me - the little bit I've read about iSCSI in the past always stresses that you should have it on its own network.
So 2 questions : - how important is it to have it on its own network? - is it OK to use an unmanaged switch (as long as it is Gigabit), or are there some features of a managed switch that are desirable/required with iSCSI?
thanks, -Alan
On Fri, Dec 9, 2011 at 11:27 AM, Alan McKay alan.mckay@gmail.com wrote
So 2 questions :
- how important is it to have it on its own network?
I would say very important, but probably not required. A separate network segregates the traffic, and you can secure it better. You can also have failover, etc, and potentially use cheaper switches.
- is it OK to use an unmanaged switch (as long as it is Gigabit), or are
there some features of a managed switch that are desirable/required with iSCSI?
I've setup two iSCSI storage networks. The first was with unmanaged Dell switches... each was only about $200 I think, and it worked great. For the second, I'm not using switches at all and connecting directly from the NIC on the server to the NIC on the disk array. Fortunately, we only have a couple of servers and the IBM disk array comes with an additional card that has 4 NICs on it.
You can use managed switches, even ones that are currently supporting your LAN, but I would create some VLANs to separate the traffic. You should also make sure jumbo frames are enabled as well. If you're concerned about maximizing throughput, then a managed switch will have more options to fine-tune this such as link aggregation, but in my cases, I wasn't worried about this because the "default" setup was fast enough for our needs.
In the current direct connect setup, the iSCSI network is supporting virtual machines. So even without much tinkering, the speed is good enough. However, everyone's requirements are different.
...adam ____________________________________________ Adam Wead Systems and Digital Collections Librarian Rock and Roll Hall of Fame and Museum
On 12/09/2011 11:27 AM, Alan McKay wrote:
Hey folks,
I had some general questions and when reading through the list archives I came across an iSCSI discussion back in February where a couple of individuals were going back and forth about drafting up a "best practices" doc and putting it into a wiki. Did that ever happen? And if so, where is it?
Now my questions : We are not using iSCIS yet at work but I see a few places where it would be useful e.g. a number of heavy-use NFS mounts (from my ZFS appliance) that I believe would be slightly more efficient if I converted them to iSCSI. I also want to introduce some virtual machines which I think would work out best if I created iSCSI drives for them back on my Oracle/Sun ZFS appliance.
As you are aware, NFS vs. iSCSI is an apples/oranges comparison. As two or more machines will see the same "block device" using iSCSI, it falls on higher layers to ensure that the storage is accessed safely (ie: using clustered LVM, GFS2, etc). Alternatively, you need to ensure that no two nodes access the same partitions at the same time, which precludes live migration of VMs.
I mentioned iSCSI to the guy whose work I have taken over here so that he can concentrate on his real job, and when I mentioned that we should have a separate switch so that all iSCSI traffic is on it's own switch, he balked and said something like "it is a switched network, it should not matter". But that does not sit right with me - the little bit I've read about iSCSI in the past always stresses that you should have it on its own network.
"Switched network" simply means that data going from machine A to machine B isn't sent to machine C. It doesn't speak to capacity issues.
So 2 questions :
- how important is it to have it on its own network?
From what standpoint? I always have a dedicated network, primarily to
ensure that if/when the network is saturated, other traffic isn't interrupted. This is particularly important when you have latency-sensitive applications like clustering.
There are also arguments for security, but this is only half-true. A VLAN would isolate the network, and encrypting the traffic (with it's performance trade-offs) would protect it from sniffing.
- is it OK to use an unmanaged switch (as long as it is Gigabit), or are
there some features of a managed switch that are desirable/required with iSCSI?
My concern wouldn't be with managed/unmanaged so much as slow/fast. Cheap switches generally have low(er) internal switching bandwidth, so be sure to look up that in the switch's specs and compare it to your expected loads. There are other differences, too, like MAC table sizes and whatnot.
thanks, -Alan
This may or may not be of use to you, but here is a link to an (in-progress, incomplete) tutorial I am working on. The block diagram just below this link shows how I configure my (VM Cluster) networks. It uses a dedicate "SN" (Storage Network) for DRBD replication traffic, so it's not for iSCSI but the concept is similar.
https://alteeve.com/w/2-Node_Red_Hat_KVM_Cluster_Tutorial#Network
The network in this configuration is completely redundant (bonding mode=1 (Active/Passive) across two switches). In my case, there are just two managed switches, but there is no real reason that you can't use a pair of unmanaged switches for each subnet (paired for redundancy), so long as those switches are sufficient for your expected load/growth.
Cheers
On Fri, Dec 9, 2011 at 10:27 AM, Alan McKay alan.mckay@gmail.com wrote:
Now my questions : We are not using iSCIS yet at work but I see a few places where it would be useful e.g. a number of heavy-use NFS mounts (from my ZFS appliance) that I believe would be slightly more efficient if I converted them to iSCSI. I also want to introduce some virtual machines which I think would work out best if I created iSCSI drives for them back on my Oracle/Sun ZFS appliance.
This doesn't directly apply, but this nfs appliance vendor wants you to think that nfs isn't as bad as you might think: http://www.bluearc.com/bluearc-resources/downloads/analyst-reports/BlueArc-A... Overcommitting for de-dup/compression might be harder with iscsi - resizing filesystems would be a lot harder.
I mentioned iSCSI to the guy whose work I have taken over here so that he can concentrate on his real job, and when I mentioned that we should have a separate switch so that all iSCSI traffic is on it's own switch, he balked and said something like "it is a switched network, it should not matter".
Is it a single switch? Otherwise you share the bandwidth on the trunk connections.
But that does not sit right with me - the little bit I've read about iSCSI in the past always stresses that you should have it on its own network.
So 2 questions :
- how important is it to have it on its own network?
- is it OK to use an unmanaged switch (as long as it is Gigabit), or are
there some features of a managed switch that are desirable/required with iSCSI?
I've seen recommendations to use jumbo frames for iscsi - and if you do that, everything on that subnet needs to be configured for them.
On Fri, Dec 9, 2011 at 1:40 PM, Karanbir Singh mail-lists@karan.org wrote:
I've seen recommendations to use jumbo frames for iscsi - and if you do that, everything on that subnet needs to be configured for them.
unless you have some really slow storage machines, why would you not use JF ?
I thought everything in the path had to be configured for the same frame size - which shouldn't be a problem if you build out a dedicated storage network but might if you are trying to share trunk connections.
The big issue in corporate land would be security. Yes you can do vlans and/or encrypt it, but that is going to add overhead, either management (*people) or CPU, both of which take away from any speed advantages you might get.
On Fri, 9 Dec 2011, Alan McKay wrote:
Hey folks,
I had some general questions and when reading through the list archives I came across an iSCSI discussion back in February where a couple of individuals were going back and forth about drafting up a "best practices" doc and putting it into a wiki. Did that ever happen? And if so, where is it?
Now my questions : We are not using iSCIS yet at work but I see a few places where it would be useful e.g. a number of heavy-use NFS mounts (from my ZFS appliance) that I believe would be slightly more efficient if I converted them to iSCSI. I also want to introduce some virtual machines which I think would work out best if I created iSCSI drives for them back on my Oracle/Sun ZFS appliance.
I mentioned iSCSI to the guy whose work I have taken over here so that he can concentrate on his real job, and when I mentioned that we should have a separate switch so that all iSCSI traffic is on it's own switch, he balked and said something like "it is a switched network, it should not matter". But that does not sit right with me - the little bit I've read about iSCSI in the past always stresses that you should have it on its own network.
So 2 questions :
- how important is it to have it on its own network?
- is it OK to use an unmanaged switch (as long as it is Gigabit), or are
there some features of a managed switch that are desirable/required with iSCSI?
thanks, -Alan
---------------------------------------------------------------------- Jim Wildman, CISSP, RHCE jim@rossberry.com http://www.rossberry.net "Society in every state is a blessing, but Government, even in its best state, is a necessary evil; in its worst state, an intolerable one." Thomas Paine
On Dec 9, 2011, at 11:27 AM, Alan McKay alan.mckay@gmail.com wrote:
So 2 questions :
- how important is it to have it on its own network?
The traffic should definitely be segregated for security reasons and to make sure there is minimal crosstalk. Whether to put on a separate switch or VLAN depends on your current network capacity.
- is it OK to use an unmanaged switch (as long as it is Gigabit), or are
there some features of a managed switch that are desirable/required with iSCSI?
Unmanaged should be used only if you plan to restrict it to iSCSI traffic only and then get a high quality one.
Managed switches come in very high quality versions that give a flexibility, security and performance.
The Dell 6224 or 6248 switches are priced low and provide excellent features for iSCSI. Plus they can have up to 4 10Gbps ports that can be used for uplinks OR 10Gbps endpoints.
-Ross
The Dell 6224 or 6248 switches are priced low
Hmmm, we seem to have different definitions of "priced low" :-)
http://search.dell.com/results.aspx?s=bsd&c=ca&l=en&cs=cabsdt1&a...
$2000 for the 24 port.
I can get a Cisco small business switch for less than 1/4 that. Speaking of which, would this one be OK for iSCSI?
http://www.shoprbc.com/ca/shop/product_details.php?pid=71436
On Dec 10, 2011, at 9:35 AM, Alan McKay alan.mckay@gmail.com wrote:
The Dell 6224 or 6248 switches are priced low
Hmmm, we seem to have different definitions of "priced low" :-)
http://search.dell.com/results.aspx?s=bsd&c=ca&l=en&cs=cabsdt1&a...
$2000 for the 24 port.
I can get a Cisco small business switch for less than 1/4 that. Speaking of which, would this one be OK for iSCSI?
http://www.shoprbc.com/ca/shop/product_details.php?pid=71436
Actually the 5424 would be a better then the 6224, the base is around $1200, if you call a rep you can probably get 33% off, given it's year end, often they will have a buy one get one free which equates to 50% off.
As far as the Cisco Small Business line is concerned, that line was formed from the acquisition of Linksys, so it's basically Linksys small business switches and you get what you pay for.
If this is for a home media server, then I'm sure it will be more then adequate, if this is a business system I would invest in equipment whose cost is in relation to the cost of the data contained within.
You wouldn't want a cheap switch that locks up when the traffic gets heavy to ruin your data integrity.
Lump the cost of the switches, nics and cables into the total cost of the storage solution and justify that cost by the cost of the integrity of the data it contains. There is your business rational for the investment.
-Ross
----- Original Message ----- | > The Dell 6224 or 6248 switches are priced low | | Hmmm, we seem to have different definitions of "priced low" :-) | | http://search.dell.com/results.aspx?s=bsd&c=ca&l=en&cs=cabsdt1&a... | | $2000 for the 24 port. | | I can get a Cisco small business switch for less than 1/4 that. | Speaking of which, would this one be OK for iSCSI? | | http://www.shoprbc.com/ca/shop/product_details.php?pid=71436
LOL! Cisco. If I told you that that particular device used to be called Linksys, would it change your opinion of the device? I've got a Linksys ADSL gateway that I'm quite sure couldn't keep up with the Dell. In fact, I used to have that *exact* Linksys device and it died within 18 months and it's performance sucked!
LOL! Cisco. If I told you that that particular device used to be called Linksys, would it change your opinion of the device? I've got a Linksys ADSL gateway that I'm quite sure couldn't keep up with the Dell. In fact, I used to have that *exact* Linksys device and it died within 18 months and it's performance sucked!
Yes I know it is LinkSys - we used them at my old job and yeah we had one die but the price was worth it. I am mainly concerned with the speed of the thing and managability. My current job is a lot less sensitive to outages than my last one, as well.
On Dec 10, 2011, at 2:36 PM, Alan McKay alan.mckay@gmail.com wrote:
LOL! Cisco. If I told you that that particular device used to be called Linksys, would it change your opinion of the device? I've got a Linksys ADSL gateway that I'm quite sure couldn't keep up with the Dell. In fact, I used to have that *exact* Linksys device and it died within 18 months and it's performance sucked!
Yes I know it is LinkSys - we used them at my old job and yeah we had one die but the price was worth it. I am mainly concerned with the speed of the thing and managability. My current job is a lot less sensitive to outages than my last one, as well.
Outage is one thing, but having the disk volumes disappear mid-transaction can be detrimental to a file system's health.
-Ross
Outage is one thing, but having the disk volumes disappear mid-transaction can be detrimental to a file system's health.
To get this back on-topic and closer to the OP's requests, are there any particular iscsi settings one should consider to increase resiliency and minimise the impact of e.g. a rebooting switch? timeout settings? The big disadvantage of iscsi is that you add another layer that can fail (compared to having virtual machine images on a local disk).
Am 12.12.2011 14:49, schrieb lhecking@users.sourceforge.net:
Outage is one thing, but having the disk volumes disappear mid-transaction can be detrimental to a file system's health.
To get this back on-topic and closer to the OP's requests, are there any particular iscsi settings one should consider to increase resiliency and minimise the impact of e.g. a rebooting switch? timeout settings? The big disadvantage of iscsi is that you add another layer that can fail (compared to having virtual machine images on a local disk).
you should always have two links to your iSCSI device and two different switches so that it does not matter if one switch dies or reboots
On Mon, Dec 12, 2011 at 3:52 PM, Reindl Harald h.reindl@thelounge.net wrote:
Am 12.12.2011 14:49, schrieb lhecking@users.sourceforge.net:
Outage is one thing, but having the disk volumes disappear mid-transaction can be detrimental to a file system's health.
To get this back on-topic and closer to the OP's requests, are there any particular iscsi settings one should consider to increase resiliency and minimise the impact of e.g. a rebooting switch? timeout settings? The big disadvantage of iscsi is that you add another layer that can fail (compared to having virtual machine images on a local disk).
you should always have two links to your iSCSI device and two different switches so that it does not matter if one switch dies or reboots
And then you still have the iSCSI applicance / server to worry about. It can fail as well. Even with redundancy PSU's it could fail - the RAM, CPU, motherboard, controller card, expensive RAID card, etc can fail as well.
Am 12.12.2011 15:13, schrieb Rudi Ahlers:
On Mon, Dec 12, 2011 at 3:52 PM, Reindl Harald h.reindl@thelounge.net wrote:
Am 12.12.2011 14:49, schrieb lhecking@users.sourceforge.net:
Outage is one thing, but having the disk volumes disappear mid-transaction can be detrimental to a file system's health.
To get this back on-topic and closer to the OP's requests, are there any particular iscsi settings one should consider to increase resiliency and minimise the impact of e.g. a rebooting switch? timeout settings? The big disadvantage of iscsi is that you add another layer that can fail (compared to having virtual machine images on a local disk).
you should always have two links to your iSCSI device and two different switches so that it does not matter if one switch dies or reboots
And then you still have the iSCSI applicance / server to worry about. It can fail as well. Even with redundancy PSU's it could fail - the RAM, CPU, motherboard, controller card, expensive RAID card, etc can fail as well.
get the right hardware and you do not have this problem http://h10010.www1.hp.com/wwpc/us/en/sm/WF04a/12169-304616-241493-241493-241...
you have TWO of all components with hotplug
Reindl Harald wrote:
Am 12.12.2011 15:13, schrieb Rudi Ahlers:
On Mon, Dec 12, 2011 at 3:52 PM, Reindl Harald h.reindl@thelounge.net wrote:
Am 12.12.2011 14:49, schrieb lhecking@users.sourceforge.net:
Outage is one thing, but having the disk volumes disappear mid-transaction can be detrimental to a file system's health.
To get this back on-topic and closer to the OP's requests, are there any particular iscsi settings one should consider to increase resiliency and minimise the impact of e.g. a rebooting switch? timeout settings? The big disadvantage of iscsi is that you add another layer that can fail (compared to having virtual machine images on a local disk).
you should always have two links to your iSCSI device and two different switches so that it does not matter if one switch dies or reboots
And then you still have the iSCSI applicance / server to worry about. It can fail as well. Even with redundancy PSU's it could fail - the RAM, CPU, motherboard, controller card, expensive RAID card, etc can fail as well.
get the right hardware and you do not have this problem http://h10010.www1.hp.com/wwpc/us/en/sm/WF04a/12169-304616-241493-241493-241...
you have TWO of all components with hotplug
Until you have a fire, or a water line breaks, or.... I worked somewhere I shouldn't mention about a dozen years ago, that had three H/A clusters... on either side of a drywall wall, not even fire resistant.
mark
Am 12.12.2011 22:37, schrieb m.roth@5-cent.us:
Reindl Harald wrote:
get the right hardware and you do not have this problem http://h10010.www1.hp.com/wwpc/us/en/sm/WF04a/12169-304616-241493-241493-241...
you have TWO of all components with hotplug
Until you have a fire, or a water line breaks, or.... I worked somewhere I shouldn't mention about a dozen years ago, that had three H/A clusters... on either side of a drywall wall, not even fire resistant.
we are speaking about redundancy backups and off-site backups are a total other story
Reindl Harald wrote:
Am 12.12.2011 22:37, schrieb m.roth@5-cent.us:
Reindl Harald wrote:
get the right hardware and you do not have this problem http://h10010.www1.hp.com/wwpc/us/en/sm/WF04a/12169-304616-241493-241493-241...
you have TWO of all components with hotplug
Until you have a fire, or a water line breaks, or.... I worked somewhere I shouldn't mention about a dozen years ago, that had three H/A clusters...on either side of a drywall wall, not even fire resistant.
we are speaking about redundancy backups and off-site backups are a total other story
I wasn't talking about backups, I was talking about redundancy in such a manner as to not be affected by an external cause, as opposed to hardware failure.
mark
Am 12.12.2011 23:25, schrieb m.roth@5-cent.us:
Reindl Harald wrote:
Am 12.12.2011 22:37, schrieb m.roth@5-cent.us:
Reindl Harald wrote:
get the right hardware and you do not have this problem http://h10010.www1.hp.com/wwpc/us/en/sm/WF04a/12169-304616-241493-241493-241...
you have TWO of all components with hotplug
Until you have a fire, or a water line breaks, or.... I worked somewhere I shouldn't mention about a dozen years ago, that had three H/A clusters...on either side of a drywall wall, not even fire resistant.
we are speaking about redundancy backups and off-site backups are a total other story
I wasn't talking about backups, I was talking about redundancy in such a manner as to not be affected by an external cause, as opposed to hardware failure.
well you can not have redundancy for all sort of natural damages that is what backups for - ok you CAN but you can not pay it usually
On 12/12/2011 09:13 AM, Rudi Ahlers wrote:
On Mon, Dec 12, 2011 at 3:52 PM, Reindl Harald h.reindl@thelounge.net wrote:
Am 12.12.2011 14:49, schrieb lhecking@users.sourceforge.net:
Outage is one thing, but having the disk volumes disappear mid-transaction can be detrimental to a file system's health.
To get this back on-topic and closer to the OP's requests, are there any particular iscsi settings one should consider to increase resiliency and minimise the impact of e.g. a rebooting switch? timeout settings? The big disadvantage of iscsi is that you add another layer that can fail (compared to having virtual machine images on a local disk).
you should always have two links to your iSCSI device and two different switches so that it does not matter if one switch dies or reboots
And then you still have the iSCSI applicance / server to worry about. It can fail as well. Even with redundancy PSU's it could fail - the RAM, CPU, motherboard, controller card, expensive RAID card, etc can fail as well.
I handle this by setting up two servers running DRBD in active/active with a simple two-node red hat cluster managing a floating IP address. The storage network link uses a simple Active/Passive (mode=1) bond with either link go to separate switches.
I've been able to export the SAN to a second cluster, managing the SAN space using clustered LVM, to back VMs. I can live-migrate the VMs between nodes and totally fail the primary SAN and the backup will pick up seemlessly, VMs don't need to reboot.
This offers a substantially less expensive HA option than some commercial HA SAN solutions and avoids the headaches of multipath (which only makes the link redundant, not the SAN itself).
On 12/12/11 6:43 AM, Digimer wrote:
I handle this by setting up two servers running DRBD in active/active with a simple two-node red hat cluster managing a floating IP address. The storage network link uses a simple Active/Passive (mode=1) bond with either link go to separate switches.
DRBD with synchronous writes? doesn't that slow things down considerably? if its asychronous, recently written data will be lost on a failure.
proper storage appliances implement a shared cache between the master and standby storage controllers so that if the master fails, the standby has all data, including cached writes. as far as I know, there's no way to easily implement this with open source, its part of the secret sauce of proper redundant storage.
without this, its not safe to run a transactional database server on your storage
On 12/12/2011 01:17 PM, John R Pierce wrote:
On 12/12/11 6:43 AM, Digimer wrote:
I handle this by setting up two servers running DRBD in active/active with a simple two-node red hat cluster managing a floating IP address. The storage network link uses a simple Active/Passive (mode=1) bond with either link go to separate switches.
DRBD with synchronous writes? doesn't that slow things down considerably? if its asychronous, recently written data will be lost on a failure.
proper storage appliances implement a shared cache between the master and standby storage controllers so that if the master fails, the standby has all data, including cached writes. as far as I know, there's no way to easily implement this with open source, its part of the secret sauce of proper redundant storage.
without this, its not safe to run a transactional database server on your storage
With synchronous writes ("protocol C" in DRBD terms), I've found minimal slow-down. With a little tuning and fast enough networking, it's quite small, given the HA win. Given this, there is no loss in data, even if a node fails mid-write as the caller never gets ack and knows to retry.
Shared cache is, I think, a single-point-of-failure. I'll make no arguments towards transactional DB use and I am not a DBA, but I can restate that a write is not acknowledged until it's been committed to both nodes in DRBD, making it very robust with minimal performance hit.
On 12/12/11 10:23 AM, Digimer wrote:
Shared cache is, I think, a single-point-of-failure.
no, its done with replication over a private channel between the storage controllers. standard feature on all redundant controller hardware/appliance storage controllers such as IBM DS series, HP MSA, etc etc.
On Monday, December 12, 2011 02:02:41 PM John R Pierce wrote:
On 12/12/11 10:23 AM, Digimer wrote:
Shared cache is, I think, a single-point-of-failure.
no, its done with replication over a private channel between the storage controllers. standard feature on all redundant controller hardware/appliance storage controllers such as IBM DS series, HP MSA, etc etc.
EMC Clariion CX/CX3/CX4 and VNX, also.
no, its done with replication over a private channel between the storage controllers. standard feature on all redundant controller hardware/appliance storage controllers such as IBM DS series, HP MSA, etc etc.
EMC Clariion CX/CX3/CX4 and VNX, also.
Ditto D-Link's DSN-5110 series.
On 12-12-11 22:11, Drew wrote:
no, its done with replication over a private channel between the storage controllers. standard feature on all redundant controller hardware/appliance storage controllers such as IBM DS series, HP MSA, etc etc.
EMC Clariion CX/CX3/CX4 and VNX, also.
Ditto D-Link's DSN-5110 series.
Isn't that D-Link DSN-5110 series a rebadged Dot Hill box?
Regards, Patrick
Isn't that D-Link DSN-5110 series a rebadged Dot Hill box?
Rebranded iStor Networks iS512. Incidentally iStor was bought out by Promise so they now OEM the product line for D-Link.
On 12/12/2011 02:02 PM, John R Pierce wrote:
On 12/12/11 10:23 AM, Digimer wrote:
Shared cache is, I think, a single-point-of-failure.
no, its done with replication over a private channel between the storage controllers. standard feature on all redundant controller hardware/appliance storage controllers such as IBM DS series, HP MSA, etc etc.
The same (effective, if not technical) thing can be achieved using two nodes with a dedicated storage link using RAID controllers with battery or flash backed cache.
DRBD will return as soon as the write is committed to the controller's cache instead of the storage proper, so the replication performance should still be very, very fast.
Am 12.12.2011 19:17, schrieb John R Pierce:
On 12/12/11 6:43 AM, Digimer wrote:
I handle this by setting up two servers running DRBD in active/active with a simple two-node red hat cluster managing a floating IP address. The storage network link uses a simple Active/Passive (mode=1) bond with either link go to separate switches.
DRBD with synchronous writes? doesn't that slow things down considerably?
there is always a compromise between performance / integrity / costs
if its asychronous, recently written data will be lost on a failure.
the main question is: how hard is it to lose some recent data by a total crash how often happens a total crash what costs you have for the best compromise
proper storage appliances implement a shared cache between the master and standby storage controllers so that if the master fails, the standby has all data, including cached writes. as far as I know, there's no way to easily implement this with open source, its part of the secret sauce of proper redundant storage.
the question is have you the budget for a high-end SAN-storage where all components are redundant, and even if - you have to make backups for the totally worst case
for me it took a long time to get a HP SAN Storage with two controllers and multipath SAS-disk which i never want to miss again, but these things are really expensive - but well, it is the best solution since you have redundant sotrage/controllers/disks/power-supply and a maximum perfomracne with the two dedicated CPU's and 1 GB cache per controller
but remind the costs and look how important is a zero-downtime in worst-case in what environement, if it is only a internal-storage iand you are a small company it is no problem to say "guys, go home, i need a day for restore backups"
if you have public customers-services it is not an option
On Monday, December 12, 2011 09:13:13 AM Rudi Ahlers wrote:
And then you still have the iSCSI applicance / server to worry about. It can fail as well. Even with redundancy PSU's it could fail - the RAM, CPU, motherboard, controller card, expensive RAID card, etc can fail as well.
These problems were all solved once before, with a different SAN technology..... that is optimized for the storage task, incidentally.....
That being said, the Fibre-channel bog-standard two-SPE and dual-attach drives is a good architecture to follow for iSCSI as well for high availability. Using clustered LVM to back your LUNs can allow this. Dual attach with SAS is possible, too.
Incidentally, upstream is now doing a storage appliance, for those interested.....
On Mon, 12 Dec 2011, lhecking@users.sourceforge.net wrote:
Outage is one thing, but having the disk volumes disappear mid-transaction can be detrimental to a file system's health.
To get this back on-topic and closer to the OP's requests, are there any particular iscsi settings one should consider to increase resiliency and minimise the impact of e.g. a rebooting switch? timeout settings? The big disadvantage of iscsi is that you add another layer that can fail (compared to having virtual machine images on a local disk).
Make sure you're going through multipath even if you've only got a single link the iSCSI box. Just using iscsid and using the devices directly *will* cause you grief when you have failures. If you go via multipath the failures are masked, and it all behaves as you'd hope.
jh
----- Original Message ----- | Hey folks, | | I had some general questions and when reading through the list | archives I | came across an iSCSI discussion back in February where a couple of | individuals were going back and forth about drafting up a "best | practices" | doc and putting it into a wiki. Did that ever happen? And if so, where | is it? | | Now my questions : | We are not using iSCIS yet at work but I see a few places where it | would be | useful e.g. a number of heavy-use NFS mounts (from my ZFS appliance) | that I | believe would be slightly more efficient if I converted them to iSCSI. | I | also want to introduce some virtual machines which I think would work | out | best if I created iSCSI drives for them back on my Oracle/Sun ZFS | appliance. | | I mentioned iSCSI to the guy whose work I have taken over here so that | he | can concentrate on his real job, and when I mentioned that we should | have a | separate switch so that all iSCSI traffic is on it's own switch, he | balked | and said something like "it is a switched network, it should not | matter". | But that does not sit right with me - the little bit I've read about | iSCSI | in the past always stresses that you should have it on its own | network. | | So 2 questions : | - how important is it to have it on its own network? | - is it OK to use an unmanaged switch (as long as it is Gigabit), or | are | there some features of a managed switch that are desirable/required | with | iSCSI? | | thanks, | -Alan
It is not imperative to have a separate switch but it certainly helps, especially if your switch is not managed. Managed switches will allow you to create VLANs that can be both jumbo frame and non-jumbo frame on the same device.
Jumbo frames is really the important thing when it comes to iSCSI. Having 9000 byte packets verses 1500 byte packets will dramatically increase your performance per interrupt. Most cheaper unmanaged switches cannot do this.
Second, if their cheap switches, you'll likely saturate the backplane no matter what. Get good switches for this type of work.
On Dec 10, 2011, at 1:49 PM, "James A. Peltier" jpeltier@sfu.ca wrote:
Jumbo frames is really the important thing when it comes to iSCSI. Having 9000 byte packets verses 1500 byte packets will dramatically increase your performance per interrupt. Most cheaper unmanaged switches cannot do this.
I want to break the myth here, jumbo frames are really only important for 10Gbe+. On 1Gbe (or slow CPUs) they actually slow sequential throughput due to the time to process each packet on 1Gbe and how that in turn slows down packet interleaving and thus total throughput.
-Ross
----- Original Message ----- | On Dec 10, 2011, at 1:49 PM, "James A. Peltier" jpeltier@sfu.ca | wrote: | | > Jumbo frames is really the important thing when it comes to iSCSI. | > Having 9000 byte packets verses 1500 byte packets will dramatically | > increase your performance per interrupt. Most cheaper unmanaged | > switches cannot do this. | | I want to break the myth here, jumbo frames are really only important | for 10Gbe+. On 1Gbe (or slow CPUs) they actually slow sequential | throughput due to the time to process each packet on 1Gbe and how that | in turn slows down packet interleaving and thus total throughput. | | -Ross
On cheap network switches... When you've got a switch that has 1 or more 10Gb uplinks and the rest of the switch is Gb, I doubt this is an issue, but I will agree that on *some* switches it may be detrimental to performance.
On Dec 10, 2011, at 2:05 PM, "James A. Peltier" jpeltier@sfu.ca wrote:
----- Original Message ----- | On Dec 10, 2011, at 1:49 PM, "James A. Peltier" jpeltier@sfu.ca | wrote: | | > Jumbo frames is really the important thing when it comes to iSCSI. | > Having 9000 byte packets verses 1500 byte packets will dramatically | > increase your performance per interrupt. Most cheaper unmanaged | > switches cannot do this. | | I want to break the myth here, jumbo frames are really only important | for 10Gbe+. On 1Gbe (or slow CPUs) they actually slow sequential | throughput due to the time to process each packet on 1Gbe and how that | in turn slows down packet interleaving and thus total throughput. | | -Ross
On cheap network switches... When you've got a switch that has 1 or more 10Gb uplinks and the rest of the switch is Gb, I doubt this is an issue, but I will agree that on *some* switches it may be detrimental to performance.
It's not a function of the switching fabric that changes things, but the speed of the ports.
If each packet takes 6 times longer to take off the wire then the system can handle 1/6th the number of simultaneous streams.
-Ross
On Fri, Dec 9, 2011 at 11:27 AM, Alan McKay alan.mckay@gmail.com wrote:
So 2 questions :
- how important is it to have it on its own network?
- is it OK to use an unmanaged switch (as long as it is Gigabit), or are
there some features of a managed switch that are desirable/required with iSCSI?
At the very least keep the iSCSI traffic on its own vlan. You don't want unnecessary broadcast traffic on your storage network. I would also stay away from unmanaged switches. You want a higher end switch that supports cut through mode. Store and forward will cause higher latency.
Ryan
On 12/10/11 8:56 PM, Ryan Wagoner wrote:
So 2 questions :
- how important is it to have it on its own network?
- is it OK to use an unmanaged switch (as long as it is Gigabit), or are
there some features of a managed switch that are desirable/required with iSCSI?
At the very least keep the iSCSI traffic on its own vlan. You don't want unnecessary broadcast traffic on your storage network. I would also stay away from unmanaged switches. You want a higher end switch that supports cut through mode. Store and forward will cause higher latency.
and, yet another reason to isolate iSCSI to a dedicated SAN segment is that its normally uses rather minimal security, just using the server's IP address for the host masking. yes, it can be used with higher levels of security at the expense of increased complexity, but if its on its own dedicated network, this is sufficient.
-
Adam Wead -
Alan McKay -
Digimer -
Drew -
James A. Peltier -
Jim Wildman -
John Hodrien -
John R Pierce -
Karanbir Singh -
Lamar Owen -
Les Mikesell -
lhecking＠users.sourceforge.net -
m.roth＠5-cent.us -
Patrick Lists -
Reindl Harald -
Ross Walker -
Rudi Ahlers -
Ryan Wagoner