On 02/07/2011 05:09 PM, Dr. Ed Morbius wrote: > on 15:19 Mon 07 Feb, Ross Walker (rswwalker at gmail.com) wrote: >> On Mon, Feb 7, 2011 at 2:36 PM, Dr. Ed Morbius <dredmorbius at gmail.com> wrote: >>> on 13:56 Mon 07 Feb, Jason Brown (jason.brown at millbrookprinting.com) wrote: >>>> I am currently going through the process of installing/configuring an >>>> iSCSI target and cannot find a good write up on how to prepare the disks >>>> on the server. I would like to mirror the two disks and present them to >>>> the client. Mirroring isn't the question, its how I go about it is the >>>> problem. When I partitioned the two drives and mirrored them together, >>>> then presented them to the client, it showed to the client as a disk out >>>> no partion on it. Should I partition the drive again and then lay the >>>> file system down on top of that? Or should I delete the partitions on >>>> the target server and just have sda and sdb mirrored, then when the >>>> client attaches the disk, then partion it (/dev/sdc1) and write the file >>>> system. >>> >>> What are you using for your iSCSI target (storage array)? >>> >>> Generally, RAID management of the storage is managed on the target side. >>> You'd use the target's native abilities to create/manage RAID arrays to >>> configure one or more physical disks as desired. If you're using a >>> dedicated vendor product, it should offer these capabilities through >>> some interface or another. >>> >>> Presentation of the iSCSI device is as a block storage device to the >>> initiator (host mounting the array). That's going to be an >>> unpartitioned device. You can either further partition this device or >>> use it as raw storage. If you're partitioning it, and using multipath, >>> you'll have to muck with kpartx to make multipath aware of the >>> partitions. >>> >>> We've elected to skip this locally and create a filesystem on the iSCSI >>> device directly. >>> >>> Creating and mounting filesystems are both generally managed on the >>> initiator. >>> >>> Truth is, there's a lot of flexibility with iSCSI, but not a lot of >>> guidance as to best practices that I could find. Vendor docs have >>> tended to be very poor. Above is my recommendation, and should >>> generally work. Alternate configurations are almost certainly possible, >>> and may be preferable. >> >> If a best practices doc could be handed to you right now, what would >> you like it to contain? > > <grin> > > I've got about 35 pages of that document sitting on my local HD now. > Negotiating with management about releasing some of it in some form or > another. > > It's based on some specific vendor experience (Dell MD3200i), and > CentOS. Among the problems: I suspect there's a range of experiences > and configuration issues. > >> I would suspect that it would be different whether your setting up an >> initiator or a target, so maybe start by splitting it into two >> sections. > > Absolutely -- if you're setting up your own target, you've got a set of > problems (and opportunities) not facing those with vendor-supplied kit > with its various capabilities and limitations. > > > Basics: > > - Terminology. I found the Wikipedia article and the Open-iSCSI > README to be particularly good here. > > - Presentation of the devices. It took us a while to realize that we > were going to get both a set of raw SCSI devices, AND a set of > multipath (/dev/dm-<number>) devices, and that it was the latter we > wanted to play with. We spent a rediculous amount of time trying to > debug and understand what turned out to be expected and correct > behavior, though this was either undocumented or very unclearly > documented. How much of this is specific to the MD3xxxi kit with > its multiple ports and controllers isn't clear to me. > > - Basic components. For us these were vendor tools (and identifying > which of these were relevant was itself non-trivial), iscsiadm, > multipath, and the CentOS netfs config scripts / environment. > The device-mapper-mulitpath docs are shite. > > - Overview of target and initiator setup: phyisical, cabling, network > topology (a dedicated switched LAN or vLAN being recommended, away > from core network traffic). > > - As appropriate: multipath, its role, where it is/isn't needed, what > it provides. > > - Target-side set-up, including virtual disk creation, RAID > configuration, network configuration, host-to-LUN mapping, IQN > generation / identification, CHAP configuration (if opted), etc. > > I'd slice this into Linux/CentOS target configuration (specific > tasks), preceded by a more generic "things you'll want to take care > of" section. If people/vendors want to fill in their specific > configuration procedures in additional sub-sections, that would be > an option. > > - Initiator-side set-up: installation of base packages, verifying > iscsi functionality, verifying multipath functionality, verifying > netfs functionality. Preparing storage (partitioning, filesystem > creation). > > - Target discovery from initiator. CHAP configuration (if opted), > session log-in, state querying, log out. DiscoveryDB querying, > update, manipulation. > > - Configuring/disabling persistent / on-login iscsi sessions. > > - Not applicable to us, but booting to an iscsi-mounted root > filesystem. > > - Log / dmesg error/informative messages, interpretation, debugging. > > - What to do when things go wrong. An all too infrequent > documentation section. > > - Monitoring and assessment. "How can I tell if I'm properly > configured", target/initiator-side error/issue reporting. Good ways > to tie into existing reporting systems/regimes (SNMP, Nagios, Cacti, > ...). > > - Peformance issues. > > - Security. Best I can tell this divides into: > - Session authentication (CHAT/CHAP) > - Target/initiator host security (specific to protocols used to > access either). > - Physical and network security -- not necessarily related, but the > point being that iSCSI traffic and hardware should generally be > isolated. I'm not aware of encrypted network transports, though I > suspect some sort of socket forwarding or IPSEC could be layered > in. > - Fileystem encryption. I'd suspect this would be managed > client-side, though implications on network datastreams should > probably be noted. Hrm... if it _is_ handled client-side, the > network traffic should by ciphertext, not clear. > > If nothing else, this should make a good framework for developing useful > docs -- the Christmas tree from which to hang the ornaments. It's > rather much how my own document started -- unanswered questions / > unclear elements of the configuration, filled in by research and > experience. > > >> I would be happy to draft something up and put it on a wiki somewhere, >> but I would need a list of talking points to start with. > > How's this do you? > In our configuration, we are going to have our iSCSI targets and initiators all connected to the same layer 3 switch and isolate the iSCSI traffic on separate networks. Would it be beneficial to also set up multipath for this as well?