Hi all.
I have a new server and am playing with the installation. (centos 6). My onboard sata card has 6 drives attached to it, yet the server has 8 bays. I bought a sata controller card (adaptec) and added it to the mix, adding two more drives.
Everything works and all is accessible, but there is one annoying issue.
When installing with anaconda, the 2 drives located on the add-on sata card are being listed as sda and sdb instead of going to the back of the line.
I would rather my a and b drives be the 0 and 1 sata ports of the onboard controller, but it seems the add-on is taking priority.
My only thought is to install without the 2 drives connected and then add them afterwords. But again, that makes no sense to have to do it that way.
Is this just normal for the add on sata cards? If I add the drives to the system after I install it all, will linux barf on it and change the drive letters?
I do not want my raid 1 mirror OS to be on sdc, sdd, and sde....it just looks weird.
thanks
On Sep 28, 2011, at 4:47 PM, Bob Hoffman wrote:
Hi all.
I have a new server and am playing with the installation. (centos 6). My onboard sata card has 6 drives attached to it, yet the server has 8 bays. I bought a sata controller card (adaptec) and added it to the mix, adding two more drives.
Everything works and all is accessible, but there is one annoying issue.
When installing with anaconda, the 2 drives located on the add-on sata card are being listed as sda and sdb instead of going to the back of the line.
I would rather my a and b drives be the 0 and 1 sata ports of the onboard controller, but it seems the add-on is taking priority.
My only thought is to install without the 2 drives connected and then add them afterwords. But again, that makes no sense to have to do it that way.
Is this just normal for the add on sata cards? If I add the drives to the system after I install it all, will linux barf on it and change the drive letters?
I do not want my raid 1 mirror OS to be on sdc, sdd, and sde....it just looks weird.
---- get over it - it really doesn't matter and you seemingly want to do cartwheels & headstands to fight the system.
It may actually be as simple as re-ordering the drives in BIOS but only you can get into that.
If you install with the extra drives removed, you probably will have a difficult time booting once you put them back in (probably will have to 'grub-install /dev/sda' and edit /boot/grub/grub.conf to change the drive discovery).
If you install with the extra drives inserted and then remove them later, you may also run into the same issues with booting.
You really should see if you can fix the drive ordering within BIOS itself.
If not, you should probably just live your mirrors on drives other than /dev/sda & /dev/sdb
Craig
Craig,
I think you misunderstand. It is just the lettering. The extra sata card is not going to be booted from. The bios only sets the boot order, not which drive letter linux is going to assign it. it is the drive letters that just annoy me and was hoping for an easy changeable fix. I was surprised anaconda installer had no option to change them.
It is like installing windows on your f drive and having c and d as storage drives...just rubs ya wrong.
It boots fine from sdc with or without card in it...it is centos labeling based on how it looks at the hardware. Apparently an add on sata card comes first in the order linux analyzes the system.
On Wed, Sep 28, 2011 at 7:05 PM, Bob Hoffman bob@bobhoffman.com wrote:
Craig,
I think you misunderstand. It is just the lettering. The extra sata card is not going to be booted from. The bios only sets the boot order, not which drive letter linux is going to assign it. it is the drive letters that just annoy me and was hoping for an easy changeable fix. I was surprised anaconda installer had no option to change them.
It is like installing windows on your f drive and having c and d as storage drives...just rubs ya wrong.
It boots fine from sdc with or without card in it...it is centos labeling based on how it looks at the hardware. Apparently an add on sata card comes first in the order linux analyzes the system.
If you are doing raid, once the partitions are set up you will only see /dev/md? device names except in madam commands or when looking at /proc/mdstat. And you can move the drives around later so the actual device names would be different without affecting the md assemblies. I think you are being a little picky.... You could always move the controller cables around, but I'm not sure there is any guarantee that the drives will be detected in the same order every time.
Les Mikesell wrote ----------------------------------------------------------------------------
If you are doing raid, once the partitions are set up you will only see /dev/md? device names except in madam commands or when looking at /proc/mdstat. And you can move the drives around later so the actual device names would be different without affecting the md assemblies. I think you are being a little picky.... You could always move the controller cables around, but I'm not sure there is any guarantee that the drives will be detected in the same order every time. --------------------------------------------------------------
I am just worried that it would affect some programming or scripting, but if that is not the case then it will be fine....I hope. I looked into the guts of etc and boot and grub...they proper drives are placed hd0 hd1 , etc... Those things look good.
It just seems changing that lettering would be scary, guess linux uses them as kind of a virtual naming convention while keeping the underlying info the same (hd0, etc..).
well, nothing I can do to change it I guess, so I will move on.
Thanks for the help all. Definitely an interesting thing to look at tonight.
bob
On Wed, Sep 28, 2011 at 08:05:53PM -0400, Bob Hoffman wrote:
It is like installing windows on your f drive and having c and d as storage drives...just rubs ya wrong.
Fortunately, linux is not so braindead that the actual drive assignments cause any problems. Just don't worry about it. The only time you might experience problems is if the drives get different assignments later-- I have seen this, for example, when doing a fairly major kernel update, if drivers get loaded in a different order than previously. But this would be something you'd look for after doing a kernel update, so it shouldn't be a problem either way.
As Les already mentioned, nothing in CentOS will care that your boot drive is sdc as long as it's properly configured to boot. It sounds like it is working for you, so you should just leave the drive assignments the way they are--it's not worth the effort to try to change it, because it won't get you any benefit.
--keith
From: Bob Hoffman bob@bobhoffman.com
When installing with anaconda, the 2 drives located on the add-on sata card are being listed as sda and sdb instead of going to the back of the line.
Check in the bios if it proposes a ctrl detection order.
JD
On Wednesday, September 28, 2011 07:47:15 PM Bob Hoffman wrote:
I do not want my raid 1 mirror OS to be on sdc, sdd, and sde....it just looks weird.
It's related to PCI enumeration order, and may not be changeable. You could try the add-on card in another slot.
However, if you think that's weird, I want you to note the portion of the output of mount below, and note the drive device my /boot is on (and yes, that is actually the real booting drive):
[root@www ~]# mount|grep boot /dev/sdag1 on /boot type ext4 (rw) [root@www ~]# cat /etc/issue Red Hat Enterprise Linux Server release 6.1 (Santiago) Kernel \r on an \m
[root@www ~]#
----- Original Message -----
From: "Lamar Owen" lowen@pari.edu To: "CentOS mailing list" centos@centos.org Sent: Thursday, September 29, 2011 1:36:28 PM Subject: Re: [CentOS] add on sata card relabeling drives, installation
On Wednesday, September 28, 2011 07:47:15 PM Bob Hoffman wrote:
I do not want my raid 1 mirror OS to be on sdc, sdd, and sde....it just looks weird.
It's related to PCI enumeration order, and may not be changeable. You could try the add-on card in another slot.
However, if you think that's weird, I want you to note the portion of the output of mount below, and note the drive device my /boot is on (and yes, that is actually the real booting drive):
[root@www ~]# mount|grep boot /dev/sdag1 on /boot type ext4 (rw) [root@www ~]# cat /etc/issue Red Hat Enterprise Linux Server release 6.1 (Santiago) Kernel \r on an \m
[root@www ~]# _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
This type of issue is why relying on /dev/sdX is bad. Mounting based on uuid or label when available is best. Unfortunately, there are controller cards that present all disks as the same uuid. It makes using mdadm that can only see /dev/sdX a pain to use.
David.
On Thu, Sep 29, 2011 at 4:09 PM, David C. Miller millerdc@fusion.gat.com wrote:
This type of issue is why relying on /dev/sdX is bad. Mounting based on uuid or label when available is best. Unfortunately, there are controller cards that present all disks as the same uuid. It makes using mdadm that can only see /dev/sdX a pain to use.
So how do you propose getting a uuid or label on a disk in the first place if you can't identify which is which physically? And how do you know which to move when you want the content in some other box?
Les Mikesell wrote:
On Thu, Sep 29, 2011 at 4:09 PM, David C. Miller millerdc@fusion.gat.com wrote:
This type of issue is why relying on /dev/sdX is bad. Mounting based on uuid or label when available is best. Unfortunately, there are controller cards that present all disks as the same uuid. It makes using mdadm that can only see /dev/sdX a pain to use.
So how do you propose getting a uuid or label on a disk in the first place if you can't identify which is which physically? And how do you know which to move when you want the content in some other box?
When I build, our PXEboot ks partitions and labels the partitions. When I add or replace, I make the partition, the fs, and e2label them. I've gotten to really appreciate labeling. I hate the UUIDs - they're ludicrously too long, and bear no relationship to what device they are, or where they go.
mark
On Thu, Sep 29, 2011 at 4:19 PM, m.roth@5-cent.us wrote:
This type of issue is why relying on /dev/sdX is bad. Mounting based on uuid or label when available is best. Unfortunately, there are controller cards that present all disks as the same uuid. It makes using mdadm that can only see /dev/sdX a pain to use.
So how do you propose getting a uuid or label on a disk in the first place if you can't identify which is which physically? And how do you know which to move when you want the content in some other box?
When I build, our PXEboot ks partitions and labels the partitions. When I add or replace, I make the partition, the fs, and e2label them. I've gotten to really appreciate labeling. I hate the UUIDs - they're ludicrously too long, and bear no relationship to what device they are, or where they go.
What happens when you move the disks around among machines? Or don't you ever do that after they contain data?
Les Mikesell wrote:
On Thu, Sep 29, 2011 at 4:19 PM, m.roth@5-cent.us wrote:
This type of issue is why relying on /dev/sdX is bad. Mounting based on uuid or label when available is best. Unfortunately, there are controller cards that present all disks as the same uuid. It makes using mdadm that can only see /dev/sdX a pain to use.
<snip>
When I build, our PXEboot ks partitions and labels the partitions. When I add or replace, I make the partition, the fs, and e2label them. I've gotten to really appreciate labeling. I hate the UUIDs - they're ludicrously too long, and bear no relationship to what device they are, or where they go.
What happens when you move the disks around among machines? Or don't you ever do that after they contain data?
Other than offline backups, we don't move disks, other than to replace.
mark
Hi Les, On 29/09/11 22:25, Les Mikesell wrote:
On Thu, Sep 29, 2011 at 4:19 PM,m.roth@5-cent.us wrote:
When I build, our PXEboot ks partitions and labels the partitions. When I add or replace, I make the partition, the fs, and e2label them. I've gotten to really appreciate labeling. I hate the UUIDs - they're ludicrously too long, and bear no relationship to what device they are, or where they go.
What happens when you move the disks around among machines? Or don't you ever do that after they contain data?
Why would you move disks around machines unless you're recovering them after a failure? Then just make sure they don't exist on the recovery server.
Maybe it's the way the machines I get involved are used, they're mostly database servers and their lifetime are measured in 3-5 years so once they're up and running, not a lot of people touches them. If a disk is being moved around, it gets decomissioned and wiped out first, not after.
Also if you stick to more descriptive labels I think you'd be safe over the long run. Just don't call all of them "data". :-)
On Fri, Sep 30, 2011 at 4:56 AM, Hakan Koseoglu hakan@koseoglu.org wrote:
What happens when you move the disks around among machines? Or don't you ever do that after they contain data?
Why would you move disks around machines unless you're recovering them after a failure?
Because I can. Why wouldn't you? Mine are nearly all in swappable carriers and it is a lot faster to move them than to ship data any other way.
Then just make sure they don't exist on the recovery server.
And how would I know that?
Maybe it's the way the machines I get involved are used, they're mostly database servers and their lifetime are measured in 3-5 years so once they're up and running, not a lot of people touches them. If a disk is being moved around, it gets decomissioned and wiped out first, not after.
Some of our machines are like that, some aren't.
Also if you stick to more descriptive labels I think you'd be safe over the long run. Just don't call all of them "data". :-)
That doesn't any more sense than having to label all your shipping containers descriptively before you know what you are going to put in them. And besides, most of the labels are applied by the installer without user input.
Les Mikesell wrote:
On Fri, Sep 30, 2011 at 4:56 AM, Hakan Koseoglu hakan@koseoglu.org wrote:
What happens when you move the disks around among machines? Or don't you ever do that after they contain data?
Why would you move disks around machines unless you're recovering them after a failure?
Because I can. Why wouldn't you? Mine are nearly all in swappable carriers and it is a lot faster to move them than to ship data any other way.
<snip> Ok, ours go into use, and stay in their servers. Besides, we have Dells, and Penguins (several different models), and the few Suns, and the HP, and EVERY BLOODY MANUFACTURER not only has their own sleds, but they *change* them....
mark, with another dozen sleds to remove and unscrew in preparation for sanitizing....
On Friday, September 30, 2011 11:41:02 AM Les Mikesell wrote:
Because I can. Why wouldn't you?
...
That doesn't any more sense than having to label all your shipping containers descriptively before you know what you are going to put in them. And besides, most of the labels are applied by the installer without user input.
While finding the corner cases seems to be your specialty, Les, recognize that there will always be a corner case not covered by any filesystem labeling/naming scheme, no matter what scheme is used.
It's not at all hard to change the labels after the install.
Linux disk device names aren't reliable (with iSCSI and other SAN technologies they never have been).
LVM has name collision issues (if two sets of one volume group name are found, hilarity ensues, part of the reason why LVM volume group names picked by the installer are now in EL6 based on hostname and not just generic names as before).
Labels of course have their own collision issues, but a label is the one thing that is the most easily modified by the user; use the chosen filesystem's labeling command (e2label for ext2/3/4; other filesystems have their own) and change it in /etc/fstab as well; next reboot it will get picked up. Labels have serious multipathing issues.
UUID is, IMHO at least, the worst of all worlds due to the length and the user-unfriendliness of it all (it's been the Ubuntu default for a while, though!). It is guaranteed unique (until you use complete clones), but is the most difficult to change and use.
Doing it by controller, channel, and logical unit makes a lot of sense until you change things around a few times (and with SAN technologies change is very easy). My boot drive on one box gets a new drive 'letter' (yuck, DOSism at its worst!) nearly every boot due to the highly dynamic and multipathed nature of of the SAN fabric connection to it, and the fibre channel HBA is being enumerated before the boot RAID controller (3Ware).
And it needs to be that way because of the different PCI-X speeds involved, as well as cable lengths and clearance issues inside the 2U server's chassis. But having /dev/sdag as my boot drive doesn't bother me in the least; everything is either LVM or label-based mounting, and I haven't had any collision problems (but multipath problems are a different story).
But my multipathing issues relate to my situation being one of those corner cases (the normal multipathing assumes an A and a B side redundancy from HBA ports through the fabric to the storage processors to the backend loops; while I will soone be there I am not right now, with machines seeing four paths to every LUN and not just two). When I get things into the recommended dual-path HA state either the standard EL-provided multipathing or the EMC PowerPath routing will work as designed, so I can't actually complain that my situation is working properly.
On Fri, Sep 30, 2011 at 11:03 AM, Lamar Owen lowen@pari.edu wrote:
While finding the corner cases seems to be your specialty, Les, recognize that there will always be a corner case not covered by any filesystem labeling/naming scheme, no matter what scheme is used.
I've found that it is a good idea to find concepts and implementations that weren't very well thought out before they bite you somewhere, so yes, I do go out of my way. For example when mounting by label was first implemented, having a duplicate label (very likely if you move disks around at all since the installer always used the same labels) would keep the system from booting at all. You had to just say 'what were they thinking...' - and wonder about the rest of the system.
It's not at all hard to change the labels after the install.
To what? It's something that is going to hold some data in the future. And you may not know you need to re-mount it until the machine that labeled it is gone or dead and the drive is all that is left.
Within 5.x I've found auto-assembled md devices to be pretty reliable at identifying themselves, but booting the 6.x livecd completely messed that up on the one machine where I tried it.
On Friday, September 30, 2011 12:26:28 PM Les Mikesell wrote:
On Fri, Sep 30, 2011 at 11:03 AM, Lamar Owen lowen@pari.edu wrote:
For example when mounting by label was first implemented, having a duplicate label (very likely if you move disks around at all since the installer always used the same labels) would keep the system from booting at all. You had to just say 'what were they thinking...' - and wonder about the rest of the system.
Again I'll say that no matter what scheme were to be used there are issues and problems. 'What were they thinking?' is something that obviously the nameless 'they' must answer for themselves, but at the same time I'm reminded of the old engineering adage 'the better is the enemy of the good enough' meaning that while you can always make a product 'better' you must recognize when it is good enough for the targeted use case. And if your particular corner case is not the targeted use case... well, things do break. Try not to have known corner cases or be prepared to work around the breakage.
But 'breakage' and 'bugginess' are not synonyms; something can be broken for a corner case but not be a bug in the general sense. Is the current filesystem mounting standard broken? In certain use cases most certainly. Is the current filesystem mounting standard buggy? For the targeted use cases probably not. After all, upstream developers and CentOS builders all operate within finite resource limits; it takes infinite resources to reach perfection.
It's not at all hard to change the labels after the install.
To what? It's something that is going to hold some data in the future. And you may not know you need to re-mount it until the machine that labeled it is gone or dead and the drive is all that is left.
The only truly unique identifier belonging to the drive and externally visible is the drive serial number. Or you can literally and physically label the disk with information about its filesystems; I've both seen that done and have done it in certain hotswap cases.
Within 5.x I've found auto-assembled md devices to be pretty reliable at identifying themselves, but booting the 6.x livecd completely messed that up on the one machine where I tried it.
There seem to be enough differences in the md scheme of 5.x and 6.x to discourage disk interchange among the two in mdraid cases. Having said that, I have an EL6.1 (upstream EL) machine with this: [root@www ~]# cat /proc/mdstat Personalities : [raid1] md127 : active raid1 sdae1[0] sdaf1[1] 732570841 blocks super 1.2 [2/2] [UU]
unused devices: <none> [root@www ~]#
Yeah, md127. But it works reliably, so why change it? When LUNs go away or whatnot, the member disks, between boots, will change in terms of device name (for a while they were /dev/sdw and /dev/sdx, then I added some LUNs to the fibre channel and they went to /dev/sdz and /dev/sdaa; I've added a LUN or two since then (and thanks to multipathing) they are now at /dev/sdae and /dev/sdaf; the mirror hasn't broken. And this md set was created under CentOS 5 a couple of years back.
This would definitely break things if mount points in /etc/fstab are keyed by md number; that's not the case here, the filesystem is mounted by label.
But is that buggy? Depends entirely on use case.
On Fri, Sep 30, 2011 at 11:57 AM, Lamar Owen lowen@pari.edu wrote:
But 'breakage' and 'bugginess' are not synonyms; something can be broken for a corner case but not be a bug in the general sense. Is the current filesystem mounting standard broken? In certain use cases most certainly. Is the current filesystem mounting standard buggy? For the targeted use cases probably not.
I think the first incarnation of the 'labels in fstab' that I saw would have died a horrible death if you did a dual boot install with 2 copies of it, something that should have been planned as a normal use case. That might have been a fedora version though, and I'm not sure what happens in that case now.
After all, upstream developers and CentOS builders all operate within finite resource limits; it takes infinite resources to reach perfection.
But there are only 2 hard problems in computer science: naming things, cache invalidation, and off-by-one errors. (Hmmm, can't find the right attribution for that now).
On Sat, Oct 1, 2011 at 5:03 AM, Lamar Owen lowen@pari.edu wrote:
UUID is, IMHO at least, the worst of all worlds due to the length and the user-unfriendliness of it all (it's been the Ubuntu default for a while, though!). It is guaranteed unique (until you use complete clones), but is the most difficult to change and use.
Surely it is a single command? Two if you have to generate the UUID. On my home machine I cloned my boot drive, then set the UUID of the second drive with:
prompt> uuidgen then cut and paste into prompt> tune2fs /dev/sdb1 -U c491d94e-7004-4b08-9993-4c9a7a25b6b1 and check it with prompt> blkid
Cheers,
Cliff
On Saturday, October 01, 2011 12:56:46 AM Cliff Pratt wrote:
prompt> tune2fs /dev/sdb1 -U c491d94e-7004-4b08-9993-4c9a7a25b6b1
As the saying goes, try typing that fast ten times.... and see how many times the UUID ends up being fat-fignered.
Unless the UUID contains spellable words that use only the hex digits (like deadbeef, cafebabe, or similar). (you can find a list of 1196 hex words at http://nedbatchelder.com/text/hexwords.html )
Mnemonics are essential for jogging the memory... oh, wait....
"Now, was that filesystem with the backup copy of that priceless one-in-a-lifetime video c491d94e-7004-4b08-9993-4c9a7a25b6b1 or was it bb6c2bb9-f01e-3135-a8de-9f885a7afdef or maybe it was f82ffa31-2587-3db8-970a-36e54e72621b... oh, I don't remember!"
But I guess if you physically label the disk with the partitioning and the UUID's of each filesystem, it might be workable.
Too bad many, if not most, drive serial numbers are not spellable in hex....
On Sun, Oct 2, 2011 at 5:24 AM, Lamar Owen lowen@pari.edu wrote:
On Saturday, October 01, 2011 12:56:46 AM Cliff Pratt wrote:
prompt> tune2fs /dev/sdb1 -U c491d94e-7004-4b08-9993-4c9a7a25b6b1
As the saying goes, try typing that fast ten times.... and see how many times the UUID ends up being fat-fignered.
I said, in a bit that you snipped, cut-and-paste.
Unless the UUID contains spellable words that use only the hex digits (like deadbeef, cafebabe, or similar). (you can find a list of 1196 hex words at http://nedbatchelder.com/text/hexwords.html )
Mnemonics are essential for jogging the memory... oh, wait....
"Now, was that filesystem with the backup copy of that priceless one-in-a-lifetime video c491d94e-7004-4b08-9993-4c9a7a25b6b1 or was it bb6c2bb9-f01e-3135-a8de-9f885a7afdef or maybe it was f82ffa31-2587-3db8-970a-36e54e72621b... oh, I don't remember!"
That's silly. The UUID is probably only of interest when the disk or partition is being mounted. If it isn't mounted, mount it and *look*.
But I guess if you physically label the disk with the partitioning and the UUID's of each filesystem, it might be workable.
Too bad many, if not most, drive serial numbers are not spellable in hex....
Cheers,
Cliff
On 9/30/2011 8:41 AM, Les Mikesell wrote:
On Fri, Sep 30, 2011 at 4:56 AM, Hakan Koseogluhakan@koseoglu.org wrote:
Why would you move disks around machines unless you're recovering them after a failure?
Because I can. Why wouldn't you? Mine are nearly all in swappable carriers and it is a lot faster to move them than to ship data any other way.
Because you are wearing the machine's connectors out. They are rated to be *infrequently* changed out. When you do it on a regular basis it will just be a matter of time until they develop electrical/physical problems.
If you want to use drives to ship data around plug in a USB hub and connect USB drives to it. That way when the connectors inevitably wear out all you need to replace is the hub (and/or the drives).
On Fri, Sep 30, 2011 at 11:20 AM, Benjamin Franz jfranz@freerun.com wrote:
Why would you move disks around machines unless you're recovering them after a failure?
Because I can. Why wouldn't you? Mine are nearly all in swappable carriers and it is a lot faster to move them than to ship data any other way.
Because you are wearing the machine's connectors out. They are rated to be *infrequently* changed out. When you do it on a regular basis it will just be a matter of time until they develop electrical/physical problems.
Source? The numbers I've seen are on the order of 50,000 insertions.
Benjamin Franz wrote:
On 9/30/2011 8:41 AM, Les Mikesell wrote:
On Fri, Sep 30, 2011 at 4:56 AM, Hakan Koseogluhakan@koseoglu.org wrote:
Why would you move disks around machines unless you're recovering them after a failure?
Because I can. Why wouldn't you? Mine are nearly all in swappable carriers and it is a lot faster to move them than to ship data any other way.
Because you are wearing the machine's connectors out. They are rated to be *infrequently* changed out. When you do it on a regular basis it will just be a matter of time until they develop electrical/physical problems.
<snip> Most of our servers have all drives in hot swap bays (and the older ones that don't are being surplussed as fast as we can)... *ALL* of which have sleds they have to fit in. The only drives I swap on a regular basis are our offline backups (of the online backups), and that's every two weeks, and for that I've got a dual bay eSATA base, just drop them in, then push it up. Nothing else moves until it dies.
mark
On Fri, Sep 30, 2011 at 12:55 PM, m.roth@5-cent.us wrote:
Most of our servers have all drives in hot swap bays (and the older ones that don't are being surplussed as fast as we can)... *ALL* of which have sleds they have to fit in. The only drives I swap on a regular basis are our offline backups (of the online backups), and that's every two weeks, and for that I've got a dual bay eSATA base, just drop them in, then push it up.
If 750gb disks are big enough, you can get a cute little internal trayless hot swap bay for 2 - 2.5" SATA drives that fits in the space a 3.5" floppy would have taken. The WD ''Scorpio Black" drives are pretty snappy - and you can toss your backup in your shirt pocket. I think someone even has a 1 Tb drive in the standard laptop height now. Until recently there were 2.5" 1 and 1.5 Tb drives but they were too tall for standard enclosures.
Les Mikesell wrote:
On Fri, Sep 30, 2011 at 12:55 PM, m.roth@5-cent.us wrote:
Most of our servers have all drives in hot swap bays (and the older ones that don't are being surplussed as fast as we can)... *ALL* of which have sleds they have to fit in. The only drives I swap on a regular basis are our offline backups (of the online backups), and that's every two weeks, and for that I've got a dual bay eSATA base, just drop them in, then push it up.
If 750gb disks are big enough, you can get a cute little internal trayless hot swap bay for 2 - 2.5" SATA drives that fits in the space a 3.5" floppy would have taken. The WD ''Scorpio Black" drives are pretty snappy - and you can toss your backup in your shirt pocket. I think someone even has a 1 Tb drive in the standard laptop height now. Until recently there were 2.5" 1 and 1.5 Tb drives but they were too tall for standard enclosures.
#insert "rocking_chair.h" Why, Ah remember when my bosses at a job long ago gave me a *big*, brand new drive as a holiday gift, knowing I'd be working at home. Why, it was all of 30MB!
Sorry, the 750GB's are going, going, gone, and even the 1TB's are "smaller", except for the HPC clusters, which don't need a lot of disk. And no, I do not toss my backups at work in my shirt: the offline backups, fully encrypted disks, go in the fire safe in the locked server room. Some of the systems they're backing up have HIPAA and PII data.
mark "why, yes, I *am* where some of your US tax dollars are going"
On Fri, Sep 30, 2011 at 1:39 PM, m.roth@5-cent.us wrote:
And no, I do not toss my backups at work in my shirt:
Figuratively speaking, of course - the 2.5" drives are just easier for any use where the capacity makes sense.
the offline backups, fully encrypted disks, go in the fire safe in the locked server room. Some of the systems they're backing up have HIPAA and PII data.
Do you encrypt at the disk level (hardware support?), the filesystem, or just per file with tar-type files? And has the technique ever caused data loss?
Les Mikesell wrote:
On Fri, Sep 30, 2011 at 1:39 PM, m.roth@5-cent.us wrote:
And no, I do not toss my backups at work in my shirt:
Figuratively speaking, of course - the 2.5" drives are just easier for any use where the capacity makes sense.
Um, we just went from the 1TB drives to 3TB drives, so I only need four.
the offline backups, fully encrypted disks, go in the fire safe in the locked server room. Some of the systems they're backing up have HIPAA and PII data.
Do you encrypt at the disk level (hardware support?), the filesystem, or just per file with tar-type files? And has the technique ever caused data loss?
As I said, the disks are fully encrypted, using LUKS. Can't do hardware support - these are just ordinary drives. Never had data loss, unless the drive started failing.
mark
On 09/30/11 12:27 PM, m.roth@5-cent.us wrote:
Um, we just went from the 1TB drives to 3TB drives, so I only need four.
yes, but 12 2.5" 1TB drives take the same amount of space (1U), and are capable of higher IO throughput due to being more spindles.
John R Pierce wrote:
On 09/30/11 12:27 PM, m.roth@5-cent.us wrote:
Um, we just went from the 1TB drives to 3TB drives, so I only need four.
yes, but 12 2.5" 1TB drives take the same amount of space (1U), and are capable of higher IO throughput due to being more spindles.
You missed what I originally said: I have a two-bay eSATA dock, like this http://www.bizrate.com/hard-drives/2095977853.html. And it's only on one channel. It's not a high priority, so it takes me the better part of a week to do the backups *shrug*, it also doesn't strain the network, esp. when some folks are moving *large* amounts of data (we do actual scientific computing here). Network lag would be very much of a bad thing.
mark
On 29/09/11 22:19, m.roth@5-cent.us wrote:
When I build, our PXEboot ks partitions and labels the partitions. When I add or replace, I make the partition, the fs, and e2label them. I've gotten to really appreciate labeling. I hate the UUIDs - they're ludicrously too long, and bear no relationship to what device they are, or where they go.
Although I'm a fan of labelling, for some weird reason I had seriously weird issues with multipath SAN setups in the last couple of months. I'm on holiday at the moment so I can't the logs up but more than once, with multipath, labels have caused me too much headache than their worth. In one instance the label would latch to one of the individual paths, not the multipath and then all hell would break loose. Any suggestions on the list are much welcome.
I haven't found a new good practice yet, UUIDs are pretty unwieldly and no one can expect to remember one whereas a label of "database" or "redo1" or "redo2" are just meaningful and can be parsed by a normal human! :)
On Thursday, September 29, 2011 05:16:16 PM Les Mikesell wrote:
So how do you propose getting a uuid or label on a disk in the first place if you can't identify which is which physically? And how do you know which to move when you want the content in some other box?
Drive model number plus serial number. Really the only way; when putting systems together you just need to note the drive model and serial number(s).
-- Lamar Owen wrote
Drive model number plus serial number. Really the only way; when putting systems together you just need to note the drive model and serial number(s)
--
That seems like an important thing.
After some research I have decided the only two options are
1) to leave as is and deal with any software (such as monitoring tools, mdadm reports, etc) as they come and try to fix them.
2) You can actually change the order that UDEV looks at things and you can assign permanent drive letters AFTER the install. However, there is not much documentation on that, so option 1 seems easiest.
I do not understand the technical reasons why linux decided to dynamically label things sda, sdb, etc. After looking at udev files I find the drives are listed by UUID and a deeper labeling system like hd0, hd1, hd2.
So why even have the labels in the first place if the system doesn't use them, you cannot rely on them, and apparently they don't matter? It would be better for the anaconda installer to just list them as hd0, hd1, hd2, etc.
Where this is an issue is when you are cloning drives, adding grubs, etc...because all my instructions rely on using sda, sdb, etc...and not UUID or other things. And reports from mdadm (and all tech instructions) use the sda sdb.
IT is what it is. I am sure there is a reason. Just odd.
Well, first thing is I got lucky and not bought all the same drives. If I had all the same I would never have known I put my OS on the two drives added with the new sata card, something I DID NOT want to do.
If they were all 1 tb drives, it would have been a disaster to me. Luckily I set up P0-P2 drives with my 500gb and saw the issue immediately.
Below is the issue I am talking about. The system ignores the sda/sdb etc labeling to use UUID and things like hd0, hd1... Yet mdstat shows you the drives in the useless labeling way...sda sdab
--------------------------- lamar owen wrote There seem to be enough differences in the md scheme of 5.x and 6.x to discourage disk interchange among the two in mdraid cases. Having said that, I have an EL6.1 (upstream EL) machine with this: [root at www ~]# cat /proc/mdstat Personalities : [raid1] md127 : active raid1 sdae1[0] sdaf1[1] 732570841 blocks super 1.2 [2/2] [UU] -----------------------------
taking out a dead drive and adding a new one negates a real ability to re-label them as you want since the new drive would have a different UUID/serial, etc..and screw up any label system you might have done.
That mdstat is not the only monitoring software that pays attention to the 'useless' labels like sda sdb sdc, etc...
and that leads to my fear of not getting accurate reports on problems with drives or partitions...
(in my case I have 8 drives with 2 raid1 arrays made up of 3 drives each, each one having a hot spare....and then a few drives as backups)
I am going to just leave it alone, but I feel their is something missing in the whole theory... if the labels sda, sdb, etc mean nothing to the system admin, then why are they used in monitoring software...and worse, why are they the only drive identification listed in the installer???
And the disk manager, graphical one, in centos 6 does not list the UUID, it calls it 'worldwide ID'.
By forcing the use of UUID it makes monitoring impossible. If your scripts use the UUID, then you must know the UUID and add them manually. If you change out a drive, your script now fails to pick up the new UUID....very annoying.
I am sure it is needed, but I still see UDEV using unchanging labeling (things I would use to add grub on a drive) like 'HD0,HD1, etc.
why even use the labels, just use HD1, HD2 and the UUID and let it be.
On Friday, September 30, 2011 03:36:50 PM Bob Hoffman wrote:
Below is the issue I am talking about. The system ignores the sda/sdb etc labeling to use UUID and things like hd0, hd1... Yet mdstat shows you the drives in the useless labeling way...sda sdab
...
Useless? Those names are what the currently running kernel sees, and thus correspond to the current devices. You then have to do a little more digging with other tools, but that is what it is. The output from a cat of /proc/mdstat is raw data, and is more useful for rebuild status than anything else.
taking out a dead drive and adding a new one negates a real ability to re-label them as you want since the new drive would have a different UUID/serial, etc..and screw up any label system you might have done.
For software RAID, yes. It makes it a challenge keeping up with what device is what. However, palimpsest (the GNOME Disk Utility included in EL 6) shows more detail about RAID components; also, if you check the drive itself there is a link in the page to the array device (Go to array).
It's not quite as easy as using, say, EMC Unisphere to manage arrays, but it's not bad. EMC arrays show you on the array itself, using an amber LED, exactly which drive is faulted, and also lights the amber fault LED on the enclosure, and numbers the drives by backend bus number, enclosure number, and drive number, so that, say, drive 14 in enclosure 7 on bus 3 would be device 3_7_14 (zero origin on all these numbers). But when you can have up to 960 drives in a single array such 'luxuries' become necessities (960 drives in a CX4-960; we have a CX4-480 that only will allow 480 drives, currently populated with 230). I can't imagine managing 480 drives on a Linux box with the current device naming.
That mdstat is not the only monitoring software that pays attention to the 'useless' labels like sda sdb sdc, etc...
/proc/mdstat? It's not monitoring software; it's a kernel API mounted on the /proc filesystem; those names can and will change if disk enumeration order changes. File a bug report with the kernel developers to see this changed.
I am going to just leave it alone, but I feel their is something missing in the whole theory... if the labels sda, sdb, etc mean nothing to the system admin, then why are they used in monitoring software...and worse, why are they the only drive identification listed in the installer???
Hmm, are they? I'll have to do another install to double check, but I think the drive model numbers/types are listed, but it has been a little bit since I did the install (of my RHEL 6.1 box at least).
If you want the disks listed differently in the installer, the right thing to do is file a bug report upstream (with Red Hat, in other words) on bugzilla.redhat.com, as CentOS is 100% going to do what upstream does in this regard.
And the disk manager, graphical one, in centos 6 does not list the UUID, it calls it 'worldwide ID'.
That's because it is the WWN in the case of fibre channel, and a unique ID otherwise. WWN is unique, for fibre channel. Fibre channel: the original serial-attached SCSI. And, well, FC is giving way to dual-attach SAS in the enterprise space; 2.5 inch enterprise SAS drives in the EMC VNX series, for instance.
I have found palimpsest (the EL6 'Disk Utility' in the Applications -> System Tools menu) to give a lot of good information in one very nice display. It doesn't yet deal well with LVM; for that you use system-config-lvm. But the serial number, the filesystem label, RAID components, etc all in one place. Also, for one of the controllers I have it shows which port on the card a particular disk is attached to.
While many around this list detest GUI's on servers, I'll be honest; this is one very handy and useful utility that works quite well using ssh tunnelled X. A similar text-mode interface tool would be super nice, but someone has to write it, it's not in there right now.
I am sure it is needed, but I still see UDEV using unchanging labeling (things I would use to add grub on a drive) like 'HD0,HD1, etc.
The grub device names are the BIOS names; the grub hd0 should be the BIOS boot drive. I say 'should be' because I have had a mismatch there before. So in my example (where /boot is on /dev/sdag) it just so happens that /dev/sdag is grub's hd0, but once the kernel comes up and enumerates the disks what was first shall become last... (hmmm, seems like a quote from the world's most popular book...).
why even use the labels, just use HD1, HD2 and the UUID and let it be.
The /dev/sdX names are older than udev. Historical, (or hysterical, depending upon your PoV). HD1, HD2, etc would be far worse.
In the general case with PC hardware this is where things stand. With general PC hardware it becomes the responsibility of the individual systems administrator to simply know the machine inside and out, and document the machine thoroughly.
----- Original Message -----
From: "Les Mikesell" lesmikesell@gmail.com To: "CentOS mailing list" centos@centos.org Sent: Thursday, September 29, 2011 2:16:16 PM Subject: Re: [CentOS] add on sata card relabeling drives, installation
On Thu, Sep 29, 2011 at 4:09 PM, David C. Miller millerdc@fusion.gat.com wrote:
This type of issue is why relying on /dev/sdX is bad. Mounting based on uuid or label when available is best. Unfortunately, there are controller cards that present all disks as the same uuid. It makes using mdadm that can only see /dev/sdX a pain to use.
So how do you propose getting a uuid or label on a disk in the first place if you can't identify which is which physically? And how do you know which to move when you want the content in some other box?
-- Les Mikesell lesmikesell@gmail.com
I just had to come up with a solution for this recently. Here is what I did.
I create a small 2 block partition on each disk and gave them labels that is the drives serial number when I format them as ext3/4. I dedicate the rest of the disk to Linux auto RAID. Something like this to create a label.
mkfs.ext3 -L $DRIVE_SN /dev/sd1
I then have a script that mounts the small partitions by label to a directory with the same name as the label.
mount LABEL=$DRIVE_SN /mnt/drive-check/$DRIVE_SN
So if you do a df it will show something like.
Filesystem Size Used Avail Use% Mounted on /dev/md0 71G 3.9G 63G 6% / tmpfs 12G 0 12G 0% /dev/shm /dev/sdc1 16M 1.2M 14M 8% /mnt/drive-check/WD-WMAY03208723 /dev/sdd1 16M 1.2M 14M 8% /mnt/drive-check/WD-WMAY03287844 /dev/sde1 16M 1.2M 14M 8% /mnt/drive-check/WD-WMAY03247298 /dev/sdf1 16M 1.2M 14M 8% /mnt/drive-check/WD-WMAY03247844 /dev/sdg1 16M 1.2M 14M 8% /mnt/drive-check/WD-WMAY03674888 /dev/sdh1 16M 1.2M 14M 8% /mnt/drive-check/WD-WMAY03678644 /dev/sdi1 16M 1.2M 14M 8% /mnt/drive-check/WD-WMAY03674814 /dev/sdj1 16M 1.2M 14M 8% /mnt/drive-check/WD-WMAY03675850 /dev/sdk1 16M 1.2M 14M 8% /mnt/drive-check/WD-WMAY03675194 /dev/sdl1 16M 1.2M 14M 8% /mnt/drive-check/WD-WMAY03288196 /dev/sdm1 16M 1.2M 14M 8% /mnt/drive-check/WD-WMAY03672314 /dev/sdn1 16M 1.2M 14M 8% /mnt/drive-check/WD-WMAY03287843 /dev/sdo1 16M 1.2M 14M 8% /mnt/drive-check/WD-WMAY03674460 /dev/sdp1 16M 1.2M 14M 8% /mnt/drive-check/WD-WMAY03585344 /dev/sdq1 16M 1.2M 14M 8% /mnt/drive-check/WD-WMAY03508896 /dev/sdr1 16M 1.2M 14M 8% /mnt/drive-check/WD-WMAY03209984
Now if I'm using mdadm to make a software RAID and it is complaining /dev/sdf2 is missing. I can run my script to mount all the small partitions and the one that complains it can't mount is easily identified by the serial number. Sure I can just let the hardware RAID card handle everything but I don't trust them from past experiences seeing failed cards and corrupted arrays. With the disks seen by linux as raw block devices I can put these disks on any JBOD controller and mount my raid using mdadm. I'm not tied to a particular controller if it fails.
David.
-
Benjamin Franz -
Bob Hoffman -
Cliff Pratt -
Craig White -
David C. Miller -
Hakan Koseoglu -
John Doe -
John R Pierce -
Keith Keller -
Lamar Owen -
Les Mikesell -
m.roth＠5-cent.us