I really hope someone has a clue on this one.... :)
I've got a 2.2TB array mounted on a 3ware 9500 controller. I installed it last week ensuring that LBA was enabled and using gpt with parted to get the full size available on one array. I also think I chose reiserfs for the array. We then moved ALOT of data to the array.
Problem now is that after a reboot LBA doesn't seem to be in effect. From dmesg:
sdc : very big device. try to use READ CAPACITY(16). SCSI device sdc: 4374845440 512-byte hdwr sectors (2239921 MB) SCSI device sdc: drive cache: none sdc : very big device. try to use READ CAPACITY(16). SCSI device sdc: 4374845440 512-byte hdwr sectors (2239921 MB) SCSI device sdc: drive cache: none sdc: sdc1 Attached scsi disk sdc at scsi1, channel 0, id 0, lun 0 Attached scsi generic sg0 at scsi0, channel 0, id 0, lun 0, type 0 Attached scsi generic sg1 at scsi0, channel 0, id 7, lun 0, type 0 Attached scsi generic sg2 at scsi1, channel 0, id 0, lun 0, type 0
I can no longer mount sdc1; the error is the usual invalid superblock. parted does however list the correct size and filesystem.
Any tips on how I can get the array mounted again? And what might have changed since last boot? :) I've rolled my own kernel to get reiser support, but I haven't recompiled or done any changes...
Regards, Harald
HaraldFinnås spamcatcher@lantrix.no wrote:
I really hope someone has a clue on this one.... :) I've got a 2.2TB array mounted on a 3ware 9500 controller. I installed it last week ensuring that LBA was enabled and using gpt with parted to get the full size available on one array.
Just because you can slice (partition) an array doesn't mean the filesystem will format and/or be usable.
I also think I chose reiserfs for the array.
I would very much recommend against that on any Fedora-based distro. ReiserFS support is not tested or guaranteed.
We then moved ALOT of data to the array. Problem now is that after a reboot LBA doesn't seem to be in effect. ... I can no longer mount sdc1; the error is the usual invalid superblock.
That doesn't surprise me. Given the comments you were getting from the kernel, it sounds like there is some inconsistency in how it handles 1-2+TiB (1.1-2.2+TB) filesystems.
parted does however list the correct size and filesystem.
But Parted just reads basic information from the slice. It's not actually doing anything on the filesystem.
Any tips on how I can get the array mounted again? And what might have changed since last boot? :)
I've rolled my own kernel to get reiser support, but I haven't recompiled or done any changes...
Er, um, there are a _lot_ of considerations beyond 1TiB (1.1TB) that _must_ be considered when you build a kernel with various filesystem support. There's a reason why you should stick with the distro's kernels (regardless of distro).
Any reason you must have ReiserFS? And if so, I'd consider SuSE instead. Of course, some things don't work on SuSE-based/ReiserFS as they do on Fedora-based/Ext3, such as NFS.
Just because you can slice (partition) an array doesn't mean the filesystem will format and/or be usable.
It formatted just fine, showed up with df just fine, and happily accepted the data copied to it...
I've rolled my own kernel to get reiser support, but I haven't recompiled or done any changes...
Any reason you must have ReiserFS? And if so, I'd consider SuSE instead. Of course, some things don't work on
The box has ALOT of disks, and we didn't feel like doing all that copying just to change filesystem when if after all was working just fine. I think the box initially had suse or mdk installed which is the main reason for using reiserfs. I also read some comments on ext3 not beeing too clever with large disks, but I'm not sure if they still apply.
All the other disks which are just below the 2TB limit and has reiser works just fine; the problem is this large beast...
Regards, Harald
HaraldFinnås spamcatcher@lantrix.no wrote:
It formatted just fine, showed up with df just fine, and happily accepted the data copied to it...
I've seen that before, only for something to choke at 1-2+TiB (1.1-2.2+TB). That's why I always recommend, as a general rule, never to create filesystems bigger than 1TiB (1.1TB). There's just too many variables that can go wrong.
As much as there have been increases to support 2, 4 or even 8TiB with Ext3 and ReiserFS, I don't trust them to be universally compatible across various distros and kernels.
About the only kernel/filesystem combination I've ever had bliss with were the official SGI XFS releases for kernel 2.4. Allegedly late SuSE Linux Enterprise Server (SLES) 9 updates now has good support for XFS, but I've also seen people openly complain about things not working.
The box has ALOT of disks, and we didn't feel like doing all that copying just to change filesystem when if after
all
was working just fine.
But you formatted a _new_ filesystem. If you decided to do that, and were running CentOS, you should have stuck with Ext3. Just because your existing data is in ReiserFS, doesn't mean your new, large filesystem had to be ReiserFS. ;->
I think the box initially had suse or mdk installed which
is
the main reason for using reiserfs.
And I can understand that for your _existing_ volumes/filesystems. But for anything _new_, stick with the distro's kernel and its filesystem support.
I also read some comments on ext3 not beeing too clever with large disks, but I'm not sure if they still apply.
And I could tell you a lot about ReiserFS that would scare you. The reality is that if you have data you care about, you should stick with the distro's kernel and what it supports for all new data.
All the other disks which are just below the 2TB limit and has reiser works just fine; the problem is this large
beast...
Yes, which is why multi-terabyte support is always an open question I don't like to leave to chance.
I think the box initially had suse or mdk installed which
is
the main reason for using reiserfs.
And I can understand that for your _existing_ volumes/filesystems. But for anything _new_, stick with the distro's kernel and its filesystem support.
I see your point, but the reason for choosing reiser was trying to be consistent. I figured it might create ever bigger problems (from an admin point of view, not the os) to mix the filsystems.
But IMHO, this seems like a kernel / partition type issue, not a filesystem issue..? Or maybe I'm reading the output from dmesg incorrectly?
Yes, which is why multi-terabyte support is always an open question I don't like to leave to chance.
Good to know that all the other disks are larger than 1.6TB too then. ;)
Regards, Harald
HaraldFinnås spamcatcher@lantrix.no wrote:
I see your point, but the reason for choosing reiser was trying to be consistent.
I know it's hindsight now, but the consistency of assumption (or the assumption of consistency) can be its own issue. ;->
I figured it might create ever bigger problems (from an admin point of view, not the os) to mix the filsystems.
Why? Filesystems are distintively separate modules. The Virtual Filesystem (VFS) subsystem ensures part of that too.
I reguarly mixed Ext3 and XFS on official SGI XFS 2.4 kernels, no issues whatsoever.
The bigger issue is when you run a filesystem that is _unsupported_ on a kernel distro. ;->
But IMHO, this seems like a kernel / partition type issue, not a filesystem issue..?
There are dozens of various -- and it takes is *1*.
I.e., not only do you have the "disk label" (partition table) support issues, but then you have the general virtual filesystem (VFS) subsytem and then specific "filesystem" support issues. Again, it only takes *1* to cause issues.
I rather not tempt fate, I've seen too much breakage in 1-2+TiB support because of a random disk label, VFS or filesystem issues that still seem to creep up, almost repeatedly. The only vendor I ever saw that tested >1TiB support well was SGI for its XFS kernel 2.4 releases.
Or maybe I'm reading the output from dmesg incorrectly?
I'd have to research it's error message.
Good to know that all the other disks are larger than 1.6TB too then. ;)
There are issues at 1, 2, 4 and 8TiB. It's clear you're running into one at 2TiB, which seems to be the "most common" one.
As I said, I just avoid anything larger than 1TiB, so there's no chance.
I figured it might create ever bigger problems (from an admin point of view, not the os) to mix the filsystems.
Why? Filesystems are distintively separate modules. The Virtual Filesystem (VFS) subsystem ensures part of that too.
That other admins might try to force fsck.reiserfs on a ext3 partition for instance (or even worse things for that matter).
There ARE other admins in the world but us you know. ;)
Regards, Harald
More info in case there are any reiserfs gurus out there. Found a piece of log from an earlier boot when it actually mounted ok. I cannot understand what changes can have happened except filling up lots of the space...
Dec 27 18:39:29 somebox kernel: sdc : very big device. try to use READ CAPACITY(16). Dec 27 18:39:29 somebox kernel: SCSI device sdc: 4374845440 512-byte hdwr sectors (2239921 MB) Dec 27 18:39:29 somebox kernel: SCSI device sdc: drive cache: none Dec 27 18:39:29 somebox kernel: sdc: sdc1 Dec 27 18:39:31 somebox kernel: sdc : very big device. try to use READ CAPACITY(16). Dec 27 18:39:31 somebox kernel: SCSI device sdc: 4374845440 512-byte hdwr sectors (2239921 MB) Dec 27 18:39:31 somebox kernel: SCSI device sdc: drive cache: none Dec 27 18:39:31 somebox kernel: sdc: sdc1 Dec 27 18:42:15 somebox kernel: ReiserFS: sdc1: using ordered data mode Dec 27 18:42:15 somebox kernel: ReiserFS: sdc1: journal params: device sdc1, size 8192, journal first block 18, max trans len 1024, max batch 900, max commit age 30, max trans age 30 Dec 27 18:42:15 somebox kernel: ReiserFS: sdc1: checking transaction log (sdc1) Dec 27 18:42:16 somebox kernel: ReiserFS: sdc1: Using r5 hash to sort names
Regards, Harald
On Tue, 2006-01-03 at 20:51 +0100, Harald=?ISO-8859-1?B?RmlubuVz?= wrote:
More info in case there are any reiserfs gurus out there. Found a piece of log from an earlier boot when it actually mounted ok. I cannot understand what changes can have happened except filling up lots of the space...
Dec 27 18:39:29 somebox kernel: sdc : very big device. try to use READ CAPACITY(16). Dec 27 18:39:29 somebox kernel: SCSI device sdc: 4374845440 512-byte hdwr sectors (2239921 MB) Dec 27 18:39:29 somebox kernel: SCSI device sdc: drive cache: none Dec 27 18:39:29 somebox kernel: sdc: sdc1 Dec 27 18:39:31 somebox kernel: sdc : very big device. try to use READ CAPACITY(16). Dec 27 18:39:31 somebox kernel: SCSI device sdc: 4374845440 512-byte hdwr sectors (2239921 MB) Dec 27 18:39:31 somebox kernel: SCSI device sdc: drive cache: none Dec 27 18:39:31 somebox kernel: sdc: sdc1 Dec 27 18:42:15 somebox kernel: ReiserFS: sdc1: using ordered data mode Dec 27 18:42:15 somebox kernel: ReiserFS: sdc1: journal params: device sdc1, size 8192, journal first block 18, max trans len 1024, max batch 900, max commit age 30, max trans age 30 Dec 27 18:42:15 somebox kernel: ReiserFS: sdc1: checking transaction log (sdc1) Dec 27 18:42:16 somebox kernel: ReiserFS: sdc1: Using r5 hash to sort names
Regards, Harald
Are you using the CentOSPlus kernel or another one?
HaraldFinnås spamcatcher@lantrix.no wrote:
Rolled my own 2.6.14.5.
Which means you're dealing with more variables that one could even hope to address. Especially on an i686 release, although I wouldn't be too much happier on x86-64 either.
Bryan J. Smith wrote:
HaraldFinnås spamcatcher@lantrix.no wrote:
I really hope someone has a clue on this one.... :) I've got a 2.2TB array mounted on a 3ware 9500 controller. I installed it last week ensuring that LBA was enabled and using gpt with parted to get the full size available on one array.
Just because you can slice (partition) an array doesn't mean the filesystem will format and/or be usable.
I also think I chose reiserfs for the array.
I would very much recommend against that on any Fedora-based distro. ReiserFS support is not tested or guaranteed.
<snip>
Er, um, there are a _lot_ of considerations beyond 1TiB (1.1TB) that _must_ be considered when you build a kernel with various filesystem support. There's a reason why you should stick with the distro's kernels (regardless of distro).
Any reason you must have ReiserFS? And if so, I'd consider SuSE instead. Of course, some things don't work on SuSE-based/ReiserFS as they do on Fedora-based/Ext3, such as NFS.
Roger that last observation :-). I have been having problems w/ ReiserFS & NFS (twixt a SuSE 9.2 box & some SGI's) for a few weeks now, though they *might* be calming down a bit (after some apparently prescient recommendations from TheBS :-) ) ....
"William A. Mahaffey III" wam@HiWAAY.net wrote:
Roger that last observation :-). I have been having problems w/ ReiserFS & NFS (twixt a SuSE 9.2 box & some SGI's) for a few weeks now, though they *might* be calming down a bit (after some apparently prescient recommendations from TheBS :-) ) ....
A lot of people request that I cite various documentation on this stuff. I can't. Occasionally I'll find a vendor's bug report on various things, but outside of maybe Red Hat (who has always come under a lot of fire for heavily patching the kernel), a lot of vendors just blame things on the kernel support and refuse to address it in their distros.
That's why I have only experience to go on. And multi-TB support is not something I put my faith in under Linux. Especially not when various geometry, structure and other issues keep creeping up, have to have workaround for, only to repeat themselves in various kernel/filesystem implementations.
Bryan J. Smith wrote:
"William A. Mahaffey III" wam@HiWAAY.net wrote:
Roger that last observation :-). I have been having problems w/ ReiserFS & NFS (twixt a SuSE 9.2 box & some SGI's) for a few weeks now, though they *might* be calming down a bit (after some apparently prescient recommendations from TheBS :-) ) ....
A lot of people request that I cite various documentation on this stuff. I can't. Occasionally I'll find a vendor's bug report on various things, but outside of maybe Red Hat (who has always come under a lot of fire for heavily patching the kernel), a lot of vendors just blame things on the kernel support and refuse to address it in their distros.
That's why I have only experience to go on. And multi-TB support is not something I put my faith in under Linux. Especially not when various geometry, structure and other issues keep creeping up, have to have workaround for, only to repeat themselves in various kernel/filesystem implementations.
Knowlege acquired from hard experience doesn't need to be cited, especially when it proves to be correct :-).
"William A. Mahaffey III" wam@HiWAAY.net wrote:
Knowlege acquired from hard experience doesn't need to be cited, especially when it proves to be correct :-).
<US southern accent=on> Yer not fum 'round hear, r ya? </US southern>
Seriously now, you'll find that there are some regular characters on this list, and a few of the lesser traveled Red Hat lists (i.e., were you find virtually no Red Hat developers), that reguarly beat me for citations and challenge anything I say.
As I always say, some people are free to assume I pull things right out of my rectum. But I'm glad people like yourself can take what I say and apply your own experiences to see if I am correct or not, and base future considerations of what I say on those.
Trust is earned over years. I've been around on a few lists for years though. Not this one, but several others. ;->
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Tue, Jan 03, 2006 at 10:09:44AM -0600, William A. Mahaffey III wrote:
Roger that last observation :-). I have been having problems w/ ReiserFS & NFS (twixt a SuSE 9.2 box & some SGI's) for a few weeks now, though they *might* be calming down a bit (after some apparently prescient recommendations from TheBS :-) ) ....
Try the userspace nfsdaemon (unfs) instead of the kernel based on. It is supposed to work with ReiserFS with less problems.
Best Regards,
- -- Rodrigo Barbosa rodrigob@suespammers.org "Quid quid Latine dictum sit, altum viditur" "Be excellent to each other ..." - Bill & Ted (Wyld Stallyns)
Rodrigo Barbosa rodrigob@suespammers.org wrote:
Try the userspace nfsdaemon (unfs) instead of the kernel based on. It is supposed to work with ReiserFS with less problems.
And the NFS userspace daemon lacks all sorts of things that make matter much worse IMHO. ;->
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Tue, Jan 03, 2006 at 10:03:53AM -0800, Bryan J. Smith wrote:
Rodrigo Barbosa rodrigob@suespammers.org wrote:
Try the userspace nfsdaemon (unfs) instead of the kernel based on. It is supposed to work with ReiserFS with less problems.
And the NFS userspace daemon lacks all sorts of things that make matter much worse IMHO. ;->
Depends on what you are trying to acomplish. Yes, I would never build a supercomputer cluster with those, but for a simple setup, it works well enough.
- -- Rodrigo Barbosa rodrigob@suespammers.org "Quid quid Latine dictum sit, altum viditur" "Be excellent to each other ..." - Bill & Ted (Wyld Stallyns)
Rodrigo Barbosa rodrigob@suespammers.org wrote:
Depends on what you are trying to acomplish. Yes, I would never build a supercomputer cluster with those,
I wouldn't use it for a production network at all. No locking, lack of direct inode access, etc...
but for a simple setup, it works well enough.
Er, um, I'd have to say it's not worth it at all.
I'd rather put any data I need NFS exported on an Ext3 filesystem and avoid the entire issue altogether.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Okey, I'm moving this to linux-practices, since there are some other points I would like to discuss. :)
On Tue, Jan 03, 2006 at 10:17:21AM -0800, Bryan J. Smith wrote:
Rodrigo Barbosa rodrigob@suespammers.org wrote:
Depends on what you are trying to acomplish. Yes, I would never build a supercomputer cluster with those,
I wouldn't use it for a production network at all. No locking, lack of direct inode access, etc...
but for a simple setup, it works well enough.
Er, um, I'd have to say it's not worth it at all.
I'd rather put any data I need NFS exported on an Ext3 filesystem and avoid the entire issue altogether.
- -- Rodrigo Barbosa rodrigob@suespammers.org "Quid quid Latine dictum sit, altum viditur" "Be excellent to each other ..." - Bill & Ted (Wyld Stallyns)
-
Bryan J. Smith -
HaraldFinnås -
Johnny Hughes -
Rodrigo Barbosa -
William A. Mahaffey III