[CentOS] 8-15 TB storage: any recommendations?

Fri Jan 8 02:42:41 UTC 2010
Warren Young <warren at etr-usa.com>

On 1/7/2010 6:01 PM, Christopher Chan wrote:
>
>> I'm not recommending OpenSolaris on purpose.
>
> Serious system administrators are not Linux fans I don't think.

I think I must have been sent back in time, say to 1997 or so, because I 
can't possibly be reading this in 2010.  I base this on the fact that 
your statement logically means there are no serious Linux sysadmins, 
which is of course is so much hooey that no one believes this any more 
in the time I come from.  Therefore, I must have been sent far enough 
back in time that such statements were still uttered with complete 
seriousness.

I guess the other possibility is that someone's gatewayed a Usenet 
advocacy group to this list.

> I find pkg on OpenSolaris to be more akin to yum or apt than ports

In some ways, sure.  Ports is definitely a different way of doing 
things, though, I think, not a bad one.

There are several areas where OpenSolaris' package system falls down:

1. No free updates.  Even if you just want security fixes, you have to 
buy a support contract.  (If you think this is reasonable, why are you 
here on the CentOS list, a place for discussing a solution to a 
different but similar problem?)

2. There is no upgrade path from release to release other than 
"reinstall", and releases are spaced just 6 months apart.  Between this 
and the previous problem, it means I have to reinstall my server every 6 
months to keep up to date if I don't want to buy a support contract. 
Those serious sysadmins where you come from like this sort of thing?  In 
my world, we prefer OSes with long term support so we can stay current 
on a release for years at a time.

3. The main package repo is pretty sparse.  If you want anything even a 
little bit nonstandard you end up downloading sources from the Internet 
and compiling by hand, which may not even succeed since Solaris is down 
in the third tier or so of popularity these days.  At least with 
FreeBSD's ports, you're pretty much guaranteed that it will build and 
install with "sudo make install clean", even chasing down dependencies 
for you automatically.

4. At least back in the 2008.05 and 2008.11 days when I last tried to 
really use OpenSolaris, I found IPS to be quite immature.  I managed, 
twice, to render a machine unbootable simply by removing packages I 
thought I didn't need, using the GUI package manager.  No warnings, just 
boot...bang.  Now maybe I'm being unrealistic, but I would think one of 
the basic requirements for a package manager is that it know enough 
about dependencies to refuse to let me uninstall core system components.

After discovering all that, I'm afraid I rather lost interest in trying 
to make serious use of OpenSolaris.  I keep a VM of it around merely to 
test compatibility with a free software project I maintain.  I won't 
install it on anything critical now, not without taking the time to do a 
complete reeval of it, anyway.  It's been a year...maybe it's time.

> and then there is always nexenta if I
> really want a complete GNU userland and apt/dpkg.

How many different machines have you tried it on?  Perhaps you have been 
lucky, and have found that it installs on everything you want it to run 
on.

In my experience, both NCP and NexentaStor made me jump through quite a 
few hoops to find a hardware configuration they were happy with.  Even 
after I got them working, neither seemed valuable enough to bother 
sticking with them, compared to OSes I already know and trust to just run.

> Does it support direct sharing/exporting as nfs/cifs/iscsi

NFS, yes, that's how I'm using it.

CIFS, no, as there is no CIFS support in FreeBSD's kernel.  Of course, 
you can always just use Samba.

iSCSI, no, because there isn't yet any iSCSI serving support in FreeBSD 
of any kind.  Since I didn't want my ZFS pools to be directly attached 
to another machine, but rather shared among multiple machines in 
traditional file-server manner, this didn't cause a problem for me.

Let me bounce this ball back in your court: how about AFS, for the Macs 
in your organization?  ZFS has no direct support for it on either 
platform, but at least on FreeBSD and most Linuxes, it's a supported 
package, available on demand, already preconfigured for that system. 
All you have to do is do local customizations to the configuration, set 
it to start automatically, and you're done.  With OpenSolaris, it's a 
fully manual process.

> Does it support using ZFS for booting

Not as part of the OS installer, but it can be done:

http://lulf.geeknest.org/blog/freebsd/Setting_up_a_zfs-only_system/

This doesn't interest me because it shares the same limitation as on 
Solaris, which is that it will only work with a mirror.  I don't want to 
dedicate two disks just to the OS if I want a RAID-Z pool for actual data.

My solution for high root FS reliability was to put it on a CF card.  In 
addition to being solid state, it has a few side benefits:

- It lets me use an otherwise unused ATA connection.

- It's small enough that I can mount it in otherwise dead space in the 
chassis, instead of taking up a precious disk bay.

Once I got the system installed, I moved some top-level trees into 
dedicated ZFS pools, so my root filesystem is now quite small and rarely 
touched.

> lot more on vinum than there is on zfs in the FreeBSD manual.

I did most of my FreeBSD ZFS setup using the Solaris ZFS Admin Guide 
PDF.  Everything it asked me to do worked fine on FreeBSD.

Yes, I'm sure you can point to places where a thing will work on Solaris 
and not on FreeBSD, but I haven't found anything that actually *matters* 
to me yet.