Hi:
I need advice on implementing a storage server. I really do not have the $ to spend for a Dell iSCSI storage divice and I am thinking trunning CentOS 5.x with ftp or FreeNAS. Here is what I am looking at and concerned about.
Situation: My current storage needs are approximately 1.5 TB annually. This will increase to about 3.5 TB annually over the next 5 years (rough est.). This box will just be a data archive and once it is full it will only be used very infrequently if not used at all. Files are small up to 10 MB but numerous.
CentOS: Upgrading to the newer CentOS flavors. I will not have the ability to archive this data to tape and I am concerned about loosing the data when upgrading the OS. How best to handle this?
Storage limitation. It is my understanding that there is a 2 TB storage limitation with Linux (and windows) in general particularly for stability. I see that ReiserFS can go up to 16 TB. Is any one using this? If so, how has it been for you?
FreeNAS Anyone using FreeNAS? What is your experience? How easy is it to add new drives and keep your data? Upgrading to newer versions?
Thanks,
Ed
On Tue, 6 May 2008 at 12:11pm, Ed Morrison wrote
Situation: My current storage needs are approximately 1.5 TB annually. This will increase to about 3.5 TB annually over the next 5 years (rough est.). This box will just be a data archive and once it is full it will only be used very infrequently if not used at all. Files are small up to 10 MB but numerous.
CentOS: Upgrading to the newer CentOS flavors. I will not have the ability to archive this data to tape and I am concerned about loosing the data when upgrading the OS. How best to handle this?
You have to be careful, but it's quite easy to leave partitions (and thus their data) alone when you are updating/reinstalling the OS.
Storage limitation. It is my understanding that there is a 2 TB storage limitation with Linux (and windows) in general particularly for stability. I see that ReiserFS can go up to 16 TB. Is any one using this? If so, how has it been for you?
You cannot boot from a device larger than 2TiB, but that's the only limitation at that size. I run several multi-TB servers (including over 8TB) on CentOS-5 with no issues (using ext3).
You do not want to use ReiserFS. It's not supported under CentOS, and it's future is far less than certain (and I do not want to restart *that* OT conversation). ext3 is the default FS under CentOS and works pretty well.
On Tue, May 6, 2008 at 3:11 PM, Ed Morrison edward.morrison@gmail.com wrote:
Hi:
I need advice on implementing a storage server. I really do not have the $ to spend for a Dell iSCSI storage divice and I am thinking trunning CentOS 5.x with ftp or FreeNAS. Here is what I am looking at and concerned about.
Situation: My current storage needs are approximately 1.5 TB annually. This will increase to about 3.5 TB annually over the next 5 years (rough est.). This box will just be a data archive and once it is full it will only be used very infrequently if not used at all. Files are small up to 10 MB but numerous.
CentOS: Upgrading to the newer CentOS flavors. I will not have the ability to archive this data to tape and I am concerned about loosing the data when upgrading the OS. How best to handle this?
Storage limitation. It is my understanding that there is a 2 TB storage limitation with Linux (and windows) in general particularly for stability. I see that ReiserFS can go up to 16 TB. Is any one using this? If so, how has it been for you?
FreeNAS Anyone using FreeNAS? What is your experience? How easy is it to add new drives and keep your data? Upgrading to newer versions?
Thanks,
Ed
I haven't used this and maybe I understand the concept, but what about RedHat's GFS? From what has been told to me, you take a cluster of servers and it turns them into a large disk array. Someone correct me if I'm wrong.
Actually, I am on rh436 course now. Why not set up centos51 as iscsi target itself?
On 5/6/08, Matt Shields mattboston@gmail.com wrote:
On Tue, May 6, 2008 at 3:11 PM, Ed Morrison edward.morrison@gmail.com wrote:
Hi:
I need advice on implementing a storage server. I really do not have the
$
to spend for a Dell iSCSI storage divice and I am thinking trunning CentOS 5.x with ftp or FreeNAS. Here is what I am looking at and concerned
about.
Situation: My current storage needs are approximately 1.5 TB annually. This will increase to about 3.5 TB annually over the next 5 years (rough est.).
This
box will just be a data archive and once it is full it will only be used very infrequently if not used at all. Files are small up to 10 MB but numerous.
CentOS: Upgrading to the newer CentOS flavors. I will not have the ability to archive this data to tape and I am concerned about loosing the data when upgrading the OS. How best to handle this?
Storage limitation. It is my understanding that there is a 2 TB storage limitation with Linux (and windows) in general particularly for stability. I see that ReiserFS can go up to 16 TB. Is any one using this? If so,
how
has it been for you?
FreeNAS Anyone using FreeNAS? What is your experience? How easy is it to add
new
drives and keep your data? Upgrading to newer versions?
Thanks,
Ed
I haven't used this and maybe I understand the concept, but what about RedHat's GFS? From what has been told to me, you take a cluster of servers and it turns them into a large disk array. Someone correct me if I'm wrong.
-- -matt _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Scott Thistle wrote:
Actually, I am on rh436 course now. Why not set up centos51 as iscsi target itself?
that just pushes the file management issues off to another system, where you still have to solve them
he wants an archive server, which is more of a NAS device then a SAN. iSCSI addresses the SAN space, providing flexible block storage to multiple systems
Ed Morrison wrote:
Hi:
I need advice on implementing a storage server. I really do not have the $ to spend for a Dell iSCSI storage divice and I am thinking trunning CentOS 5.x with ftp or FreeNAS. Here is what I am looking at and concerned about.
Situation: My current storage needs are approximately 1.5 TB annually. This will increase to about 3.5 TB annually over the next 5 years (rough est.). This box will just be a data archive and once it is full it will only be used very infrequently if not used at all. Files are small up to 10 MB but numerous.
infrastructure to support lots of SATA drives isn't real cheap regardless. you really don't want to just bolt a bunch of drives up inside a jumbo desktop tower and call it a server. 5 years at that run rate is going to be something like 12TB total storage, which using commodity 500GB SATA drives in raid10 will take around 48 drives. Thats a lot of SATA channels...
With that many spindles, you'll also want to allocate several hot spares.
I dislike raid5 for a number of reasons, and would recommend sticking with mirroring, eg raid1 or raid10. You /never/ want to build a raid5 much over about 6-8 disks, or the raid rebuild times get ridiculous and double drive failures will lose huge amounts of storage.
hey, have you considered the Sun x4500 ? its a 4U(?) dual dualcore opteron server that comes with 48 x 500GB SATA drives. ***
CentOS: Upgrading to the newer CentOS flavors. I will not have the ability to archive this data to tape and I am concerned about loosing the data when upgrading the OS. How best to handle this?
as others have said, as long as your critical data is on seperate file systems, there should be no issue here.
Storage limitation. It is my understanding that there is a 2 TB storage limitation with Linux (and windows) in general particularly for stability. I see that ReiserFS can go up to 16 TB. Is any one using this? If so, how has it been for you?
since your data is archival in nature, it really shouldn't be that hard to manage it as multiple 2 TB chunks on seperate file systems. when you fill 2TB, take 8 x 500GB more SATA drives, raid10 them, and mount them as another file system, /u01, /u02, .... keep an index file somewhere which logs which backups are where.
FreeNAS Anyone using FreeNAS? What is your experience? How easy is it to add new drives and keep your data? Upgrading to newer versions?
I setup OpenFiler once, that worked quite nicely, supported NFS, SMB, and iSCSI, and was pretty easy to use. I'd have to assume FreeNAS is similar.
*** heresy (for this list), Solaris 10, with its ZFS file system, is extremely good at handling very large storage configurations like this.
John R Pierce wrote:
infrastructure to support lots of SATA drives isn't real cheap regardless. you really don't want to just bolt a bunch of drives up inside a jumbo desktop tower and call it a server. 5 years at that run rate is going to be something like 12TB total storage, which using commodity 500GB SATA drives in raid10 will take around 48 drives. Thats a lot of SATA channels...
1TB drives are available now. 5 years from now, who knows?
since your data is archival in nature, it really shouldn't be that hard to manage it as multiple 2 TB chunks on seperate file systems. when you fill 2TB, take 8 x 500GB more SATA drives, raid10 them, and mount them as another file system, /u01, /u02, .... keep an index file somewhere which logs which backups are where.
If it's really rarely used and you have a sensible scheme to find it you could just have a drawer full of inexpensive external 1TB drives that you can plug in on demand, using USB, firewire, or sata connections.
I just purchased an equallogic SAN with 16 1TB drives for 52k at work. Love it, scheduled snapshots, thin provisioning, iscsi only but fairly swift at 16 spindles in a RAID 50.
Jason www.cyborgworkshop.org
John R Pierce wrote:
Ed Morrison wrote:
Hi:
I need advice on implementing a storage server. I really do not have the $ to spend for a Dell iSCSI storage divice and I am thinking trunning CentOS 5.x with ftp or FreeNAS. Here is what I am looking at and concerned about.
Situation: My current storage needs are approximately 1.5 TB annually. This will increase to about 3.5 TB annually over the next 5 years (rough est.). This box will just be a data archive and once it is full it will only be used very infrequently if not used at all. Files are small up to 10 MB but numerous.
infrastructure to support lots of SATA drives isn't real cheap regardless. you really don't want to just bolt a bunch of drives up inside a jumbo desktop tower and call it a server. 5 years at that run rate is going to be something like 12TB total storage, which using commodity 500GB SATA drives in raid10 will take around 48 drives. Thats a lot of SATA channels...
With that many spindles, you'll also want to allocate several hot spares.
I dislike raid5 for a number of reasons, and would recommend sticking with mirroring, eg raid1 or raid10. You /never/ want to build a raid5 much over about 6-8 disks, or the raid rebuild times get ridiculous and double drive failures will lose huge amounts of storage.
hey, have you considered the Sun x4500 ? its a 4U(?) dual dualcore opteron server that comes with 48 x 500GB SATA drives. ***
CentOS: Upgrading to the newer CentOS flavors. I will not have the ability to archive this data to tape and I am concerned about loosing the data when upgrading the OS. How best to handle this?
as others have said, as long as your critical data is on seperate file systems, there should be no issue here.
Storage limitation. It is my understanding that there is a 2 TB storage limitation with Linux (and windows) in general particularly for stability. I see that ReiserFS can go up to 16 TB. Is any one using this? If so, how has it been for you?
since your data is archival in nature, it really shouldn't be that hard to manage it as multiple 2 TB chunks on seperate file systems. when you fill 2TB, take 8 x 500GB more SATA drives, raid10 them, and mount them as another file system, /u01, /u02, .... keep an index file somewhere which logs which backups are where.
FreeNAS Anyone using FreeNAS? What is your experience? How easy is it to add new drives and keep your data? Upgrading to newer versions?
I setup OpenFiler once, that worked quite nicely, supported NFS, SMB, and iSCSI, and was pretty easy to use. I'd have to assume FreeNAS is similar.
*** heresy (for this list), Solaris 10, with its ZFS file system, is extremely good at handling very large storage configurations like this. _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Ed Morrison wrote:
I need advice on implementing a storage server. I really do not have the $ to spend for a Dell iSCSI storage divice and I am thinking trunning CentOS 5.x with ftp or FreeNAS. Here is what I am looking at and concerned about.
Situation: My current storage needs are approximately 1.5 TB annually. This will increase to about 3.5 TB annually over the next 5 years (rough est.). This box will just be a data archive and once it is full it will only be used very infrequently if not used at all. Files are small up to 10 MB but numerous.
First consider whether you can organize this into 1 TB or smaller partitions that are mounted separately. If you can do that, growing the space is trivial - and you get the advantage that you can do raid1 mirrors of individual drives which gives you the ability to recover data from any single disk.
CentOS: Upgrading to the newer CentOS flavors. I will not have the ability to archive this data to tape and I am concerned about loosing the data when upgrading the OS. How best to handle this?
Don't put the permanent storage on your system drive(s) at all. Add one or more directories after installation and mount the additional partitions or raid arrays there. Then if you do a new install, just uncheck the devices as disks that can be used for the system and add the mount points back when it is done. But, there are lots of other ways to lose data. If you'd need it after a building fire/flood or operator error you should build two of these and rsync to somewhere offsite.
Storage limitation. It is my understanding that there is a 2 TB storage limitation with Linux (and windows) in general particularly for stability.
I think that's 8TB if you don't boot from it. But that's per mounted filesystem - if you can have smaller separate partitions, it won't matter.
FreeNAS Anyone using FreeNAS? What is your experience? How easy is it to add new drives and keep your data? Upgrading to newer versions?
You might look at openfiler if you want an appliance.
Ed Morrison wrote:
Hi:
I need advice on implementing a storage server. I really do not have the $ to spend for a Dell iSCSI storage divice and I am thinking trunning CentOS 5.x with ftp or FreeNAS. Here is what I am looking at and concerned about.
Situation: My current storage needs are approximately 1.5 TB annually. This will increase to about 3.5 TB annually over the next 5 years (rough est.). This box will just be a data archive and once it is full it will only be used very infrequently if not used at all. Files are small up to 10 MB but numerous.
Well that's a hell of a lot of storage for a cheap project. Instead of a Dell MD3000 appliance try a Dell 860 1u server (Quad Xeon) with the LSI PERC 5e 256/512MB RAID controller there you can chain up to 3 MD1000 JBOD SATA enclosures to it. It can handle mixed SAS/SATA drives and can hold 45 spindles across 3 enclosures per 1u server.
The 1u device will be a SPOF but you wanted cheap...
CentOS: Upgrading to the newer CentOS flavors. I will not have the ability to archive this data to tape and I am concerned about loosing the data when upgrading the OS. How best to handle this?
With the 1u server you can always upgrade the OS as the data is stored externally. Hell you can even swap out the 1u 860 for say a 2u 2950 as needs grow which gives better redundancy as well as internal storage for snapshots or some other use. Just get the 860 with 2x250GB drives, and create a software mirror out of them. You can always break the mirror, upgrade the OS and if it works re-mirror, otherwise boot the old half and re-mirror.
Storage limitation. It is my understanding that there is a 2 TB storage limitation with Linux (and windows) in general particularly for stability. I see that ReiserFS can go up to 16 TB. Is any one using this? If so, how has it been for you?
ext3 can go up to 8TB, xfs and jfs can go up to 1EB which should hold you.
FreeNAS Anyone using FreeNAS? What is your experience? How easy is it to add new drives and keep your data? Upgrading to newer versions?
You can also check out OpenFiler which has NAS and iSCSI included.
-Ross
______________________________________________________________________ This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender and permanently delete the original and any copy or printout thereof.
On Tue, May 6, 2008 at 3:11 PM, Ed Morrison edward.morrison@gmail.com wrote:
Situation: My current storage needs are approximately 1.5 TB annually. This will increase to about 3.5 TB annually over the next 5 years (rough est.). This box will just be a data archive and once it is full it will only be used very infrequently if not used at all. Files are small up to 10 MB but numerous.
The solution I found best was to buy a 2U server that has 8*750GB disks, though they'd probably be 1TB today. Put the disks into a RAID 5 or 6. Using hardware RAID, divvy them up into one 50GB drive, and one really large drive. Put the OS on the 50GB drive, mount the really big drive.
Now you have a 50GB drive and a 7*750-50 drive. When you fill that up, just buy another 2U server. When you do fill it up, the next one will be cheaper and or bigger.
The keys to this type of setup are: 1) Don't buy storage you'll need next year today. The best time to buy this kind of hardware is right before you need it. 2) Look at the overall cost per gigabyte. That's the metric that drives things. 3) Understand your tolerance for downtime and data protection. If you have another copy, or a backup, and its not mission critical data, its much cheaper not to waste disks on redundancy.
We have tape backups of our systems, and factoring in the cost of tape and other costs, its still possible to get storage with a marginal cost below $1 / GB. That includes a 3 year warranty, quad core processor, 4GB of RAM which you can probably put to use elsewhere.
I just posted this on my website, oddly enough. While you need to really understand your storage requirements to make an informed choice between hardware or software RAID, with quad core CPUs being as cheap as they are it's hard to not make the argument for software. This is just hdparm over an average of 5 runs each on very similar machines.
5 disc SAS array with 136g 10k drives and a hardware controller
Timing cached reads: 13336 MB in 2.00 seconds = 6673.96 MB/sec Timing buffered disk reads: 98 MB in 1.18 seconds = 83.31 MB/sec
4 disc RAID 5 with 3Ware 9650SE and 500g 7200RPM drives
Timing cached reads: 6576 MB in 2.00 seconds = 3293.08 MB/sec Timing buffered disk reads: 448 MB in 3.00 seconds = 149.20 MB/sec
Single 500g 7200 RPM SATA drive
Timing cached reads: 14220 MB in 2.00 seconds = 7119.78 MB/sec Timing buffered disk reads: 198 MB in 3.02 seconds = 65.51 MB/sec
6 500g 7200 RPM SATA drives in a software RAID 5 array
Timing cached reads: 14364 MB in 2.00 seconds = 7191.86 MB/sec Timing buffered disk reads: 852 MB in 3.00 seconds = 283.64 MB/sec
Jason www.cyborgworkshop.org
Michael Semcheski wrote:
On Tue, May 6, 2008 at 3:11 PM, Ed Morrison <edward.morrison@gmail.com mailto:edward.morrison@gmail.com> wrote:
Situation: My current storage needs are approximately 1.5 TB annually. This will increase to about 3.5 TB annually over the next 5 years (rough est.). This box will just be a data archive and once it is full it will only be used very infrequently if not used at all. Files are small up to 10 MB but numerous.The solution I found best was to buy a 2U server that has 8*750GB disks, though they'd probably be 1TB today. Put the disks into a RAID 5 or 6. Using hardware RAID, divvy them up into one 50GB drive, and one really large drive. Put the OS on the 50GB drive, mount the really big drive.
Now you have a 50GB drive and a 7*750-50 drive. When you fill that up, just buy another 2U server. When you do fill it up, the next one will be cheaper and or bigger.
The keys to this type of setup are:
- Don't buy storage you'll need next year today. The best time to buy
this kind of hardware is right before you need it. 2) Look at the overall cost per gigabyte. That's the metric that drives things. 3) Understand your tolerance for downtime and data protection. If you have another copy, or a backup, and its not mission critical data, its much cheaper not to waste disks on redundancy.
We have tape backups of our systems, and factoring in the cost of tape and other costs, its still possible to get storage with a marginal cost below $1 / GB. That includes a 3 year warranty, quad core processor, 4GB of RAM which you can probably put to use elsewhere.
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
-
Ed Morrison -
Jason -
John R Pierce -
Joshua Baker-LePain -
Les Mikesell -
Matt Shields -
Michael Semcheski -
Ross S. W. Walker -
Scott Thistle