[CentOS] Building a Back Blaze style POD

Sun May 8 19:12:11 UTC 2011

On Sun, May 8, 2011 at 9:06 PM, Jason <slackmoehrle.lists at gmail.com> wrote:

> Rudy,
>
>
> Do you have a recommendation of a motherboard?
>
>
Well, choose one here:
http://www.supermicro.com/products/motherboard/matrix/

I don't have specific recommendations but we've had great success with all
our SuperMicro servers, both with single & dual CPU configurations, ranging
from 4GB - 128GB RAM

> I am still reading the rest of your post. Thanks!
>
> -Jason
>
> --
> Jason
>
> On Sunday, May 8, 2011 at 11:26 AM, Rudi Ahlers wrote:
> >
> >
> > On Sun, May 8, 2011 at 8:03 PM, Jason <slackmoehrle.lists at gmail.com>
> wrote:
> > >  Hi All,
> > >
> > >  I am about to embark on a project that deals with allowing information
> archival, over time and seeing change over time as well. I can explain it a
> lot better, but I would certainly talk your ear off. I really don't have a
> lot of money to throw at the initial concept, but I have some. This device
> will host all of the operations for the first few months until I can afford
> to build a duplicate device. I already had a few parts of the idea done and
> ready to get live.
> > >
> > >  I am contemplating building a BackBlaze Style POD. The goal of the
> device is to start acting as a place to have the crawls store information,
> massage it, get it into db's and then notify the user the task is done so
> they can start looking at the results.
> > >
> > >  For reference here are a few links:
> > >
> > >
> http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/
> > >
> > >  and
> > >
> > >
> http://cleanenergy.harvard.edu/index.php?ira=Jabba&tipoContenido=sidebar&sidebar=science
> > >
> > >  There is room for 45 drives in the case (technically a few more).
> > >
> > >  45 x 1tb 7200rpm drives is really cheap, about $60 each.
> > >
> > >  45 x 1.5tb 7200rpm drives are about $70 each.
> > >
> > >  45 x 2tb 7200rpm drives are about $120 each
> > >
> > >  45 x 3tb 7200rpm drives are about $180-$230 each (or more, some are
> almost $400)
> > >
> > >  I have question before I commit to building one and I was hoping to
> get advice.
> > >
> > >  1. Can anyone recommend a mobo/processor setup that can hold lots of
> RAM? Like 24gb or 64gb or more?
> >
> > Any brand server motherboard will do. I prefer supermicro, but you can
> use Dell, HP, Intell, etc, etc.
> >
> > >
> > >  2. Hardware RAID or Software RAID for this?
> >
> > Hardware RAID will be expensive on 45 drives. IF you can, split the 45
> drives into a few smaller RAID arrays. To rebuild 1x large 45TB RAID array,
> with either hardware or software would probably take a week, or more,
> depending on which RAID type you use - i.e. RAID 5, or 6, or 10. I prefer
> RAID 10 since it's best for speed and the rebuilds are the quickest. But you
> loose half the space, i.e. 45TB drives will give you about 22TB space. 45x
> 2TB HDD's would give you about 44TB space though.
> >
> > >
> > >  3. Would CentOS be a good choice? I have never used CentOS on a device
> so massive. Just ordinary servers, so to speak. I assume that it could
> handle so many drives, a large, expanding file system.
> >
> > Yes it would be fine.
> >
> >
> > >  4. Someone recommended ZFS but I dont recall that being available on
> CentOS, but it is on FreeBSD which I have little experience with.
> >
> > I would also prefer to use ZFS for this type of setup. use one 128GB SL
> type SSD drive as a cache drive to speed up things and 2x log drives to help
> with drive recovery. With ZFS you would be able to use one large RAID array
> if you have the log drives since it was recover from driver failure much
> better than other file systems. Although you can install ZFS as user-land
> tools, which will be slower than running it via the kernel. But, it would be
> better to use Solaris or FreeBSD for this - look @ Nexenta / FreeNAS /
> OpenIndia for this.
> >
> > >
> > >  5. How would someone realistically back something like this up?
> >
> > To another one as large :)
> >
> > OR, more realistically, if you already have some backup servers, and the
> full 45TB isn't full of data yet, then simply backup what you have. By the
> sounds of it your project is still new so your data won't be that much. I
> would simply build a gluster / CLVM cluster of smaller cheaper servers -
> which basically allows you to add say 4TB / 8TB (depending on what chassis
> you use and how many drives it can take) at a time to the backup cluster,
> which will be cheaper than buying another one identical to this right now.
> >
> > >
> > >  Ultimately I know over time I need to distribute my architecture out
> and have a number of web-servers, balancing, etc but to get started I think
> this device with good backups might fit the bill.
> >
> > If this device will be used for web + mail + SQL, then you may probably
> look at using 4 quad core CPU's + 128GB RAM. With this many drives (or
> rather, this much data) you'll probably run out of RAM / CPU / Network
> resources before you run out of HDD space.
> >
> >
> >
> > With a device this big (in terms of storage) I would rather have 2
> separate "processing" servers which just mounts LUN's from this POD
> (exported as NFS / iSCSI / FCoE / etc) and then have a few faster SAS / SSD
> drives for SQL / log processing.
> >
> > >
> > >  I can be way more detailed if it helps, I just didn't want to clutter
> with information that might not be relevant.
> > > --
> > >  Jason
> > >
> > >  _______________________________________________
> > >  CentOS mailing list
> > > CentOS at centos.org
> > > http://lists.centos.org/mailman/listinfo/centos
> >
> >
> > --
> > Kind Regards
> > Rudi Ahlers
> > SoftDux
> >
> > Website: http://www.SoftDux.com
> > Technical Blog: http://Blog.SoftDux.com
> >  Office: 087 805 9573
> > Cell: 082 554 7532
> > _______________________________________________
> > CentOS mailing list
> > CentOS at centos.org
> > http://lists.centos.org/mailman/listinfo/centos
> >
>
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos
>

-- 
Kind Regards
Rudi Ahlers
SoftDux

Website: http://www.SoftDux.com
Technical Blog: http://Blog.SoftDux.com
Office: 087 805 9573
Cell: 082 554 7532
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.centos.org/pipermail/centos/attachments/20110508/ba0faf40/attachment.html>