[CentOS] 8-15 TB storage: any recommendations?

[John R Pierce]

JohnS wrote:
> Just asking is the fiber ports BiDirectional or Directional or can they
> support a Bond that is BiDirectional of 4GB/s  or can they be trunked
> into 16GB/s?  Bidirectional.   I need about 24 GB/s banwidth sustained,
> yes per second.  Also what type of sparse file I/O you get .  I see you
> stated multimode.  Some don't classify that as true BiDirectional
> Bonding.
>   

fiber is 4gbps (gigaBIT/sec) (or 1, 2, and now 8) and each FC link has 
two fibers, either of which can be used to transmit OR recieve at a 
given time (eg, each fiber is half duplex)..  MOST implementations use 
one link to read and the other to write.   A 4gbps fiber can typically 
sustain 400MByte/sec read or write, and potentially 400MByte/sec read 
*and* write.

To hit 24GBYTE/sec, yeouch.   the IO busses on most servers don't have 
that kind of bandwidth.    A PCI-Express 2.0 x16 (16 lane) card has 
8GB/sec peak burst rates.  The QPI bus on a Xeon 5500 server is around 
6GT/s peak for all IO including CPU->memory, if all transfers are 8 
bytes (64 bits), thats 48GB/sec.

BTW, in fiber, singlemode vs multimode refers to the optical modulation 
on the fiber and has nothing directly to do with the duplex or 
bonding.   single mode is more expensive, can transmit longer distances 
(dozens of kilometers), while multimode is cheaper but only suitable for 
relatively short distances (100s of meters).   Most all fiber channel 
devices use replaceable SFP transceivers, so you can use either type of 
transceiver with the appropriate fiber type.