David C. Miller <millerdc at ...> writes: > > > ----- Original Message ----- > > From: "John Doe" <jdmls at ...> > > To: "Cent O Smailinglist" <centos at ...> > > Sent: Tuesday, August 28, 2012 3:14:29 AM > > Subject: [CentOS] Is glusterfs ready? > > > > Hey, > > > > since RH took control of glusterfs, I've been looking to convert our > > old independent RAID storage servers to several non RAID glustered > > ones. > > > > The thing is that I, here and there, heard a few frightening stories > > from some users (even with latest release). > > Any one has experienced with it long enough to think one can blindly > > trust it or if it is almost there but not yet ready? > > Heya, Well I guess I'm one of the frightening stories, or at least a previous employer was. They had a mere 0.1 petabyte store over 6 bricks yet they had incredible performance and reliability difficulties. I'm talking about a mission critical system being unavailable for weeks at a time. At least it wasn't customer facing (there was another set of servers for that). The system was down more than it was up. Reading was generally OK (but very slow) but multiple threads writing caused mayhem - I'm talking lost files and file system accesses going into the multiple minutes. In the end I implemented a 1-Tb store to be fuse-unioned over the top of the thing to take the impact of multiple threads writing to it. A single thread (overnight) brought the underlying glusterfs up to date. That got us more or less running but the darned thing spent most of its time re-indexing and balancing rather than serving files. To be fair, some of the problems were undoubtedly of their own making as 2 nodes were centos and 4 were fedora-12 - apparently the engineer couldn't find the installation CD for the 2 new nodes and 'made do' with what he had! I recall that a difference in the system 'sort' command gave all sorts of grief until it was discovered, never mind different versions of the gluster drivers. I'd endorse Johnny's comments about it not handling large numbers of small files well (ie <~ 10 Mb). I believe it was designed for large multi-media files such as clinical X-Rays. ie a small number of large files. Another factor is that the available space is the physical space divided by 4 due to the replication across the nodes on top of the nodes being RAID'd themselves. Lesse now - that was all of 6 months ago - unlike most of my war stories, it's not ancient history!!