----- Original Message -----

From: Terrence Martin

To: dan1

Sent: Friday, November 05, 2004 2:01 AM

Subject: Re: [Centos] Promise raid cards - software raid

dan1 wrote:

> Hello, Terrence.
> 
> Thank you for your complete answers. That's very interesting.
> 
>
> > I am not sure what you mean about the file system crashing?
> 
> I meant that it becomes unrecoverable, or that some datas are missing.
> 

These are two seperate problems with two seperate solutions.

You can have loss of data without the file system having any problems.
That is data is missing, perfectly working file system. In fact
journaled file systems pick preserving the file system over missing data
every time.

Eg.

You have a database and are writing a query to the database that inerts
1000 records. If you do not insert all 1000 records then you cannot use
any of them. That is record 1000 depends on all the previous 999.  On
record 678 the power fails. When the system comes back what happened to
the first 678 inserts? You assume they complete but since you need all
1000 for the data to be valid you basically have to delete the ones you
did insert and start again, if you can.

Deleting means that you first have to know what records to delete.

Depending on the complexity of the inserts (they could touch dozens of
tables and interrelationships) you could have a lot of work ahead of you
to manually find what is a record that is part of that incomplete set
and what records are not.

This is where a journal comes in. A journal records when data is
written. Basically you record what you write after you finish writing a
record so that worst case you can replay that journal of changes to back
out of what happened. This is what a journaled file system does. It
allows you to more easily back out of the incomplete data problem
quickly. This is opposed to the old method of file system consistency
checking which was like a manual search of the entire database. If you
only have to go over the changes, rather than searching the whole
database it is faster.  Also the long manual check is complicated and
prone to error. For speed and accuracy a journal is better.

However journaled file systems do not save the data. In fact journaled
file systems will throw away data if it is incomplete. Say in the above
example for some reason that you could use the first 678 records, or
that there was no other way to recover the 1000 records again so 678 was
better than nothing. Well a journal does not care. It simply looks to
see if the whole transactions completed of 1000 records. If it didn't it
deletes everything up to the failure. Even if 999 of the 1000 records
was written it will still delete the 999. The assumption being that it
is better to have a consistent file system and protect the good data
than have partially written data, that while valuable is inconsistent.

If you want to ensure that even partial data is preserved you have to do
other things to protect it.  A battery protect RAID card is one very
very narrow approach that solves one specific type of failure state.
Where data is written to the RAID card but not to disk yet.

Lets go back to our above example

You have a power failure on record 678. The raid card has memory to
store 5 records. At the time of the power failure it has only sent the
first 673 records to the disks for writting. The other 5 are in the
controller cache. If you have a battery on that memory you will save
those 5 records. However does it matter? Afterall 678 or 673 they are
still not 1000. Also the disks themselves may store in their cache 2
records. So the disk has only written records up to 670 with records 671
and 672 still waiting in volatile RAM with no battery backup attached
directly to the disk (write back cache on all PATA drives).

The power fails and the system comes back online. The RAID card writes
records 673-678 to the disks and they write them. Unfortunately records
671 and 672 are lost because they were in volatile disk cache on the
disk itself.

So you have records 0-670,637-678. You in fact have a hole in what you
have on disk, and who cares anyway because the journaled file system is
going to delete all those records when it goes to work to ensure
integrity over data preservation.

Basically RAID batteries buy you something, but not much, and they buy
you even less when they are attached to ATA drives that have write back
cache that essentially makes the RAID cache moot.

> > I do not recommend ext3 for anything over about 120GB.
> 
> OK, that's interesting.
> 


I work with compute cluster and with file systems that are in the
terabytes of size. Basically nothing else has come close to XFS in
practice.  The guys that admin the really big stuff that we collaborate
with will not touch anything but XFS and they have petabytes of storage.
If you can go with XFS. Even RHEL4 should finally have XFS standard
since fedora core 2 and later has it as an option, even for the root disk.

> > My biggest question is why at this point are you even bothering with
> PATA drives? Compared to SATA drives they are unreliable and poor
> performing for about the same cost.
> This is what I get from my ISP (I think). However it doesn't change a
> lot my conception and thoughts about raid. The flush problem remains
> the same.
> Also I am more familiar to PATA.

The flush problem as I hope I have demonstrated is not at all addressed
by battery backups on RAID card ram. Battery based backup of RAID memory
is a good gimmick, but in practice is useless. It covers such a narrow
part of the problem space as to be irrelevant.

If you are concerned that power failure will loose data get a UPS for
the entire system. It is the only thing that will help you because it is
the only thing that will allow your entire system, from software to
hardware to achieve a consistent state before shutting down. Otherwise
you may save a few bytes of data that was in cache on the RAID card but
that does not matter since you will still end up with an incomplete file
system transaction that the journaled file system is going to delete
anyway.

The linux journaled file systems are very good at preserving integrity,
even in the face of underlying hardware failure in some cases. Choosing
a good file system is all you need to do there to ensure that aspect. As
far as data loss, aside from backups after the fact the only solution
that will work in practice is Uninteruptible Power Supplies that will
give you enough time to shutdown the entire system in a consistent way.

Terrence

> 
> Thank you for your interesting advices. I appreciate that !
> 
> Best regards,
> 
> Daniel
> 
> 
>