Let me comment some questions in one single mail:
My basic requirement with what I'm doing is to use standard tools and formats so that archives I write today can be readable in 10 years.
Star becomes 30 in 4 months, any archive created since it's early beginning in summer 1982 can still be read back.
I don't think there is any such general consensus. Are you reading something that favors Solaris/*bsd over GNU based systems?
Schily tools (and in special star) implement support for Linux specific extensions. This is what you do not get from gtar at all. So why do Linux distros prefer gtar even though there is no Linux support?
I doubt if they are as well maintained in linux distros as the GNU tool set, particularly in terms of having recent fixes backported into the versions carried in enterprise distros.
gtar still did not fix bugs I reported in 1993 (e.g. the bug that causes gtar to complain with "skipping to next header" even on it's own archives). I am thus sure that star not not worse than gtar....
It shouldn't matter if you don't use either of the version's extensions, and for archiving you probably don't need them. For example, star and GNUtar use very different concepts for incremental backups - star is sort of like dump and must work on filesystem boundaries where GNUtar's --listed incremental needs a file to hold state but will work on arbitrary directories and can span mount points. But for archiving, you probably only care about the maximum size of a file it can handle.
When I implemented incremental restores for star in September 2004, I wrote a simple script for a incremental testcase and tested the deltas with ufsdump/ufsrestore, gtar and the star version at that time. Gtar was unable to deal with my testcase, so I stopped testing it any further.
If you like to discuss incrementals, you definitely need to discuss behavior at restore time and restoring incrementals definitely does not work correclty with gtar if you renamed directories.
Star is used to do incremental backups/restores on a dayly base in Berlios since September 2004. Since Spring 2005, not a single problem was seen, so there are more that 2500 successful incremental restores that verify no problem even under stange conditions.
I don't think so - I'm fairly sure I've seen GNUtar complain about bad headers, say 'skipping to next header' and then find something. It won't do that if you used the -z option because you generally can't recover from errors in compression. But, I've never seen a tape drive recover from an error and continue past it anyway so in practice that's not going to matter. If you are concerned about errors, keep more copies.
This problem is not caused by compression or not, it is a general gtar bug that I reported in 1993 already. Nobody knows why it hits and the structure of the gtar sources makes it really hard to debug this problem. The FSF was interested to throw aywa gtar and replace it by star 10 years ago for this kind of problems in gtar.
afio is an archiver (available from third-party repos, not base) which can compress yet still recover--it basically compresses each file individually instead of compressing the entire archive, so the file might be unrecoverable but the rest of the archive is still intact. I use it for my tape backups (though your point of not knowing if it'll fit on the tape is valid).
Be careful with what you believe. The CPIO archive format in general is worse with resyncing to a defective archive that TAR is. Also note that afio greates arhives that may start to be non CPIO compliant somewhere in the middle. So you can never know whether you are able to restore with anything other than afio. What if afio does not compile on your new platform because it is no longer maintained?
Also note that the POSIX standard dropped CPIO as an actively supported archiveformat because (different from TAR) any extention to CPIO results in creating a new incompatible archive format.
The following other problems are known with gtar:
- gtar is with aprox. 5% probability unable to read it's own continuation archives from multi volume archives. This cannot be fixed as it is caused by the concept used for multi volume archives in gtar.
- gtar created archives with defective sparse file until a few years ago (up to ~ 2005) in case a file was bigger than a few GB.
- gtar has much less features than star
- gtar does not inlcude libfind, so there is no support for the find(1) command line syntax in gtar.
- gtar needs aprox. 3x more CPU time as star
Jörg
On Tue, Feb 7, 2012 at 9:26 AM, Joerg Schilling Joerg.Schilling@fokus.fraunhofer.de wrote:
When I implemented incremental restores for star in September 2004, I wrote a simple script for a incremental testcase and tested the deltas with ufsdump/ufsrestore, gtar and the star version at that time. Gtar was unable to deal with my testcase, so I stopped testing it any further.
My testcase for star was moving a subdirectory of what my backup runs covered onto a mounted volume. Star failed and I stopped testing it any further.
If you like to discuss incrementals, you definitely need to discuss behavior at restore time and restoring incrementals definitely does not work correclty with gtar if you renamed directories.
If that is the issue I recall, you can recover without losing data.
Star is used to do incremental backups/restores on a dayly base in Berlios since September 2004. Since Spring 2005, not a single problem was seen, so there are more that 2500 successful incremental restores that verify no problem even under stange conditions.
So it all that time you have not mounted a new volume somewhere? No remote backups of hosts where someone else might add space? No one ever wanted to restore onto a different mount topology than the one where the backups were taken? Being able to do those things is the reason I use tar instead of filesystem-dependent dump.
This problem is not caused by compression or not, it is a general gtar bug that I reported in 1993 already. Nobody knows why it hits and the structure of the gtar sources makes it really hard to debug this problem. The FSF was interested to throw aywa gtar and replace it by star 10 years ago for this kind of problems in gtar.
So, you have a repeatable test case for this and no one has looked into it? That's surprising considering the number of people who have contributed to gnutar. And what drove the decision not to adopt star?
The following other problems are known with gtar:
- gtar is with aprox. 5% probability unable to read it's own
continuation archives from multi volume archives. This cannot be fixed as it is caused by the concept used for multi volume archives in gtar.
I assume you mean the version where you let the tape drive hit the end, not where you tell it the length. Does star always work in that scenario?
- gtar has much less features than star
Unless you would like it to do incrementals properly across mount points... And I thought there was another reason regarding features that amanda used gtar as well - maybe it was the ability to quickly estimate sizes of incrementals. If it had worked for amanda, I would probably have been using it for ages. When I looked at it, it couldn't because of missing features. These days I mostly use backuppc with rsync as the transport since online access is so much nicer than tapes and rsync obviously excels at detecting differences in incrementals, but I suppose there is still a place for archives.
Les Mikesell lesmikesell@gmail.com wrote:
On Tue, Feb 7, 2012 at 9:26 AM, Joerg Schilling Joerg.Schilling@fokus.fraunhofer.de wrote:
When I implemented incremental restores for star in September 2004, I wrote a simple script for a incremental testcase and tested the deltas with ufsdump/ufsrestore, gtar and the star version at that time. Gtar was unable to deal with my testcase, so I stopped testing it any further.
My testcase for star was moving a subdirectory of what my backup runs covered onto a mounted volume. Star failed and I stopped testing it any further.
So you tried to do something that cannot work for a filesystem oriented program. This is not a problem from star but you just did not use star the right way.
Note that gtar uses a big database that is needed at the dump side and that still does not hold enough data to let gtar work correctly.
Star on the other side just needs to remember dump dates and creates a full data base et extract side when doing incremental restores. This is also more reliable than what gtar does as all needed information for the restore is in the archives.
If you like to discuss incrementals, you definitely need to discuss behavior at restore time and restoring incrementals definitely does not work correclty with gtar if you renamed directories.
If that is the issue I recall, you can recover without losing data.
You are correct, you could in theory recover the data but this must be done manually.
Star is used to do incremental backups/restores on a dayly base in Berlios since September 2004. Since Spring 2005, not a single problem was seen, so there are more that 2500 successful incremental restores that verify no problem even under stange conditions.
So it all that time you have not mounted a new volume somewhere? No remote backups of hosts where someone else might add space? No one ever wanted to restore onto a different mount topology than the one where the backups were taken? Being able to do those things is the reason I use tar instead of filesystem-dependent dump.
Star of course can do what you like. You just need to create more than one backup once you split filesystems. It would be easy to handle if you like to use it....
This problem is not caused by compression or not, it is a general gtar bug that I reported in 1993 already. Nobody knows why it hits and the structure of the gtar sources makes it really hard to debug this problem. The FSF was interested to throw aywa gtar and replace it by star 10 years ago for this kind of problems in gtar.
So, you have a repeatable test case for this and no one has looked into it? That's surprising considering the number of people who have contributed to gnutar. And what drove the decision not to adopt star?
I had a repeatable case in 1993, but at that time, I send them an archive created by star. In any case, there is more than one gtar user who would be able and willing to provide gtar archives that trigger that case.
The reason for not adopting star was that RMS did send me a contract that was illegal in Europe and RMS was unwilling to convert his contract into something that I could legally sign.
BTW: there have been two attempts to replace gtar by star and both ended the same way.
The following other problems are known with gtar:
- gtar is with aprox. 5% probability unable to read it's own
continuation archives from multi volume archives. This cannot be fixed as it is caused by the concept used for multi volume archives in gtar.
I assume you mean the version where you let the tape drive hit the end, not where you tell it the length. Does star always work in that scenario?
As far as I can tell, yes. Star uses a completely different method to verify followup volumes and holds diffent sets of data that cannot cause this kind of failure. Gtar fails when it splits an archive at a location that is inside the (probably extended) tar header.
BTW: As star intentionally does not implement the "verification method" from gtar, star is able to restore such multi volume archives.
- gtar has much less features than star
Unless you would like it to do incrementals properly across mount points... And I thought there was another reason regarding features that amanda used gtar as well - maybe it was the ability to quickly estimate sizes of incrementals. If it had worked for amanda, I would probably have been using it for ages. When I looked at it, it couldn't because of missing features. These days I mostly use backuppc with rsync as the transport since online access is so much nicer than tapes and rsync obviously excels at detecting differences in incrementals, but I suppose there is still a place for archives.
AFAIK, amanda has too few features or there are no people who are willing to put efforts in adopting to star.
I currently cannot believe that there is really any important feature that is missing in star.
Jörg
On Tue, Feb 7, 2012 at 12:03 PM, Joerg Schilling Joerg.Schilling@fokus.fraunhofer.de wrote:
When I implemented incremental restores for star in September 2004, I wrote a simple script for a incremental testcase and tested the deltas with ufsdump/ufsrestore, gtar and the star version at that time. Gtar was unable to deal with my testcase, so I stopped testing it any further.
My testcase for star was moving a subdirectory of what my backup runs covered onto a mounted volume. Star failed and I stopped testing it any further.
So you tried to do something that cannot work for a filesystem oriented program.
It does work with GNUtar.
This is not a problem from star but you just did not use star the right way.
No, star does not do what I need, so I don't use it at all.
Note that gtar uses a big database that is needed at the dump side and that still does not hold enough data to let gtar work correctly.
How is it not correct?
Star on the other side just needs to remember dump dates and creates a full data base et extract side when doing incremental restores.
So what happens when you don't exactly match the source mount tree configuration when you try to restore? You complain about GNUtar not working in some special case - how is that worse than star not working in the very common case of moving some mount points around? I do that all the time. I don't think I've ever done the weird sequence that causes trouble with a gnutar incremental restore.
This is also more reliable than what gtar does as all needed information for the restore is in the archives.
Except when it isn't, because it is tied to the filesystem, not the directory tree structure. If I wanted filesystem dependencies I'd use dump. I expect tar to follow directory trees and be agnostic to mount points.
If you like to discuss incrementals, you definitely need to discuss behavior at restore time and restoring incrementals definitely does not work correclty with gtar if you renamed directories.
If that is the issue I recall, you can recover without losing data.
You are correct, you could in theory recover the data but this must be done manually.
How do you recover from the mount point change case for star? If it happens on the source side, do you lose data?
So it all that time you have not mounted a new volume somewhere? No remote backups of hosts where someone else might add space? No one ever wanted to restore onto a different mount topology than the one where the backups were taken? Being able to do those things is the reason I use tar instead of filesystem-dependent dump.
Star of course can do what you like. You just need to create more than one backup once you split filesystems. It would be easy to handle if you like to use it....
I handled it by pointing amanda at remote systems. And I expected it to keep those systems backed up even if someone else mounted some new disks in places where they needed some more space. I don't see how star fits into that scheme.
The reason for not adopting star was that RMS did send me a contract that was illegal in Europe and RMS was unwilling to convert his contract into something that I could legally sign.
Well, no one has ever accused RMS of being reasonable...
And I thought there was another reason regarding features that amanda used gtar as well - maybe it was the ability to quickly estimate sizes of incrementals. If it had worked for amanda, I would probably have been using it for ages. When I looked at it, it couldn't because of missing features. These days I mostly use backuppc with rsync as the transport since online access is so much nicer than tapes and rsync obviously excels at detecting differences in incrementals, but I suppose there is still a place for archives.
AFAIK, amanda has too few features or there are no people who are willing to put efforts in adopting to star.
Star simply does not do what amanda needs - or did not the last time I looked. Amanda needs a way to quickly estimate the size of a run for its brilliant feature of automatically balancing the mix of fulls and incrementals to ensure that you get at least an incremental of every target every night plus a full within the number of tapes in your rotation. And it can't fail on incrementals just because someone replaced a directory on a remote machine with a mount point.
I currently cannot believe that there is really any important feature that is missing in star.
Those features obviously aren't important to you, but they are enough to keep me - or any amanda user - from considering star. And amanda made things from several machines fit on my one non-changer tape drive every night for more than a decade with nothing on my part except swapping the tape sometime during the day (and handled things gracefully if I forgot). I don't think there was any alternative that could have worked as well.
Les Mikesell lesmikesell@gmail.com wrote:
My testcase for star was moving a subdirectory of what my backup runs covered onto a mounted volume. Star failed and I stopped testing it any further.
So you tried to do something that cannot work for a filesystem oriented program.
It does work with GNUtar.
In general, this does not work with gtar. You are exactly in the area that caused my conclusion that gtar is not useful at all for incremental backups.
This is not a problem from star but you just did not use star the right way.
No, star does not do what I need, so I don't use it at all.
Star does what you need, you are just using it the wrong way.
Note that gtar uses a big database that is needed at the dump side and that still does not hold enough data to let gtar work correctly.
How is it not correct?
As mentioned, gtar has major problems whe directory is envolved. How many successful restores did you pass?
Star on the other side just needs to remember dump dates and creates a full data base et extract side when doing incremental restores.
So what happens when you don't exactly match the source mount tree configuration when you try to restore? You complain about GNUtar not working in some special case - how is that worse than star not working in the very common case of moving some mount points around? I do that all the time. I don't think I've ever done the weird sequence that causes trouble with a gnutar incremental restore.
The difference between gtar and star is that gtar never works correctly for all possible changes, while star works correct and you just need to follow it's constraints. As gtar does not work on a simple testcase, I am not willing to trust in gtar's incrementals.
This is also more reliable than what gtar does as all needed information for the restore is in the archives.
Except when it isn't, because it is tied to the filesystem, not the directory tree structure. If I wanted filesystem dependencies I'd use dump. I expect tar to follow directory trees and be agnostic to mount points.
It may be that you miss the fact that modern filesystems like e.g. ZFS may be able to move a directory tree into a "new filesystem" without affecting ctime.
If gtar would be implemented correctly, it would need to maintain a huge database for the lifetime of the filesystems tobackup.
Star on the other side only needs to remember the last backup dates for every different incremental level.
You also seem to miss the point that in case you like to move a sub-tree into a different filesystem (something that is extremely unlikely to happen with modern filesystems), you need to do a lot of adminstrative work anyway. Adding a new FS to the list of FSs in the backup system is a minor task in this context.
What star allows you to do is to have incrementals that are as reliable as ufsdump but that are independent from the OS and the FS.
You are correct, you could in theory recover the data but this must be done manually.
How do you recover from the mount point change case for star? If it happens on the source side, do you lose data?
Star handles directory changes correctly, there is no loss of data.
If you remove a sub-tree on the source side, this is also done at incremental restore time.
So it all that time you have not mounted a new volume somewhere? No remote backups of hosts where someone else might add space? No one ever wanted to restore onto a different mount topology than the one where the backups were taken? Being able to do those things is the reason I use tar instead of filesystem-dependent dump.
Star of course can do what you like. You just need to create more than one backup once you split filesystems. It would be easy to handle if you like to use it....
I handled it by pointing amanda at remote systems. And I expected it to keep those systems backed up even if someone else mounted some new disks in places where they needed some more space. I don't see how star fits into that scheme.
Well as mentioned before, if you reconfigure your server, you also need to reconfigure the backup to match the new situation at the server.
AFAIK, amanda has too few features or there are no people who are willing to put efforts in adopting to star.
Star simply does not do what amanda needs - or did not the last time I looked. Amanda needs a way to quickly estimate the size of a run for its brilliant feature of automatically balancing the mix of fulls and incrementals to ensure that you get at least an incremental of every target every night plus a full within the number of tapes in your rotation. And it can't fail on incrementals just because someone replaced a directory on a remote machine with a mount point.
This is an unproven claim from you. I grant you that star supports all you need for a reliable incremental backup and restore.
I currently cannot believe that there is really any important feature that is missing in star.
Those features obviously aren't important to you, but they are enough to keep me - or any amanda user - from considering star. And amanda
If amanda does not support star, it seems that the amanda people are not interested in reliable backups.
Star supports everything you need, if Amanda does not support star, this is obviously a filure at amanda's side.
Jörg
On Wed, Feb 8, 2012 at 6:20 AM, Joerg Schilling < Joerg.Schilling@fokus.fraunhofer.de> wrote:
Les Mikesell lesmikesell@gmail.com wrote:
(For bystanders following this conversion, it is only about the non-standard extensions (--listed-incremental, etc.) to tar. Standard modes are fine on either gnutar or star).
My testcase for star was moving a subdirectory of what my backup runs
covered onto a mounted volume. Star failed and I stopped testing it any further.
So you tried to do something that cannot work for a filesystem
oriented program.
It does work with GNUtar.
In general, this does not work with gtar. You are exactly in the area that caused my conclusion that gtar is not useful at all for incremental backups.
No, a general file-oriented case would handle FAT and NTFS filesystems, and continue to work even if you mount one of those in the path of your planned backup or restore. Star fails that.
This is not a problem from star but you just did not use star the
right way.
No, star does not do what I need, so I don't use it at all.
Star does what you need, you are just using it the wrong way.
I don't plan to arrange my life or my data around star's restricted capabilities.
Note that gtar uses a big database that is needed at the dump side and
that
still does not hold enough data to let gtar work correctly.
How is it not correct?
As mentioned, gtar has major problems whe directory is envolved. How many
successful restores did you pass?
All of them. Even the contrived case you sent me when we had this same conversation years ago. The saves always work and the only issue there has ever been is when a restored directory is supposed to be deleted and replaced by a file of the same name as an incremental update (or maybe the other way around...). It's something that has never happened in years of operations and recoverable if it does - just delete the offending item and repeat the restore of the thing that ends up there. And it might be fixed now - I haven't been concerned enough to check.
The difference between gtar and star is that gtar never works correctly
for all
possible changes, while star works correct and you just need to follow it's constraints. As gtar does not work on a simple testcase, I am not willing to trust in gtar's incrementals.
Here's the simple test case. Plug in a FAT-formatted USB key in the path of your backup. Change something on it and repeat with your incremental mode. Restore it with or without the key mounted in that position. If that doesn't preserve the data in a way you can restore, I'm not willing to trust it for my backups. Or if you are biased against FAT, use the filesystem of your choice, just do it dynamically.
Except when it isn't, because it is tied to the filesystem, not the directory tree structure. If I wanted filesystem dependencies I'd use dump. I expect tar to follow directory trees and be agnostic to mount points.
It may be that you miss the fact that modern filesystems like e.g. ZFS may
be able to move a directory tree into a "new filesystem" without affecting ctime.
That breaks the definition of ctime, doesn't it? But it doesn't matter to GNUtar because if it hits a directory in a location that doesn't match what is listed in its -listed-incremental file it will back it up with everything below it.
If gtar would be implemented correctly, it would need to maintain a huge
database for the lifetime of the filesystems tobackup.
No, it needs a small file for each state of a tree where you plan to do a subsequent higher level incremental. And amanda manages this for you.
You also seem to miss the point that in case you like to move a sub-tree into a different filesystem (something that is extremely unlikely to happen with modern filesystems), you need to do a lot of adminstrative work anyway.
Sometimes it is, sometimes it is plugging in a usb device, and sometimes it is someone else's administrative work, unrelated to the backup configuration.
Adding a new
FS to the list of FSs in the backup system is a minor task in this context.
What star allows you to do is to have incrementals that are as reliable as ufsdump but that are independent from the OS and the FS.
That would be better stated as 'as filesytem dependent as ufsdump'. It is not independent from the FS at all or I'd be able to point it at an arbitrary directory without knowing or caring if it encompasses a whole filesystem or has additional mounts. Or if the layout is the same during the restore.
I handled it by pointing amanda at remote systems. And I expected it
to keep those systems backed up even if someone else mounted some new disks in places where they needed some more space. I don't see how star fits into that scheme.
Well as mentioned before, if you reconfigure your server, you also need to reconfigure the backup to match the new situation at the server.
They are different machines. Why should I need to know if someone else moved /home to a separate volume just to correctly copy the data that has changed since the last backup run?
AFAIK, amanda has too few features or there are no people who are
willing to
put efforts in adopting to star.
Star simply does not do what amanda needs - or did not the last time I looked. Amanda needs a way to quickly estimate the size of a run for its brilliant feature of automatically balancing the mix of fulls and incrementals to ensure that you get at least an incremental of every target every night plus a full within the number of tapes in your rotation. And it can't fail on incrementals just because someone replaced a directory on a remote machine with a mount point.
This is an unproven claim from you. I grant you that star supports all you need for a reliable incremental backup and restore.
It does, under the same restricted conditions dump would - and amanda already works with dump. The reason I use tar instead of dump is to not be restricted to those conditions.
I currently cannot believe that there is really any important feature
that is
missing in star.
Those features obviously aren't important to you, but they are enough to keep me - or any amanda user - from considering star. And amanda
If amanda does not support star, it seems that the amanda people are not interested in reliable backups.
No, they already have dump for the things that star could do.
Star supports everything you need, if Amanda does not support star, this is obviously a filure at amanda's side.
It just doesn't add capabilities the way GNUtar does so there is no need for it. And I don't see any support for size estimates like dump provides, which are very necessary for amanda to arrange each set of incremental and full runs to fill a tape. Can you do anything that matches the speed of a gnutar run of : 'tar --totals -cf /dev/null /path' ? I'll agree that what GNUtar does in that special case is weird (or at least that the reason it does it is weird) if you'll agree that it is necessary for practical reasons.
Les Mikesell lesmikesell@gmail.com wrote:
In general, this does not work with gtar. You are exactly in the area that caused my conclusion that gtar is not useful at all for incremental backups.
No, a general file-oriented case would handle FAT and NTFS filesystems, and continue to work even if you mount one of those in the path of your planned backup or restore. Star fails that.
Well, I verified that gtar fails with even a simple case and I did not see any problem with star since 6 years even with thousands of incremental restores. So I am not sure what you are talkin about.
How many successful automatical incremental restores did you try with gtar so far? 1/12 dozen?
Star's incrementals are file oriented and this is why it is able to offer a OS and FS independent incremental backup.
Given the fact that gtar completely fails with it's incrementals even in simple cases, I cannot understand why people still use gtar's incrementals. Note that doing a backup is only half of the job and gtar fails with the restore...
Star does what you need, you are just using it the wrong way.
I don't plan to arrange my life or my data around star's restricted capabilities.
This is a really strange claim, as you obviously follow the rules in case you create a new filesystem.
Now please explain me why you are doing backups with a program that is known to fail with incremental restores?
As mentioned, gtar has major problems whe directory is envolved. How many
successful restores did you pass?
All of them. Even the contrived case you sent me when we had this same conversation years ago. The saves always work and the only issue there has ever been is when a restored directory is supposed to be deleted and replaced by a file of the same name as an incremental update (or maybe the other way around...). It's something that has never happened in years of operations and recoverable if it does - just delete the offending item and repeat the restore of the thing that ends up there. And it might be fixed now - I haven't been concerned enough to check.
Well gtar failed with my first simple test already, why do you still believe it works?
Here's the simple test case. Plug in a FAT-formatted USB key in the path of your backup. Change something on it and repeat with your incremental mode. Restore it with or without the key mounted in that position. If that doesn't preserve the data in a way you can restore, I'm not willing to trust it for my backups. Or if you are biased against FAT, use the filesystem of your choice, just do it dynamically.
Why do you like me to do some really strange test while your backup method is know not to work at all?
Would'nt it be better to first switch to a working backup program?
As you stay with gtar, it seems that rather you are biased.
Also note that reliable backups need to use snapshots, so please rethink your backup method under the right constraints.
It may be that you miss the fact that modern filesystems like e.g. ZFS may
be able to move a directory tree into a "new filesystem" without affecting ctime.
That breaks the definition of ctime, doesn't it? But it doesn't matter to GNUtar because if it hits a directory in a location that doesn't match what is listed in its -listed-incremental file it will back it up with everything below it.
No, ZFS is fully compatible with POSIX.
ZFS permits file renames aross different filesystems in case these filesystems are on the same storage pool.
See POSIX for rename():
[EXDEV] [CX] [Option Start] The links named by new and old are on different file systems and the implementation does not support links between file systems. [Option End]
So if really like you move a sub-tree into a separate filesystem, only the top level directories are granted to get ctime updated.
BTW: Why should gtar work in your case when it already fails with incremental restores for simple directory rename operations?
If gtar would be implemented correctly, it would need to maintain a huge
database for the lifetime of the filesystems tobackup.
No, it needs a small file for each state of a tree where you plan to do a subsequent higher level incremental. And amanda manages this for you.
You seem to have a strange concept of what's a "small" file.
The file we are talking about may easily reach a size of several GB.
Star needs less that 100 bytes per level of incrementals.
So if you do just one level 0 dump and any amount of level 1 dumps, this star state file will be less than 200 bytes per filesystem.
Sometimes it is, sometimes it is plugging in a usb device, and sometimes it is someone else's administrative work, unrelated to the backup configuration.
If you like to do reliable backups, you need to snapshots anyway. Believe me that the snapshot does not mirror the mount you are talking about, so it is obvious that the gtar concept is even wrong if you asume a reliable overall environment.
What star allows you to do is to have incrementals that are as reliable as ufsdump but that are independent from the OS and the FS.
That would be better stated as 'as filesytem dependent as ufsdump'. It is not independent from the FS at all or I'd be able to point it at an arbitrary directory without knowing or caring if it encompasses a whole filesystem or has additional mounts. Or if the layout is the same during the restore.
???
Well as mentioned before, if you reconfigure your server, you also need to reconfigure the backup to match the new situation at the server.
They are different machines. Why should I need to know if someone else moved /home to a separate volume just to correctly copy the data that has changed since the last backup run?
Because such a change would affect snapshots that are definitely needed for reliable backups.
This is an unproven claim from you. I grant you that star supports all you need for a reliable incremental backup and restore.
It does, under the same restricted conditions dump would - and amanda already works with dump. The reason I use tar instead of dump is to not be restricted to those conditions.
You told me that you use gtar and gtar has the limitation that incremental restrores do not work in many cases.
I prefer to live with a method that may have any restriction, in case it is able to support reliable incremental restores. This is what you get from star.
If amanda does not support star, it seems that the amanda people are not interested in reliable backups.
No, they already have dump for the things that star could do.
Did you ever think about the fact that the various BSD dump descendants do not share a common archive format?
Star being OS and FS independent allows you to e.g. restore any incremental on any OS and does not force you to first manually port the actual "restore" program to your current restore OS.
Star supports everything you need, if Amanda does not support star, this is obviously a filure at amanda's side.
It just doesn't add capabilities the way GNUtar does so there is no need for it. And I don't see any support for size estimates like dump provides, which are very necessary for amanda to arrange each set of incremental and full runs to fill a tape. Can you do anything that matches the speed of a gnutar run of : 'tar --totals -cf /dev/null /path' ? I'll agree that what GNUtar does in that special case is weird (or at least that the reason it does it is weird) if you'll agree that it is necessary for practical reasons.
I cannot tell when gtar introduced this strange method that makes performance tests hard....
Star introduced the -onull option more than 15 years ago and I asume that potential users start with reading the man page.
One problem for you with star may be that star does not match the performance of gtar.... star is aprox 2-3x faster than gtar ;-)
I hope you can live with this limitation.....
Jörg
On Fri, Feb 10, 2012 at 6:05 AM, Joerg Schilling < Joerg.Schilling@fokus.fraunhofer.de> wrote:
No, a general file-oriented case would handle FAT and NTFS filesystems,
and
continue to work even if you mount one of those in the path of your
planned
backup or restore. Star fails that.
Well, I verified that gtar fails with even a simple case and I did not see any problem with star since 6 years even with thousands of incremental restores. So I am not sure what you are talkin about.
Gtar doesn't fail, it just requires some extra steps to accommodate some very unlikely circumstances. Your star runs will not work at all in very likely circumstances (which you conveniently avoid testing).
How many successful automatical incremental restores did you try with gtar so far? 1/12 dozen?
Enough to know it works.
Star's incrementals are file oriented and this is why it is able to offer a
OS and FS independent incremental backup.
Our definitions of 'OS and FS independent' are very different. I expect that to mean that I can actually change the FS and have things keep working. Star doesn't. And it doesn't work over nfs, on fat, ntfs, and on and on.
I don't plan to arrange my life or my data around star's restricted
capabilities.
This is a really strange claim, as you obviously follow the rules in case you create a new filesystem.
What rules? If you are FS independent, you follow a directory tree, not a rulebook. I want the content of my files backed up whether they are stored in a way that matches your restricted rules or not. I don't want to have to take a whole filesystem. I don't want a run to be restricted to a single filesystem.
Now please explain me why you are doing backups with a program that is known
to fail with incremental restores?
In a word, amanda. If my backups don't fit on the tape there are no backups at all, and the only reason for doing incrementals at all is that you have to use them to make the data fit. I'm not interested in calculating the right mix of incremental levels on a whole bunch of directory trees to make each night's run fit. Amanda does that automatically. Amanda can use dump or gnutar. Linus Torvald has claimed that dump is not safe on live filesystems. I assume he's an expert on the topic (and without looking at the code, I'd guess that star uses the same approach to determine what to take in incrementals). That leaves gnutar. I can live with a very unlikely chance that I'll have to give a few extra commands to restore compared to not getting backups or having to compute the size of incremental levels manually every day.
And then there was the fact that at the time I implemented amanda, star did not do incremental restores anyway...
Well gtar failed with my first simple test already, why do you still believe it works?
Because the data is on the tape and can be extracted if you move the same-named but different object type thing out of the way first (and admit that it wouldn't be there except in a contrived test). The commands to do it might not be convenient, but it is much, much better than using a tool that in many circumstances won't have the data at all, or can't restore it onto a different mount topology.
Here's the simple test case. Plug in a FAT-formatted USB key in the path
of your backup. Change something on it and repeat with your incremental mode. Restore it with or without the key mounted in that position. If that doesn't preserve the data in a way you can restore, I'm not willing
to
trust it for my backups. Or if you are biased against FAT, use the filesystem of your choice, just do it dynamically.
Why do you like me to do some really strange test while your backup method is know not to work at all?
OK, here's a test that is a very common scenario. Boot a dual-boot windows/linux laptop into linux, back it up in a way that includes the mounted VFAT and NTFS partitions. Make changes on the VFAT or NTFS locations under linux or windows. Make an incremental backup that includes all the changes.
Would'nt it be better to first switch to a working backup program?
As you stay with gtar, it seems that rather you are biased.
Strictly a pragmatic choice. GNUtar works, where star did not.
Also note that reliable backups need to use snapshots, so please rethink your backup method under the right constraints.
But FS snapshots take extra disk space and not all file systems provided them back then. That does not make the file contents less important.
So if really like you move a sub-tree into a separate filesystem, only the
top level directories are granted to get ctime updated.
BTW: Why should gtar work in your case when it already fails with incremental restores for simple directory rename operations?
The one incremental restore inconvenience you found has nothing to do with the way --listed-incremental backups work. They are not tied to filesystems or mount points. They follow directory trees.
If gtar would be implemented correctly, it would need to maintain a huge
database for the lifetime of the filesystems tobackup.
No, it needs a small file for each state of a tree where you plan to do a subsequent higher level incremental. And amanda manages this for you.
You seem to have a strange concept of what's a "small" file.
The file we are talking about may easily reach a size of several GB.
So first we need room for filesystem snapshots, but now storing a file big enough to hold the equivalent of a directory listing is a problem?
What star allows you to do is to have incrementals that are as reliable
as
ufsdump but that are independent from the OS and the FS.
That would be better stated as 'as filesytem dependent as ufsdump'. It is not independent from the FS at all or I'd be able to point it at an arbitrary directory without knowing or caring if it encompasses a whole filesystem or has additional mounts. Or if the layout is the same during the restore.
???
Can you, using star, make incremental backups of a directory tree that work without knowing./caring what file systems hold that tree or where the roots or mount points are?
If you have incremental backups that were made under the restrictive conditions that star requires, can you restore them correctly on a different topology? For example, you backed up / from a small machine with no mounted partitions. It breaks and you replace it with a new machine that has an additional drive that you mount as /home. Can you restore the incremental levels of the /home portion of the backup onto this replacement system, along with some things still on the root partition?
No, they already have dump for the things that star could do.
Did you ever think about the fact that the various BSD dump descendants do not share a common archive format?
Does that matter if I have the code for the one that reads my format?
Star being OS and FS independent allows you to e.g. restore any incremental
on any OS and does not force you to first manually port the actual "restore" program to your current restore OS.
That would be nice if it were true. The most likely non-native target would be windows. Should I plan on that working? What about the simpler case of the same OS but an added mount point in the layout where I want a portion of the restore to go?
I cannot tell when gtar introduced this strange method that makes
performance tests hard....
I expect actual backups to be i/o bound to a point that whatever else they do performance-wise is irrelevant.
Star introduced the -onull option more than 15 years ago and I asume that
potential users start with reading the man page.
I didn't see an option for reporting size in the man page - it might actually be adaptable if anyone still cared about tapes. But, I find the man page very confusing. Most of the incremental options say it has to be a single Posix-conforming FS but the -dump section talks about spanning more than one.
Anyway it is mostly irrelevant for me now that online backups are feasible in terms of disk space and bandwidth to offsite storage. It is so much easier to click on a version of a file you want in backuppc's web interface than to request the right series of offsite tapes to be shipped back and restore them in the right order to get to the date it should have been there.
Les Mikesell lesmikesell@gmail.com wrote:
Well, I verified that gtar fails with even a simple case and I did not see any problem with star since 6 years even with thousands of incremental restores. So I am not sure what you are talkin about.
Gtar doesn't fail, it just requires some extra steps to accommodate some very unlikely circumstances. Your star runs will not work at all in very likely circumstances (which you conveniently avoid testing).
Trying to understand you leads me to the assumption that you of course know that gtar fails. As you seem to claim that there is a workaround but don't mention it, it seems that you like other people to fail when trying to restore incremental backups that have been made with gtar.
How many successful automatical incremental restores did you try with gtar so far? 1/12 dozen?
Enough to know it works.
Well, I did one test only and I know that gtar does not do what it should when restoring incrementals.
In addition, gtar incrementals are not space efficient:
If you e.g. have a 1 TB filesystem with one directory at top level and rename that directory, you will end up with a 1 TB incremental if you use gtar. Looking at the way gtar works, I would assume that gtar needs an intermediate disk space of 2 TB to restore the data (if the restore works at all).
Star will create an incremental archive of 10kB for the same change and star does not need additional space on the restore media.
Also note that star is able to backup ACLs and extended attributes but gtar does not have the needed features to do so.
Our definitions of 'OS and FS independent' are very different. I expect that to mean that I can actually change the FS and have things keep working. Star doesn't. And it doesn't work over nfs, on fat, ntfs, and on and on.
Do you like to tell everbody that it makes no sense to believe you, or why do you write claims that are obviously wrong?
Star gives you FS and OS independence and star is even at least 30% faster than ufsdump (under all conditions) or any fork that was created from ufsdump (such es ext*..).
What rules? If you are FS independent, you follow a directory tree, not a rulebook. I want the content of my files backed up whether they are stored in a way that matches your restricted rules or not. I don't want to have to take a whole filesystem. I don't want a run to be restricted to a single filesystem.
You again harm your credibility by writing false suggestions.
Star of course allows you to backup parts of a filesystem (e.g. a directory sub-tree). It seems that your problem is that you are not willing to inform you about any other program that gtar and it seems that you are not willing to think about the results from gtar problems.
In a word, amanda. If my backups don't fit on the tape there are no backups at all, and the only reason for doing incrementals at all is that you have to use them to make the data fit. I'm not interested in calculating the right mix of incremental levels on a whole bunch of directory trees to make each night's run fit. Amanda does that
You again harm your credibility by writing false suggestions.
Star is able to automatically deal with the unpredictable size of modern tapes with HW compression, while amanda fails completely and needs to assume the smallest possible size. This is because amanda does not make use of the related features of the backup utility. Star can detect the actual end of a tape in write mode on the fly and then request a media change.
BTW: what amanda does may make sense with gtar as gtar frequently fails to accept own followup volumes but this is a problem that is absent with star.
automatically. Amanda can use dump or gnutar. Linus Torvald has claimed that dump is not safe on live filesystems. I assume he's an expert on the topic (and without looking at the code, I'd guess that star uses the same approach to determine what to take in incrementals). That leaves gnutar.
Well, besides the fact that Torvalds is not known of being a filesystem expert it is a bad idea to invert a true claim as you do here and believe that this is still true.
ufsdump is not safe on life filesystems but gtar and star are not safe too.
The most insecure backup program for a life file system if ufsdump or it's forks. This is because ufsdump first reads all fs meta data and later reads a raw filesystem at block level without taking care of changes.
gtar is still insecure as it traverses the directory tree and is affected by changes on that tree while the backup is running.
star is definitely less insecure than gtar as star does not need to keep a database of the filesystem structure in order to be able to work.
In any case, I know of no serious sysadmin that makes backups from a life filesystem without using filesystem snapshots. If you however use snapshots that are needed for a reliable incremental backup, you cannot backup directory trees the way you seem to like.....
Star matches the reliable case and if you use gtar without snapshots, you are creating unreliable backups.
I can live with a very unlikely chance that I'll have to give a few extra commands to restore compared to not getting backups or having to compute the size of incremental levels manually every day.
And then there was the fact that at the time I implemented amanda, star did not do incremental restores anyway...
Star implements reliable incremental restores since 7 years, so you seem to be living in a long gone past.
Well gtar failed with my first simple test already, why do you still believe it works?
Because the data is on the tape and can be extracted if you move the same-named but different object type thing out of the way first (and admit that it wouldn't be there except in a contrived test). The commands to do it might not be convenient, but it is much, much better than using a tool that in many circumstances won't have the data at all, or can't restore it onto a different mount topology.
Well star always works fully automatically with incrementals.
What you like to do with gtar may require hours of manual intervention and is still based on an unreliable basic method (as you cannot use the needed snapshots).
OK, here's a test that is a very common scenario. Boot a dual-boot windows/linux laptop into linux, back it up in a way that includes the mounted VFAT and NTFS partitions. Make changes on the VFAT or NTFS locations under linux or windows. Make an incremental backup that includes all the changes.
You are trying to do something in an unrelibale way that can be done reliably and fully automatically using star instead of gtar. So why do you still use gtar?
As you stay with gtar, it seems that rather you are biased.
Strictly a pragmatic choice. GNUtar works, where star did not.
Well gtar does not work and all your replies admit (even if you try to hide this fact) that gtar does not work automatically and that you need to add some secret manual work that may take a long time in order to go on in case of failures.
Star on the other side works reliably and automatically.
But FS snapshots take extra disk space and not all file systems provided them back then. That does not make the file contents less important.
Just do not use filesystems that do not support snapshots for professional work and if you are forced to do backups on a fs without snapshots at least use the backup method with the least sensitivity against inconsistencies. This preferred backup method is star if we are comparing ufsdump, gtar and star.
The file we are talking about may easily reach a size of several GB.
So first we need room for filesystem snapshots, but now storing a file big enough to hold the equivalent of a directory listing is a problem?
There of course is a difference between the need to store some data for a few minutes (this is what an incremental star backup takes) and the need to store a lot of data for the whole life-cycle of the FS (which is what you need to do with gtar).
Also note that many filesystems store the actual data for the snapshot in the free blocks of the filesystem. In these cases, you do not even notice the need for space to support the incremental.
Can you, using star, make incremental backups of a directory tree that work without knowing./caring what file systems hold that tree or where the roots or mount points are?
Of course, if there are no mount points in that tree.
As mentioned before, the star method is enforced anyway if you like to have reliable incrementals that need snapshots. So gtar will not do what you like in case you implement reliable backups.
Did you ever think about the fact that the various BSD dump descendants do not share a common archive format?
Does that matter if I have the code for the one that reads my format?
If you like to restore more meta data than UNIX had in 1981, this definitely matters. Also note that star is at least 30% faster than ufsdump, so why do you like to use ufsdump when you can have star?
That would be nice if it were true. The most likely non-native target would be windows. Should I plan on that working? What about the simpler case of the same OS but an added mount point in the layout where I want a portion of the restore to go?
If you try to create backups on Win-DOS without using Cygwin or some equivalent system, you are lost, as MS ignores standards and does not include st_ino in struct stat. If you however use Cygwin, it works...
I expect actual backups to be i/o bound to a point that whatever else they do performance-wise is irrelevant.
If you were true, why is gtar slower than star?
If you are backing up to a real tape, this may be extremely important because being a bit too slow may cause the tape to fail streaming.
Star introduced the -onull option more than 15 years ago and I asume that
potential users start with reading the man page.
I didn't see an option for reporting size in the man page - it might actually be adaptable if anyone still cared about tapes. But, I find the man page very confusing. Most of the incremental options say it has to be a single Posix-conforming FS but the -dump section talks about spanning more than one.
If you did never work with star, why do you claim so many "failures" of star?
I recommend you to start using star and talk about it after you learn how it works. Star implements so many features, that you need to use it for some days to understand the concept, but I know nobody who did not switch to star after understanding the concept.
Most of the features from star I am using on a daily base are missing in gtar.
Jörg
On Tue, Feb 14, 2012 at 12:41 PM, Joerg Schilling Joerg.Schilling@fokus.fraunhofer.de wrote:
Gtar doesn't fail, it just requires some extra steps to accommodate some very unlikely circumstances. Your star runs will not work at all in very likely circumstances (which you conveniently avoid testing).
Trying to understand you leads me to the assumption that you of course know that gtar fails. As you seem to claim that there is a workaround but don't mention it, it seems that you like other people to fail when trying to restore incremental backups that have been made with gtar.
The only thing I've ever seen fail was where the full or earlier incremental contained a directory which was subsequently replaced with a plain file with the same name that was included in a higher level incremental. The gnutar code circa 2004 didn't correctly remove the directory that should be overwritten during the incremental restore. But the error was obvious and you already know how to remove a directory. Do it, repeat the restore of the missing item and everyone is happy. But that was a long time ago, maybe things have changed.
Well, I did one test only and I know that gtar does not do what it should when restoring incrementals.
So you don't check for changes either?
If you e.g. have a 1 TB filesystem with one directory at top level and rename that directory, you will end up with a 1 TB incremental if you use gtar. Looking at the way gtar works, I would assume that gtar needs an intermediate disk space of 2 TB to restore the data (if the restore works at all).
Star will create an incremental archive of 10kB for the same change and star does not need additional space on the restore media.
Except that star won't do it at all if you want it to follow an arbitrary directory tree.
Our definitions of 'OS and FS independent' are very different. I expect that to mean that I can actually change the FS and have things keep working. Star doesn't. And it doesn't work over nfs, on fat, ntfs, and on and on.
Do you like to tell everbody that it makes no sense to believe you, or why do you write claims that are obviously wrong?
Can it follow directory trees across mount points getting incrementals right? If I'm wrong, show me the commands needed to make that work.
Star of course allows you to backup parts of a filesystem (e.g. a directory sub-tree).
And follow across mount points?
In a word, amanda. If my backups don't fit on the tape there are no backups at all, and the only reason for doing incrementals at all is that you have to use them to make the data fit. I'm not interested in calculating the right mix of incremental levels on a whole bunch of directory trees to make each night's run fit. Amanda does that
You again harm your credibility by writing false suggestions.
Star is able to automatically deal with the unpredictable size of modern tapes with HW compression, while amanda fails completely and needs to assume the smallest possible size. This is because amanda does not make use of the related features of the backup utility. Star can detect the actual end of a tape in write mode on the fly and then request a media change.
I don't want a media change. No one is going to be there to change it. I want the software to compute the mix of fulls and incrementals that will fit my single daily tape. Amanda did that faithfully for many years..
And then there was the fact that at the time I implemented amanda, star did not do incremental restores anyway...
Star implements reliable incremental restores since 7 years, so you seem to be living in a long gone past.
Yes, I believed what you said here: http://blog.gmane.org/gmane.comp.archivers.star.user/month=20040701 and haven't had any more reason to check again than you have to check gnutar. Amanda kept doing what I needed until the tape drives wore out many years later, with no more attention than swapping the tape sometime during the day.
What you like to do with gtar may require hours of manual intervention and is still based on an unreliable basic method (as you cannot use the needed snapshots).
There is nothing that would prevent you from using snapshots with tar, but a filesystem snapshot isn't really enough to ensure that your application's view of the data is consistent anyway. Most things don't care and the ones that do have application-level ways to dump their data consistently.
You are trying to do something in an unrelibale way that can be done reliably and fully automatically using star instead of gtar. So why do you still use gtar?
Because when I needed it, star wasn't even usable. Even if it had been able to do an incremental restore, it didn't work with amanda.
So first we need room for filesystem snapshots, but now storing a file big enough to hold the equivalent of a directory listing is a problem?
There of course is a difference between the need to store some data for a few minutes (this is what an incremental star backup takes) and the need to store a lot of data for the whole life-cycle of the FS (which is what you need to do with gtar).
What good does being able to release that space do if you have to reserve it anyway? I don't see how it has any other use.
Can you, using star, make incremental backups of a directory tree that work without knowing./caring what file systems hold that tree or where the roots or mount points are?
Of course, if there are no mount points in that tree.
And if there are? I don't see why you consider gnutar's issue with a directory being replaced with a file of the same name to be a big problem and completely ignore the issue star has with a directory being replaced by a mount point. I've done the latter much more often than the former. It's not a reason to make backups of the file contents fail.
I didn't see an option for reporting size in the man page - it might actually be adaptable if anyone still cared about tapes. But, I find the man page very confusing. Most of the incremental options say it has to be a single Posix-conforming FS but the -dump section talks about spanning more than one.
If you did never work with star, why do you claim so many "failures" of star?
One failing is enough. Isn't that why you dismissed gnutar? And for me, being unable to do incrementals that cross mount points is a such a failing. And if I understand it correctly, even if I kept changing the backup runs to match the source mount layouts, I would not be able to restore correctly if I wanted the replacement system to have a different topology that would place a mount point in a location that was previously a directory . Did I misunderstand that?
I recommend you to start using star and talk about it after you learn how it works. Star implements so many features, that you need to use it for some days to understand the concept, but I know nobody who did not switch to star after understanding the concept.
I hope to never need tapes again, except possibly for archival storage where incrementals would not be a factor. And going to disk targets, rsync is much nicer than either gnutar or star.
-
Joerg.Schilling＠fokus.fraunhofer.de -
Les Mikesell