Hey folks,
I looked at the man page and don't see any way to do this - maybe it is a function of the compression program used I dunno.
Is there any way to get gtar to report on the compression it achieved?
I can't just check file sizes because I'm writing data to tape.
The basic problem is that I know how much data is there to begin with but I don't know how much room it took up on the tape so I have no idea how much room is left on the tape.
thanks, -Alan
On Wed, Feb 1, 2012 at 8:18 AM, Alan McKay alan.mckay@gmail.com wrote:
Hey folks,
I looked at the man page and don't see any way to do this - maybe it is a function of the compression program used I dunno.
Is there any way to get gtar to report on the compression it achieved?
I can't just check file sizes because I'm writing data to tape.
The basic problem is that I know how much data is there to begin with but I don't know how much room it took up on the tape so I have no idea how much room is left on the tape.
There is a --totals option, but that is before compression. I don't think there is a way to do it. You can use -f /dev/null and --totals to get a quick estimate of the uncompressed size of what matches the tar arguments (for full/incremental). Gnu tar 'special cases' output to /dev/null and doesn't bother actually reading the data but it adds up the sizes from the directory. That was added so amanda can do estimates that let it decide how to mix the fulls and incrementals each day to fill a tape.
There is a --totals option, but that is before compression. I don't think there is a way to do it.
Dang. THere is a "tell" command on "mt" which tells you what block number you are on, but according to the man page only exists for some types of drive. And evidently not mine :-(
That would have worked with some simple math.
On Wed, Feb 1, 2012 at 9:59 AM, Alan McKay alan.mckay@gmail.com wrote:
There is a --totals option, but that is before compression. I don't think there is a way to do it.
Dang. THere is a "tell" command on "mt" which tells you what block number you are on, but according to the man page only exists for some types of drive. And evidently not mine :-(
That would have worked with some simple math.
Is there some reason you aren't using amanda? Give it some holding disk space and it will run multiple backups at once, buffering on disk, and figure out how they should go on the tape for you.
Is there some reason you aren't using amanda? Give it some holding disk space and it will run multiple backups at once, buffering on disk, and figure out how they should go on the tape for you.
I'm archiving, not backing up.
I looked at Amanda for a few days and it would be really clunky to do what I want.
Anyway I found this :
[root@solexa-db tmp]# export GZIP=-v [root@solexa-db tmp]# tar czf files.tar.gz file{1,2,3,4,5} 98.4%
So I'm golden :-)
On Wed, Feb 1, 2012 at 10:10 AM, Alan McKay alan.mckay@gmail.com wrote:
Is there some reason you aren't using amanda? Give it some holding disk space and it will run multiple backups at once, buffering on disk, and figure out how they should go on the tape for you.
I'm archiving, not backing up.
Is there some limit to the number of tapes it will track or the length of a cycle?
I looked at Amanda for a few days and it would be really clunky to do what I want.
I haven't used it for a while, but I thought it had an indexing mechanism that would let you tell it what you want and it would tell you the tapes you need and the order to restore them (for full + incremental cases). And it could re-index the tapes if you lost the disk copy. Maybe that doesn't fit your use, but it seemed handy.
I haven't used it for a while, but I thought it had an indexing mechanism that would let you tell it what you want and it would tell you the tapes you need and the order to restore them (for full + incremental cases). And it could re-index the tapes if you lost the disk copy. Maybe that doesn't fit your use, but it seemed handy.
In general it is massive overkill for what I"m doing. Even if I wanted to switch backup solutions and move my backups to Amanda it would not be worthwhile to get this as an add-on because of the nature of the data I am dealing with.
Case in point I have about 300G of data that one of the scientists copied over to my server from a piece of scientific equipment. That 300G was never in my backups and I never want it to be. But he needs it archived.
Amanda is just way, way too too big for this.
In 2 weeks I've got a program written that is tailored exactly to our needs. THat's probably less time than it would have taken me to deploy Amanda. And it would not have been tailored precisely to our needs.
On Wed, Feb 1, 2012 at 10:22 AM, Alan McKay alan.mckay@gmail.com wrote:
I haven't used it for a while, but I thought it had an indexing mechanism that would let you tell it what you want and it would tell you the tapes you need and the order to restore them (for full + incremental cases). And it could re-index the tapes if you lost the disk copy. Maybe that doesn't fit your use, but it seemed handy.
In general it is massive overkill for what I"m doing. Even if I wanted to switch backup solutions and move my backups to Amanda it would not be worthwhile to get this as an add-on because of the nature of the data I am dealing with.
Case in point I have about 300G of data that one of the scientists copied over to my server from a piece of scientific equipment. That 300G was never in my backups and I never want it to be. But he needs it archived.
Amanda is just way, way too too big for this.
In 2 weeks I've got a program written that is tailored exactly to our needs. THat's probably less time than it would have taken me to deploy Amanda. And it would not have been tailored precisely to our needs.
'Deploying' amanda is a matter of installing the rpm and editing a couple of config files about the tape drive, tapes, targets, and holding space. And maybe some firewall tweaking - but nothing really complicated. You get a lot of coverage of 'real-world' problems already built in that will be hard to match in a new program, but you do have to think the way it does...
From: Les Mikesell lesmikesell@gmail.com
'Deploying' amanda is a matter of installing the rpm and editing a couple of config files about the tape drive, tapes, targets, and holding space. And maybe some firewall tweaking - but nothing really complicated. You get a lot of coverage of 'real-world' problems already built in that will be hard to match in a new program, but you do have to think the way it does...
An issue with tar is that if you have an error somewhere in the tar... it is bad news... afio compress files individually and has other nice things... Maybe star does it too.
JD
On Wed, Feb 1, 2012 at 10:47 AM, John Doe jdmls@yahoo.com wrote:
'Deploying' amanda is a matter of installing the rpm and editing a couple of config files about the tape drive, tapes, targets, and holding space. And maybe some firewall tweaking - but nothing really complicated. You get a lot of coverage of 'real-world' problems already built in that will be hard to match in a new program, but you do have to think the way it does...
An issue with tar is that if you have an error somewhere in the tar... it is bad news... afio compress files individually and has other nice things... Maybe star does it too.
In a practical sense, I don't know if that even matters. I've never seen a tape drive that could recover and read past an error in the input anyway. Maybe back in floppy disk days...
On Wed, Feb 1, 2012 at 11:32 AM, Les Mikesell lesmikesell@gmail.com wrote:
'Deploying' amanda is a matter of installing the rpm and editing a couple of config files about the tape drive, tapes, targets, and holding space. And maybe some firewall tweaking - but nothing really complicated. You get a lot of coverage of 'real-world' problems already built in that will be hard to match in a new program, but you do have to think the way it does...
Well then I guess thinking the way it does is what I was having issues with. I did have trouble wrapping my head around it. And after a fair bit of googling (and if I'm not mistaken asking on this list) I really could find no examples of a configuration as simple as the one I was looking for.
I'm happy where I am. It is all very basic stuff (knock on wood - hee, hee). And I've got stuff that Amanda cannot possibly have since it is very specific to our environment.
On Wed, Feb 1, 2012 at 2:58 PM, Alan McKay alan.mckay@gmail.com wrote:
On Wed, Feb 1, 2012 at 11:32 AM, Les Mikesell lesmikesell@gmail.com wrote:
'Deploying' amanda is a matter of installing the rpm and editing a couple of config files about the tape drive, tapes, targets, and holding space. And maybe some firewall tweaking - but nothing really complicated. You get a lot of coverage of 'real-world' problems already built in that will be hard to match in a new program, but you do have to think the way it does...
Well then I guess thinking the way it does is what I was having issues with. I did have trouble wrapping my head around it. And after a fair bit of googling (and if I'm not mistaken asking on this list) I really could find no examples of a configuration as simple as the one I was looking for.
I always thought that was why it had a woman's name. You are better off just letting her do things her own way. It really does do a good job of automating and tracking everything and is exceptionally good at the case where you have one tape a day and you want to get at least an incremental of every machine every night and a full at least within the cycle where you start re-using tapes but preferably more often if there is space. It is probably adaptable to other scenarios but it may not fit yours very well. Once it is set up, all you have to do is swap the tape sometime during the day. It takes so little attention I let my setup run even after setting up backuppc until our last tape drive died.
On Wednesday, February 01, 2012 09:18:08 AM Alan McKay wrote:
The basic problem is that I know how much data is there to begin with but I don't know how much room it took up on the tape so I have no idea how much room is left on the tape.
What I would do is use the '-' special filename to pipe the uncompressed tar to stdout, pipe to the compressor of choice, then pipe to tee, and have one branch of the tee go to the tape and the other branch go to a program to count bytes.
On Wed, Feb 1, 2012 at 1:10 PM, Lamar Owen lowen@pari.edu wrote:
On Wednesday, February 01, 2012 09:18:08 AM Alan McKay wrote:
The basic problem is that I know how much data is there to begin with but I don't know how much room it took up on the tape so I have no idea how much room is left on the tape.
What I would do is use the '-' special filename to pipe the uncompressed tar to stdout, pipe to the compressor of choice, then pipe to tee, and have one branch of the tee go to the tape and the other branch go to a program to count bytes.
Or unless you are talking about many TB per run, decouple the compression from the tape run by sending the output to a disk file that you can sweep to tape later. There are lots of advantages, like not slowing down the tape streaming waiting for compression, being able to do multiple targets at once, and not only knowing the size of what is on the tape so far, but also the total compressed size of what you are going to start to write. Plus, of course, being able to do the tar runs at night when no one is there to swap tapes.
On Wed, Feb 1, 2012 at 2:10 PM, Lamar Owen lowen@pari.edu wrote:
What I would do is use the '-' special filename to pipe the uncompressed tar to stdout, pipe to the compressor of choice, then pipe to tee, and have one branch of the tee go to the tape and the other branch go to a program to count bytes.
The GZIP environment variable is working really well. It tells me the compression ratio and even send it to STDERR for me so I can easily separate that from the gtar output.
On 02/01/2012 04:18 PM, Alan McKay wrote:
Hey folks,
I looked at the man page and don't see any way to do this - maybe it is a function of the compression program used I dunno.
Is there any way to get gtar to report on the compression it achieved?
I can't just check file sizes because I'm writing data to tape.
The basic problem is that I know how much data is there to begin with but I don't know how much room it took up on the tape so I have no idea how much room is left on the tape.
You could ask tar to automatically request tape change when reaching end of tape:
-M, --multi-volume create/list/extract multi-volume archive
Lec
-
Alan McKay -
Alexandru Chiscan -
John Doe -
Lamar Owen -
Les Mikesell