Hi
We have an issue with one of our clients , they have a mail server with the /var/spool/imap partition as 150 GB . They need to take differential backup on to /backup partition ( a different HDD of total 250 GB space ) . We have tried dar , rsync, rdiff and impasync . But its is not sufficing the need as to take a lot of time and consumes a lot of I/O .
Is there any back up solution that you can think of , that can work in this situation - open source or proprietary
2010/2/24 Agnello George agnello.dsouza@gmail.com:
Hi
We have an issue with one of our clients , they have a mail server with the /var/spool/imap partition as 150 GB . They need to take differential backup on to /backup partition ( a different HDD of total 250 GB space ) . We have tried dar , rsync, rdiff and impasync . But its is not sufficing the need as to take a lot of time and consumes a lot of I/O .
So, you need to add more disk i/o? (so, add some disk space with faster raid?)
Take a look at: http://rdiff-backup.nongnu.org/
-- Eero
Dne 24.2.2010 10:00, Agnello George napsal(a):
Hi
We have an issue with one of our clients , they have a mail server with the /var/spool/imap partition as 150 GB . They need to take differential backup on to /backup partition ( a different HDD of total 250 GB space ) . We have tried dar , rsync, rdiff and impasync . But its is not sufficing the need as to take a lot of time and consumes a lot of I/O .
Is there any back up solution that you can think of , that can work in this situation - open source or proprietary
Is seems to me, that you are using mbox format. So, differential backup is hard to achieve. Migrate to maildir, every mail is a file, easy to backup differentially. David
On Wed, Feb 24, 2010 at 3:08 PM, David Hrbáč hrbac.conf@seznam.cz wrote:
Dne 24.2.2010 10:00, Agnello George napsal(a):
Hi
We have an issue with one of our clients , they have a mail server with the /var/spool/imap partition as 150 GB . They need to take differential backup on to /backup partition ( a different HDD of total 250 GB space )
.
We have tried dar , rsync, rdiff and impasync . But its is not sufficing
the
need as to take a lot of time and consumes a lot of I/O .
Is there any back up solution that you can think of , that can work in
this
situation - open source or proprietary
Is seems to me, that you are using mbox format. So, differential backup is hard to achieve. Migrate to maildir, every mail is a file, easy to backup differentially. David
backup directory structure is /var/spool/imap/a /adomain.com/a/agnello^dsouza/
2010/2/24 Agnello George agnello.dsouza@gmail.com:
On Wed, Feb 24, 2010 at 3:08 PM, David Hrbáč hrbac.conf@seznam.cz wrote:
Dne 24.2.2010 10:00, Agnello George napsal(a):
Hi
We have an issue with one of our clients , they have a mail server with the /var/spool/imap partition as 150 GB . They need to take differential backup on to /backup partition ( a different HDD of total 250 GB space ) . We have tried dar , rsync, rdiff and impasync . But its is not sufficing the need as to take a lot of time and consumes a lot of I/O .
Is there any back up solution that you can think of , that can work in this situation - open source or proprietary
Is seems to me, that you are using mbox format. So, differential backup is hard to achieve. Migrate to maildir, every mail is a file, easy to backup differentially. David
backup directory structure is /var/spool/imap/a /adomain.com/a/agnello^dsouza/
Well, does that directory contains one file or lot of files ?
Usually maildir structure is like dir++/tmp/current/new directories and each message is in own file on mailbox all files are inside one file.
-- Eero
2010/2/24 Eero Volotinen eero.volotinen@iki.fi
2010/2/24 Agnello George agnello.dsouza@gmail.com:
On Wed, Feb 24, 2010 at 3:08 PM, David Hrbáč hrbac.conf@seznam.cz
wrote:
Dne 24.2.2010 10:00, Agnello George napsal(a):
Hi
We have an issue with one of our clients , they have a mail server
with
the /var/spool/imap partition as 150 GB . They need to take differential backup on to /backup partition ( a different HDD of total 250 GB
space
) . We have tried dar , rsync, rdiff and impasync . But its is not
sufficing
the need as to take a lot of time and consumes a lot of I/O .
Is there any back up solution that you can think of , that can work in this situation - open source or proprietary
Is seems to me, that you are using mbox format. So, differential backup is hard to achieve. Migrate to maildir, every mail is a file, easy to backup differentially. David
backup directory structure is /var/spool/imap/a /adomain.com/a/agnello^dsouza/
Well, does that directory contains one file or lot of files ?
Usually maildir structure is like dir++/tmp/current/new directories and each message is in own file on mailbox all files are inside one file.
its in a maildir format and the structure slightly different from what i
mentioned earlier : root@server1 ~]# ls -la /var/spool/imap/a/user/ajay/ total 5180 -rw------- 1 cyrus mail 34616 Feb 24 16:02 4790. -rw------- 1 cyrus mail 4490 Feb 24 16:03 4791. -rw------- 1 cyrus mail 199253 Feb 24 16:07 4792. -rw------- 1 cyrus mail 22930 Feb 24 16:09 4793. -rw------- 1 cyrus mail 8485 Feb 24 16:11 4794. -rw------- 1 cyrus mail 12664 Feb 24 16:13 4795. -rw------- 1 cyrus mail 4296 Feb 24 16:13 4796. -rw------- 1 cyrus mail 5337 Feb 24 16:15 4797. -rw------- 1 cyrus mail 111030 Feb 24 16:21 4798. -rw------- 1 cyrus mail 4805500 Feb 24 16:23 4799. -rw------- 1 cyrus mail 22920 Feb 24 16:23 cyrus.cache -rw------- 1 cyrus mail 204 Dec 10 16:27 cyrus.header -rw------- 1 cyrus mail 896 Feb 24 16:23 cyrus.index -rw------- 1 cyrus mail 8669 Feb 24 11:28 cyrus.squat
this is Just a very small user and a example
On 02/24/2010 11:21 AM, Agnello George wrote:
-rw------- 1 cyrus mail 4805500 Feb 24 16:23 4799. -rw------- 1 cyrus mail 22920 Feb 24 16:23 cyrus.cache -rw------- 1 cyrus mail 204 Dec 10 16:27 cyrus.header -rw------- 1 cyrus mail 896 Feb 24 16:23 cyrus.index -rw------- 1 cyrus mail 8669 Feb 24 11:28 cyrus.squat
this is Just a very small user and a example
About 90% of your problem is already solved here, you are using cyrus which has built in mail level replication. All you need to do is setup a lvm volume away from this main store and run your mail replica over to it. then just backup using whatever tools you want.
Free win you get is online failover, backup in whatever manner you want!
- KB
On Wed, Feb 24, 2010 at 6:19 PM, Karanbir Singh mail-lists@karan.orgwrote:
On 02/24/2010 11:21 AM, Agnello George wrote:
-rw------- 1 cyrus mail 4805500 Feb 24 16:23 4799. -rw------- 1 cyrus mail 22920 Feb 24 16:23 cyrus.cache -rw------- 1 cyrus mail 204 Dec 10 16:27 cyrus.header -rw------- 1 cyrus mail 896 Feb 24 16:23 cyrus.index -rw------- 1 cyrus mail 8669 Feb 24 11:28 cyrus.squat
this is Just a very small user and a example
About 90% of your problem is already solved here, you are using cyrus which has built in mail level replication. All you need to do is setup a lvm volume away from this main store and run your mail replica over to it. then just backup using whatever tools you want.
Free win you get is online failover, backup in whatever manner you want!
yes just spoke to my senior and confrimed that this was alreday tried out a delayed replication is possible . but the current suitation is we need to take backup on the same server on a different partition /backup :(
On 02/24/2010 01:07 PM, Agnello George wrote:
yes just spoke to my senior and confrimed that this was alreday tried out a delayed replication is possible . but the current suitation is we need to take backup on the same server on a different partition /backup :(
you can replicate to a local mail store as well. just make sure you put it on a block device that is suiteable and fits in with the rest of your backup strategy. And if you put it in an isolated enough place on the block dev, it wont contest with the users access.
- KB
On Wed, Feb 24, 2010 at 10:38:32AM +0100, David Hrbáč wrote:
Dne 24.2.2010 10:00, Agnello George napsal(a):
We have an issue with one of our clients , they have a mail server with the /var/spool/imap partition as 150 GB . They need to take differential backup on to /backup partition ( a different HDD of total 250 GB space ) . We have tried dar , rsync, rdiff and impasync . But its is not sufficing the need as to take a lot of time and consumes a lot of I/O .
Is there any back up solution that you can think of , that can work in this situation - open source or proprietary
Is seems to me, that you are using mbox format. So, differential backup is hard to achieve. Migrate to maildir, every mail is a file, easy to backup differentially.
rsync and rdiff should handle mbox format okay though. Though I agree Maildir is generally nicer for differential backups.
Agnello, how long is "a lot of time"? A backup is always going to have to walk the entire tree and checksum (or at least stat) every file, so there's a minimum cost you're always going to have. How long does a 'find /var/spool/imap -ls' take, for instance?
You might want to try brackup (http://code.google.com/p/brackup/). For very large trees of relatively small files it seems to significantly out-perform rsync-based backups. I've got brackup packages in my repository (see http://www.openfusion.net/linux/openfusion_rpm_repository).
Cheers, Gavin
On Wed, Feb 24, 2010 at 4:13 PM, Gavin Carr gavin@openfusion.com.au wrote:
On Wed, Feb 24, 2010 at 10:38:32AM +0100, David Hrbáč wrote:
Dne 24.2.2010 10:00, Agnello George napsal(a):
We have an issue with one of our clients , they have a mail server
with
the /var/spool/imap partition as 150 GB . They need to take
differential
backup on to /backup partition ( a different HDD of total 250 GB space
) .
We have tried dar , rsync, rdiff and impasync . But its is not
sufficing the
need as to take a lot of time and consumes a lot of I/O .
Is there any back up solution that you can think of , that can work in
this
situation - open source or proprietary
Is seems to me, that you are using mbox format. So, differential backup is hard to achieve. Migrate to maildir, every mail is a file, easy to backup differentially.
rsync and rdiff should handle mbox format okay though. Though I agree Maildir is generally nicer for differential backups.
Agnello, how long is "a lot of time"? A backup is always going to have to walk the entire tree and checksum (or at least stat) every file, so there's a minimum cost you're always going to have. How long does a 'find /var/spool/imap -ls' take, for instance?
You might want to try brackup (http://code.google.com/p/brackup/). For very large trees of relatively small files it seems to significantly out-perform rsync-based backups. I've got brackup packages in my repository (see http://www.openfusion.net/linux/openfusion_rpm_repository).
Cheers, Gavin
is it possible with " brackup " http://code.google.com/p/brackup/ to back it up to a different server on the same lan instead of /backup . Is there any documentation on the same .
2010/2/24 Agnello George agnello.dsouza@gmail.com:
On Wed, Feb 24, 2010 at 4:13 PM, Gavin Carr gavin@openfusion.com.au wrote:
On Wed, Feb 24, 2010 at 10:38:32AM +0100, David Hrbáč wrote:
Dne 24.2.2010 10:00, Agnello George napsal(a):
We have an issue with one of our clients , they have a mail server with the /var/spool/imap partition as 150 GB . They need to take differential backup on to /backup partition ( a different HDD of total 250 GB space ) . We have tried dar , rsync, rdiff and impasync . But its is not sufficing the need as to take a lot of time and consumes a lot of I/O .
Is there any back up solution that you can think of , that can work in this situation - open source or proprietary
Is seems to me, that you are using mbox format. So, differential backup is hard to achieve. Migrate to maildir, every mail is a file, easy to backup differentially.
rsync and rdiff should handle mbox format okay though. Though I agree Maildir is generally nicer for differential backups.
Agnello, how long is "a lot of time"? A backup is always going to have to walk the entire tree and checksum (or at least stat) every file, so there's a minimum cost you're always going to have. How long does a 'find /var/spool/imap -ls' take, for instance?
You might want to try brackup (http://code.google.com/p/brackup/). For very large trees of relatively small files it seems to significantly out-perform rsync-based backups. I've got brackup packages in my repository (see http://www.openfusion.net/linux/openfusion_rpm_repository).
Cheers, Gavin
is it possible with " brackup " to back it up to a different server on the same lan instead of /backup . Is there any documentation on the same .
rsync or rdiff-backup works on local disk or remote disk.(and other backup methods too!)
-- Eero
On Wed, Feb 24, 2010 at 4:57 PM, Eero Volotinen eero.volotinen@iki.fiwrote:
2010/2/24 Agnello George agnello.dsouza@gmail.com:
On Wed, Feb 24, 2010 at 4:13 PM, Gavin Carr gavin@openfusion.com.au
wrote:
On Wed, Feb 24, 2010 at 10:38:32AM +0100, David Hrbáč wrote:
Dne 24.2.2010 10:00, Agnello George napsal(a):
We have an issue with one of our clients , they have a mail server with the /var/spool/imap partition as 150 GB . They need to take differential backup on to /backup partition ( a different HDD of total 250 GB space ) . We have tried dar , rsync, rdiff and impasync . But its is not sufficing the need as to take a lot of time and consumes a lot of I/O .
Is there any back up solution that you can think of , that can work
in
this situation - open source or proprietary
Is seems to me, that you are using mbox format. So, differential
backup
is hard to achieve. Migrate to maildir, every mail is a file, easy to backup differentially.
rsync and rdiff should handle mbox format okay though. Though I agree Maildir is generally nicer for differential backups.
Agnello, how long is "a lot of time"? A backup is always going to have
to
walk the entire tree and checksum (or at least stat) every file, so there's a minimum cost you're always going to have. How long does a 'find /var/spool/imap -ls' take, for instance?
You might want to try brackup (http://code.google.com/p/brackup/). For very large trees of relatively small files it seems to significantly out-perform rsync-based backups. I've got brackup packages in my repository (see http://www.openfusion.net/linux/openfusion_rpm_repository).
Cheers, Gavin
is it possible with " brackup " to back it up to a different server on
the
same lan instead of /backup . Is there any documentation on the same .
rsync or rdiff-backup works on local disk or remote disk.(and other backup methods too!)
Does http://code.google.com/p/brackup/ also work in on remote machines .
2010/2/24 Agnello George agnello.dsouza@gmail.com:
On Wed, Feb 24, 2010 at 4:57 PM, Eero Volotinen eero.volotinen@iki.fi wrote:
2010/2/24 Agnello George agnello.dsouza@gmail.com:
On Wed, Feb 24, 2010 at 4:13 PM, Gavin Carr gavin@openfusion.com.au wrote:
On Wed, Feb 24, 2010 at 10:38:32AM +0100, David Hrbáč wrote:
Dne 24.2.2010 10:00, Agnello George napsal(a):
We have an issue with one of our clients , they have a mail server with the /var/spool/imap partition as 150 GB . They need to take differential backup on to /backup partition ( a different HDD of total 250 GB space ) . We have tried dar , rsync, rdiff and impasync . But its is not sufficing the need as to take a lot of time and consumes a lot of I/O .
Is there any back up solution that you can think of , that can work in this situation - open source or proprietary
Is seems to me, that you are using mbox format. So, differential backup is hard to achieve. Migrate to maildir, every mail is a file, easy to backup differentially.
rsync and rdiff should handle mbox format okay though. Though I agree Maildir is generally nicer for differential backups.
Agnello, how long is "a lot of time"? A backup is always going to have to walk the entire tree and checksum (or at least stat) every file, so there's a minimum cost you're always going to have. How long does a 'find /var/spool/imap -ls' take, for instance?
You might want to try brackup (http://code.google.com/p/brackup/). For very large trees of relatively small files it seems to significantly out-perform rsync-based backups. I've got brackup packages in my repository (see http://www.openfusion.net/linux/openfusion_rpm_repository).
Cheers, Gavin
is it possible with " brackup " to back it up to a different server on the same lan instead of /backup . Is there any documentation on the same .
rsync or rdiff-backup works on local disk or remote disk.(and other backup methods too!)
Summary from webpage: "Flexible backup tool. Slices, dices, encrypts, and sprays across the net, notably to Amazon's S3. "
-- Eero
On Wed, Feb 24, 2010 at 05:06:08PM +0530, Agnello George wrote:
On Wed, Feb 24, 2010 at 4:57 PM, Eero Volotinen eero.volotinen@iki.fiwrote:
2010/2/24 Agnello George agnello.dsouza@gmail.com:
On Wed, Feb 24, 2010 at 4:13 PM, Gavin Carr gavin@openfusion.com.au
wrote:
You might want to try brackup (http://code.google.com/p/brackup/). For very large trees of relatively small files it seems to significantly out-perform rsync-based backups. I've got brackup packages in my repository (see http://www.openfusion.net/linux/openfusion_rpm_repository).
Cheers, Gavin
is it possible with " brackup " to back it up to a different server on the same lan instead of /backup . Is there any documentation on the same .
rsync or rdiff-backup works on local disk or remote disk.(and other backup methods too!)
Does http://code.google.com/p/brackup/ also work in on remote machines .
Brackup will backup to local disk, or remotely to ftp, sftp, Amazon S3, or Rackspace CloudFiles targets/servers. So yes, on a lan you can backup over ftp or sftp just fine.
Re docs, install brackup, 'man Brackup::Manual::Overview'. I've also written a few blog posts on it: http://www.openfusion.net/tags/brackup.
Cheers, Gavin
On Wed, Feb 24, 2010 at 7:05 PM, Gavin Carr gavin@openfusion.com.au wrote:
On Wed, Feb 24, 2010 at 05:06:08PM +0530, Agnello George wrote:
On Wed, Feb 24, 2010 at 4:57 PM, Eero Volotinen <eero.volotinen@iki.fi wrote:
2010/2/24 Agnello George agnello.dsouza@gmail.com:
On Wed, Feb 24, 2010 at 4:13 PM, Gavin Carr <gavin@openfusion.com.au
wrote:
You might want to try brackup (http://code.google.com/p/brackup/).
For
very large trees of relatively small files it seems to significantly out-perform rsync-based backups. I've got brackup packages in my repository (see http://www.openfusion.net/linux/openfusion_rpm_repository).
Cheers, Gavin
is it possible with " brackup " to back it up to a different server
on
the same lan instead of /backup . Is there any documentation on
the
same .
rsync or rdiff-backup works on local disk or remote disk.(and other backup methods too!)
Does http://code.google.com/p/brackup/ also work in on remote machines
.
Brackup will backup to local disk, or remotely to ftp, sftp, Amazon S3, or Rackspace CloudFiles targets/servers. So yes, on a lan you can backup over ftp or sftp just fine.
Re docs, install brackup, 'man Brackup::Manual::Overview'. I've also written a few blog posts on it: http://www.openfusion.net/tags/brackup.
Cheers, Gavin
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
it will take me some time to try this .. will get back on its output !! .. thanks
On Wed, Feb 24, 2010 at 7:05 PM, Gavin Carr gavin@openfusion.com.au wrote:
On Wed, Feb 24, 2010 at 05:06:08PM +0530, Agnello George wrote:
On Wed, Feb 24, 2010 at 4:57 PM, Eero Volotinen <eero.volotinen@iki.fi wrote:
2010/2/24 Agnello George agnello.dsouza@gmail.com:
On Wed, Feb 24, 2010 at 4:13 PM, Gavin Carr <gavin@openfusion.com.au
wrote:
You might want to try brackup (http://code.google.com/p/brackup/).
For
very large trees of relatively small files it seems to significantly out-perform rsync-based backups. I've got brackup packages in my repository (see http://www.openfusion.net/linux/openfusion_rpm_repository).
Cheers, Gavin
is it possible with " brackup " to back it up to a different server
on
the same lan instead of /backup . Is there any documentation on
the
same .
rsync or rdiff-backup works on local disk or remote disk.(and other backup methods too!)
Does http://code.google.com/p/brackup/ also work in on remote machines
.
Brackup will backup to local disk, or remotely to ftp, sftp, Amazon S3, or Rackspace CloudFiles targets/servers. So yes, on a lan you can backup over ftp or sftp just fine.
Re docs, install brackup, 'man Brackup::Manual::Overview'. I've also written a few blog posts on it: http://www.openfusion.net/tags/brackup.
Cheers, Gavin
I am trying to install the brackup app on my system, the documentations
seems very helpful ( http://www.openfusion.net/net/fun_with_brackup) But i have a few queries with the config file : [TARGET:backups] type = Filesystem path = /backup
[SOURCE:imapsource] path = /var/spool/imap chunk_size = 5m # what does this mean ???? gpg_recipient = 5E1B3EC5 # what does this mean ????
[SOURCE:bradhome] chunk_size = 64MB path = /raid/bradfitz/ ignore = ^.thumbnails/ ignore = ^.kde/share/thumbnails/ ignore = ^.ee/minis/ ignore = ^build/ ignore = ^(gqview|nautilus)/thumbnails/
and suppose i want to backup it up to another server with scp / ssh how is this attatined . secondly in whant format is the backup maintained .
Have you read Brackup::Manual::Overview? Your questions are all answered in the man pages there or linked from there.
On Thu, Feb 25, 2010 at 05:21:47PM +0530, Agnello George wrote:
On Wed, Feb 24, 2010 at 7:05 PM, Gavin Carr gavin@openfusion.com.au wrote:
Brackup will backup to local disk, or remotely to ftp, sftp, Amazon S3, or Rackspace CloudFiles targets/servers. So yes, on a lan you can backup over ftp or sftp just fine.
Re docs, install brackup, 'man Brackup::Manual::Overview'. I've also written a few blog posts on it: http://www.openfusion.net/tags/brackup.
I am trying to install the brackup app on my system, the documentations seems very helpful ( http://www.openfusion.net/net/fun_with_brackup) But i have a few queries with the config file : [TARGET:backups] type = Filesystem path = /backup
[SOURCE:imapsource] path = /var/spool/imap chunk_size = 5m # what does this mean ???? gpg_recipient = 5E1B3EC5 # what does this mean ????
man Brackup::Manual::Overview; man Brackup::Root
[SOURCE:bradhome] chunk_size = 64MB path = /raid/bradfitz/ ignore = ^.thumbnails/ ignore = ^.kde/share/thumbnails/ ignore = ^.ee/minis/ ignore = ^build/ ignore = ^(gqview|nautilus)/thumbnails/
and suppose i want to backup it up to another server with scp / ssh how is this attatined .
man Brackup::Target::Sftp
secondly in whant format is the backup maintained .
Backups are trees of file chunks, and a metadata file to put the chunks back together as files. So you get de-duplication for free between files and across backups.
Cheers, Gavin
From: Agnello George agnello.dsouza@gmail.com
is it possible with " brackup " to back it up to a different server on the same lan instead of /backup . Is there any documentation on the same .
It apparently support: Brackup::Target::Amazon backup to Amazon's S3 service Brackup::Target::CloudFiles backup to Rackspace's CloudFiles Service Brackup::Target::Filebased Brackup::Target::Filesystem backup to a locally mounted filesystem Brackup::Target::Ftp backup to an FTP server Brackup::Target::Sftp backup to an SSH/SFTP server So, you could use ftp or sftp...
JD
Agnello George wrote:
Hi
We have an issue with one of our clients , they have a mail server with the /var/spool/imap partition as 150 GB . They need to take differential backup on to /backup partition ( a different HDD of total 250 GB space ) . We have tried dar , rsync, rdiff and impasync . But its is not sufficing the need as to take a lot of time and consumes a lot of I/O .
Is there any back up solution that you can think of , that can work in this situation - open source or proprietary
If you are just concerned about a single disk failure you could set up RAID1 on the disks (with some downtime to rebuild...) to keep the copy in realtime with little loss of speed.
Rsync should work as well as anything for snapshots but you might need to update to a 3.x version to speed up handling large numbers of files. The 2.x version included in Centos will read the entire directory tree into memory before starting the comparisons and copies. The rpmforge repo has a packaged 3.0.7 version but I haven't tried it.
Agnello George wrote:
Hi
We have an issue with one of our clients , they have a mail server with the /var/spool/imap partition as 150 GB . They need to take differential backup on to /backup partition ( a different HDD of total 250 GB space ) .
You've stated things in terms of solutions. You may possibly get better answers if you state your goal. There is some capability you are trying to achieve. Tell us what that is, and you may make more progress.
IOW, what is the purpose of the backup? As one mentioned, RAID may handle your needs.
Mike
2010/2/24 Mike McCarty Mike.McCarty@sbcglobal.net:
Agnello George wrote:
Hi
We have an issue with one of our clients , they have a mail server with the /var/spool/imap partition as 150 GB . They need to take differential backup on to /backup partition ( a different HDD of total 250 GB space ) .
You've stated things in terms of solutions. You may possibly get better answers if you state your goal. There is some capability you are trying to achieve. Tell us what that is, and you may make more progress.
IOW, what is the purpose of the backup? As one mentioned, RAID may handle your needs.
Err.. raid is NOT backup solution.
-- Eero
On 2/24/2010 1:31 PM, Eero Volotinen wrote:
We have an issue with one of our clients , they have a mail server with the /var/spool/imap partition as 150 GB . They need to take differential backup on to /backup partition ( a different HDD of total 250 GB space ) .
You've stated things in terms of solutions. You may possibly get better answers if you state your goal. There is some capability you are trying to achieve. Tell us what that is, and you may make more progress.
IOW, what is the purpose of the backup? As one mentioned, RAID may handle your needs.
Err.. raid is NOT backup solution.
Neither is a snapshot in another location on the same machine. But both will cover the most likely thing to fail, with raid doing it transparently, the snapshot losing data from the time the last snapshot copy happened. Usually what you want is raid _and_ a history of backups kept elsewhere.
On 02/24/2010 07:44 PM, Les Mikesell wrote:
Err.. raid is NOT backup solution.
Neither is a snapshot in another location on the same machine.
Thats not true, raid is an online setup - different location could be point in time, and on blockdev;s that dont share user access load. Which in turn makes it easier to do intensive complete backups to offsite without impacting user level of service the machine can deliver, amongst other things[1].
Dont compare apples to banana's and call them oranges.
- KB
[1] Changeset and data/system model over time relation mapping for an adaptive system sizing feedback loop! ( how'se that for buzzword injection! )
Karanbir Singh wrote:
On 02/24/2010 07:44 PM, Les Mikesell wrote:
Err.. raid is NOT backup solution.
Neither is a snapshot in another location on the same machine.
Thats not true, raid is an online setup - different location could be point in time, and on blockdev;s that dont share user access load. Which in turn makes it easier to do intensive complete backups to offsite without impacting user level of service the machine can deliver, amongst other things[1].
Dont compare apples to banana's and call them oranges.
Yes and no... There's an overlapping set of possibilities that they do and don't back up. They both cover single disk failures. They don't cover big operator errors (rm -rf /), building/site disasters, some types of controller/electrical issues, etc. The snapshots give you a short history that can help with small user/operator errors at the expense of being out of date when the live disk fails. So have several types of fruit to stay healthy.
On Thu, 2010-02-25 at 12:33 +0000, Karanbir Singh wrote:
[1] Changeset and data/system model over time relation mapping for an adaptive system sizing feedback loop! ( how'se that for buzzword injection! )
--- Well if you run vacum on a Postgres DB then all that goes to the crapper... So we resort to real time backups or replication. Of which with replication on a postgres db with the 2GB Blobs being inserted there is going to be a problem... even if you have enough shared memory configured. The thing is regardless you are never in RT Replication.
John
Karanbir Singh wrote:
On 02/24/2010 07:44 PM, Les Mikesell wrote:
Err.. raid is NOT backup solution.
Neither is a snapshot in another location on the same machine.
Thats not true, raid is an online setup - different location could be point in time, and on blockdev;s that dont share user access load. Which
I think that, without causing any more dispute, I can point out that "backup" covers a wide range of solutions to a less broad but still not uniquely one set of needs. No one of the means to backup is a full solution to all the needs which "backup" satisfies.
Even when one is using the term "backup" narrowly in the sense of "protection from disaster", there are still different kinds of backup. For example, there is the "full disaster recovery" or "bare metal" backup, which is intended to work with another piece of identical hardware, starting with blank fixed storage, and ending up with a working system which looks identical to the original at the epoch at which the backup was made. This is significantly different from one intended merely to restore the user altered or created data on a machine which has been newly installed with a compatible version of the OS, for example.
That's why one needs to know the intended use of the backup set before making any recommendations on procedure and content of the backup set.
Mike
Eero Volotinen wrote:
2010/2/24 Mike McCarty Mike.McCarty@sbcglobal.net:
Agnello George wrote:
Hi
We have an issue with one of our clients , they have a mail server with the /var/spool/imap partition as 150 GB . They need to take differential backup on to /backup partition ( a different HDD of total 250 GB space ) .
You've stated things in terms of solutions. You may possibly get better answers if you state your goal. There is some capability you are trying to achieve. Tell us what that is, and you may make more progress.
IOW, what is the purpose of the backup? As one mentioned, RAID may handle your needs.
Err.. raid is NOT backup solution.
Of course not. RAID is a means to achieve availability, which may be his goal. Karanbir already stated a means to do what he seemed to want, but it seemed not to satisfy his needs.
Unless the query is placed in terms of requirements and goals, instead of solutions, it'll be difficult to achieve satisfactory results.
The purpose of backup is some degree of disaster recovery, and perhaps also migration. If that's truly his goal, then ISTM that Karanbir suggested a viable solution to achieving avialability while also performing backup, by doing on-the-fly duplication of the data onto another file system which can then be backed up at liesure.
Doing so in a manner which ensures a true snapshot may be more difficult to achieve, while still ensuring availability. I normally do my backups in single user mode with all file systems mounted read only, except the one to receive the backup. That of course precludes availability during the backup procedure.
That's why I would like to see what he actually wants to achieve, instead of how he has chosen to go about it.
Mike
On Thu, Feb 25, 2010 at 1:19 AM, Mike McCarty Mike.McCarty@sbcglobal.netwrote:
Eero Volotinen wrote:
2010/2/24 Mike McCarty Mike.McCarty@sbcglobal.net:
Agnello George wrote:
Hi
We have an issue with one of our clients , they have a mail server
with
the /var/spool/imap partition as 150 GB . They need to take
differential
backup on to /backup partition ( a different HDD of total 250 GB space
) .
You've stated things in terms of solutions. You may possibly get better answers if you state your goal. There is some capability you are trying to achieve. Tell us what that is, and you may make more progress.
IOW, what is the purpose of the backup? As one mentioned, RAID may handle your needs.
Err.. raid is NOT backup solution.
Of course not. RAID is a means to achieve availability, which may be his goal. Karanbir already stated a means to do what he seemed to want, but it seemed not to satisfy his needs.
Unless the query is placed in terms of requirements and goals, instead of solutions, it'll be difficult to achieve satisfactory results.
The purpose of backup is some degree of disaster recovery, and perhaps also migration. If that's truly his goal, then ISTM that Karanbir suggested a viable solution to achieving avialability while also performing backup, by doing on-the-fly duplication of the data onto another file system which can then be backed up at liesure.
Doing so in a manner which ensures a true snapshot may be more difficult to achieve, while still ensuring availability. I normally do my backups in single user mode with all file systems mounted read only, except the one to receive the backup. That of course precludes availability during the backup procedure.
That's why I would like to see what he actually wants to achieve, instead of how he has chosen to go about it.
Mike
The requirement fro backup is not primarily for HDD failure , but human error failure . In case one of our user ( eg: the COO with huge mailbox size has delete all his certain very important mails, and he want to recover them , the contacts us as we are supposed to maintain his mail backup for a week, and we should restore his backup immediately ) this the main requirement for the backup and that too on the same server different partition .
Agnello George wrote:
The requirement fro backup is not primarily for HDD failure , but human error failure . In case one of our user ( eg: the COO with huge mailbox size has delete all his certain very important mails, and he want to recover them , the contacts us as we are supposed to maintain his mail backup for a week, and we should restore his backup immediately ) this the main requirement for the backup and that too on the same server different partition .
Have you considered using a snapshot approach? By that, I mean one which uses hard links to create the "backup", and as files get added/ modified, the data are copied, and links are created. Usually, one has a snapshot directory with something like a daily snapshot, and 24 hourly ones, something like that.
Mike
Years ago, I set up a backup tool that wrapped rsync. It has faithfully and reliably backed up a dozen hosts and too many TB of data to mention, offsite, automatically saving as many backup points as disk space allows. You're certainly welcome to try it!
http://www.effortlessis.com/thisisnotbackupbuddy/
It works on an "ascending powers" basis, EG:
1 day ago, 2 days ago, 4 days ago, 8 days ago, 16 days ago... .... until out of disk space.
=)
-Ben
On Thursday 25 February 2010 10:22:13 am Mike McCarty wrote:
Agnello George wrote:
The requirement fro backup is not primarily for HDD failure , but human error failure . In case one of our user ( eg: the COO with huge mailbox size has delete all his certain very important mails, and he want to recover them , the contacts us as we are supposed to maintain his mail backup for a week, and we should restore his backup immediately ) this the main requirement for the backup and that too on the same server different partition .
Have you considered using a snapshot approach? By that, I mean one which uses hard links to create the "backup", and as files get added/ modified, the data are copied, and links are created. Usually, one has a snapshot directory with something like a daily snapshot, and 24 hourly ones, something like that.
Mike
Greetings,
On Fri, Feb 26, 2010 at 5:50 AM, Benjamin Smith lists@benjamindsmith.comwrote:
Years ago, I set up a backup tool that wrapped rsync. It has faithfully and reliably backed up a dozen hosts and too many TB of data to mention, offsite, automatically saving as many backup points as disk space allows. You're certainly welcome to try it!
http://www.effortlessis.com/thisisnotbackupbuddy/
It works on an "ascending powers" basis, EG:
1 day ago, 2 days ago, 4 days ago, 8 days ago, 16 days ago... .... until out of disk space.
=)
<grin|smile|whatever> it is a criminal offence (in free software world) to hide this gem from the world for all this long. ;)
Regards,
Rajagopal
-
Agnello George -
Benjamin Smith -
David Hrbáč -
Eero Volotinen -
Gavin Carr -
John Doe -
JohnS -
Karanbir Singh -
Les Mikesell -
Mike McCarty -
Rajagopal Swaminathan