I realize this shouldn't happen, the file is a tgz and isn't being modified while being transmitted. This has happened maybe three times this year and unfortunately I've just had to deal with it rather than invest the time to do the research.
Harriscomputer
Leroy Tennison Network Information/Cyber Security Specialist E: leroy@datavoiceint.com
[cid:Data-Voice-International-LOGO_aa3d1c6e-5cfb-451f-ba2c-af8059e69609.PNG]
2220 Bush Dr McKinney, Texas 75070 www.datavoiceint.comhttp://www..com
This message has been sent on behalf of a company that is part of the Harris Operating Group of Constellation Software Inc.
If you prefer not to be contacted by Harris Operating Group please notify ushttp://subscribe.harriscomputer.com/.
This message is intended exclusively for the individual or entity to which it is addressed. This communication may contain information that is proprietary, privileged or confidential or otherwise legally exempt from disclosure. If you are not the named addressee, you are not authorized to read, print, retain, copy or disseminate this message or any part of it. If you have received this message in error, please notify the sender immediately by e-mail and delete all copies of the message.
________________________________ From: Chris Schanzle christopher.schanzle@nist.gov Sent: Wednesday, July 1, 2020 10:03 AM To: CentOS mailing list centos@centos.org; Leroy Tennison leroy@datavoiceint.com Subject: [EXTERNAL] Re: [CentOS] [OT] Bacula offsite replication
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Unless the file is being modified during rsync, corruption should not happen with good hardware. Consider testing your RAM. Have you noticed any other weird problems with that remote server, like programs crashing / daemons needing restarting?
On 7/1/20 10:37 AM, Leroy Tennison wrote:
What I did was used cksum to create a checksum of the source file putting it in a separate file, transmitted that via rsync as well and compared that to a cksum computed on the remote end. There are far more accurate alternatives to cksum but I felt cksum was good enough for a basic check. Like most things in the UNIX world, there are probably other ways to do this as well.
Interestingly enough, after I sent my previous response I discovered that I had yet another instance of the problem.
From: CentOS centos-bounces@centos.org on behalf of Alessandro Baggi alessandro.baggi@gmail.com Sent: Wednesday, July 1, 2020 9:26 AM To: centos@centos.org centos@centos.org Subject: [EXTERNAL] Re: [CentOS] [OT] Bacula offsite replication
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Hi Leroy,
How I can confirm that during rsync transfer corruption are not encountered?
Thank you in advance.
Harriscomputer
Leroy Tennison Network Information/Cyber Security Specialist E: leroy@datavoiceint.com
[cid:Data-Voice-International-LOGO_aa3d1c6e-5cfb-451f-ba2c-af8059e69609.PNG]
2220 Bush Dr McKinney, Texas 75070 https://linkprotect.cudasvc.com/url?a=http%3a%2f%2fwww.datavoiceint.com%2f&a...
This message has been sent on behalf of a company that is part of the Harris Operating Group of Constellation Software Inc.
If you prefer not to be contacted by Harris Operating Group please notify ushttps://linkprotect.cudasvc.com/url?a=http%3a%2f%2fsubscribe.harriscomputer.com%2f&c=E,1,ESghWsZAKB3kZUcHUH6MS2ivZGjhaE3linFZeLtQ96hbUtv37Esy1OON4XdoFr1DjlanYK_dt8Kie6diqCOVrkPalJ6KDLXEocN-5BFabl2AiHWvFfo3VvM,&typo=1.
This message is intended exclusively for the individual or entity to which it is addressed. This communication may contain information that is proprietary, privileged or confidential or otherwise legally exempt from disclosure. If you are not the named addressee, you are not authorized to read, print, retain, copy or disseminate this message or any part of it. If you have received this message in error, please notify the sender immediately by e-mail and delete all copies of the message.
Il 01/07/20 16:04, Leroy Tennison ha scritto:
I've used rsync (but probably not for the size you're referring to), it works and has enough features to meet most needs. I have had a single situation where corruption occurred during transfer (a few times, have no idea why), might want to independently confirm the integrity of the transfer.
From: CentOS centos-bounces@centos.org on behalf of Alessandro Baggi alessandro.baggi@gmail.com Sent: Wednesday, July 1, 2020 5:26 AM To: centos@centos.org centos@centos.org Subject: [EXTERNAL] [CentOS] [OT] Bacula offsite replication
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Hi everyone,
I have updated my backup server to CentOS 8.2. It runs bacula performing backup on disks. I would like to replicate backups on another offsite machine.
I read about the ability to configure a new storage daemon in the offsite location and create a Migration/Copy Jobs. If I'm not wrong, it replicates only volumes but not replicate the catalog. I will try this.
Another way to replicate the volumes on another server is using rsync.
What is your suggestion about this topic?
Thank you in advance.
Alessandro.
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Harriscomputer
Leroy Tennison Network Information/Cyber Security Specialist E: leroy@datavoiceint.com
[cid:Data-Voice-International-LOGO_aa3d1c6e-5cfb-451f-ba2c-af8059e69609.PNG]
2220 Bush Dr McKinney, Texas 75070 https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.datavoiceint.com&...
This message has been sent on behalf of a company that is part of the Harris Operating Group of Constellation Software Inc.
If you prefer not to be contacted by Harris Operating Group please notify ushttps://linkprotect.cudasvc.com/url?a=http%3a%2f%2fsubscribe.harriscomputer.com%2f&c=E,1,4UMyprULKejN76Lk4p9zM-laz6VtwtLbbjIU8e02p6oWiLS-njfZsTFuXkb0910-WrqQ8x6J4YCieJO5HeN2WGf7pqwFdtVkKJi-m_QGliIsyR6XTAVohBrv&typo=1.
This message is intended exclusively for the individual or entity to which it is addressed. This communication may contain information that is proprietary, privileged or confidential or otherwise legally exempt from disclosure. If you are not the named addressee, you are not authorized to read, print, retain, copy or disseminate this message or any part of it. If you have received this message in error, please notify the sender immediately by e-mail and delete all copies of the message.
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos _______________________________________________ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Am 01.07.20 um 17:13 schrieb Leroy Tennison:
I realize this shouldn't happen, the file is a tgz and isn't being modified while being transmitted. This has happened maybe three times this year and unfortunately I've just had to deal with it rather than invest the time to do the research.
Maybe a "RunAfterJob" configuration would help to serialize it and prevent this race condition?
-- Leon
Il 01/07/20 17:13, Leroy Tennison ha scritto:
I realize this shouldn't happen, the file is a tgz and isn't being modified while being transmitted. This has happened maybe three times this year and unfortunately I've just had to deal with it rather than invest the time to do the research.
Harriscomputer
Leroy Tennison Network Information/Cyber Security Sp
Hi Leroy,
I think that in my case I could not use a tgz archive. I'm speaking about full backups that reach 600/700GiB, compressing them and then rsync them could take so much time that it will be useless.
I setup drbd to replicate a ~50TB backuppc hive to the DR copy, an identical box in a different DC on the same campus, with approximately gigE speeds, and ran this for a year or two. It worked well enough but required babysitting from time to time. Both nodes were mdraid lvm logical volumes formatted as a single huge xfs on centos. I never automated the failover as it never failed, and as a dev/test backup, 8 hour response seemed adequate
On Thu, Jul 2, 2020, 1:22 AM Alessandro Baggi alessandro.baggi@gmail.com wrote:
Il 01/07/20 17:13, Leroy Tennison ha scritto:
I realize this shouldn't happen, the file is a tgz and isn't being
modified while being transmitted. This has happened maybe three times this year and unfortunately I've just had to deal with it rather than invest the time to do the research.
Harriscomputer
Leroy Tennison Network Information/Cyber Security Sp
Hi Leroy,
I think that in my case I could not use a tgz archive. I'm speaking about full backups that reach 600/700GiB, compressing them and then rsync them could take so much time that it will be useless.
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Hi John,
thank you for your answer, I already take in consideration DRBD but I need some test before start.
Reading you seems that this solution is not anymore available. What do you use for this?
Thank you in advance.
Il 02/07/20 10:43, John Pierce ha scritto:
I setup drbd to replicate a ~50TB backuppc hive to the DR copy, an identical box in a different DC on the same campus, with approximately gigE speeds, and ran this for a year or two. It worked well enough but required babysitting from time to time. Both nodes were mdraid lvm logical volumes formatted as a single huge xfs on centos. I never automated the failover as it never failed, and as a dev/test backup, 8 hour response seemed adequate
On Thu, Jul 2, 2020, 1:22 AM Alessandro Baggi alessandro.baggi@gmail.com wrote:
Il 01/07/20 17:13, Leroy Tennison ha scritto:
I realize this shouldn't happen, the file is a tgz and isn't being
modified while being transmitted. This has happened maybe three times this year and unfortunately I've just had to deal with it rather than invest the time to do the research.
Harriscomputer
Leroy Tennison Network Information/Cyber Security Sp
Hi Leroy,
I think that in my case I could not use a tgz archive. I'm speaking about full backups that reach 600/700GiB, compressing them and then rsync them could take so much time that it will be useless.
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
On Thu, Jul 2, 2020, 1:51 AM Alessandro Baggi alessandro.baggi@gmail.com wrote:
Hi John,
thank you for your answer, I already take in consideration DRBD but I need some test before start.
Reading you seems that this solution is not anymore available. What do you use for this?
That system was surplused 2 years ago when my group was shut down and I retired.
On 7/2/20 3:22 AM, Alessandro Baggi wrote:
Il 01/07/20 17:13, Leroy Tennison ha scritto:
I realize this shouldn't happen, the file is a tgz and isn't being modified while being transmitted. This has happened maybe three times this year and unfortunately I've just had to deal with it rather than invest the time to do the research.
Harriscomputer
Leroy Tennison Network Information/Cyber Security Sp
Hi Leroy,
I think that in my case I could not use a tgz archive. I'm speaking about full backups that reach 600/700GiB, compressing them and then rsync them could take so much time that it will be useless.
unless you use tape (of that high capacity), it is advantageous to restrict volume size to, say, 50GB. Then when you restore, search for specific files will be faster. And it will help your backup volumes transfers as well.
Valeri
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Il 02/07/20 15:02, Valeri Galtsev ha scritto:
On 7/2/20 3:22 AM, Alessandro Baggi wrote:
Il 01/07/20 17:13, Leroy Tennison ha scritto:
I realize this shouldn't happen, the file is a tgz and isn't being modified while being transmitted. This has happened maybe three times this year and unfortunately I've just had to deal with it rather than invest the time to do the research.
Harriscomputer
Leroy Tennison Network Information/Cyber Security Sp
Hi Leroy,
I think that in my case I could not use a tgz archive. I'm speaking about full backups that reach 600/700GiB, compressing them and then rsync them could take so much time that it will be useless.
unless you use tape (of that high capacity), it is advantageous to restrict volume size to, say, 50GB. Then when you restore, search for specific files will be faster. And it will help your backup volumes transfers as well.
Valeri
Hi Valeri,
thank you for your suggestion.
Is bacula the right backup system when I need to replicate data offsite? There are other backup solution that simplify this process?
Thank you in advance
On 2020-07-02 08:28, Alessandro Baggi wrote:
Il 02/07/20 15:02, Valeri Galtsev ha scritto:
On 7/2/20 3:22 AM, Alessandro Baggi wrote:
Il 01/07/20 17:13, Leroy Tennison ha scritto:
I realize this shouldn't happen, the file is a tgz and isn't being modified while being transmitted. This has happened maybe three times this year and unfortunately I've just had to deal with it rather than invest the time to do the research.
Harriscomputer
Leroy Tennison Network Information/Cyber Security Sp
Hi Leroy,
I think that in my case I could not use a tgz archive. I'm speaking about full backups that reach 600/700GiB, compressing them and then rsync them could take so much time that it will be useless.
unless you use tape (of that high capacity), it is advantageous to restrict volume size to, say, 50GB. Then when you restore, search for specific files will be faster. And it will help your backup volumes transfers as well.
Valeri
Hi Valeri,
thank you for your suggestion.
Is bacula the right backup system when I need to replicate data offsite? There are other backup solution that simplify this process?
Bacula is great enterprise level open source backup system. I switched to its fork bareos at some point; I use bacula/bareos for at least a decade. And with this your extra requirement I still would stay with bareos (or bacula).
If I were to have two sets of backup: on site and off site, I would just set up separate bacula/bareos director and storage daemon(s) off site. Add to FDs (file daemons) extra instances of director - offsite one with different passwords for the sake of security. Then there will be a set of everything off site, not only a set of volumes. Of course, if you only have a set of volumes, but everything else has evaporated, you still will be able to restore everything, including database records by scanning set of volumes. Which will take forever. I would alternate dates of backups in offsite/onsite schedules, or define times of backups so that they do not overlap.
Another good news of this vs just rsyncing volumes is: bacula/bareos verifies checksum of every backed up file after receiving it. This will ensure consistency of files in remote volumes, for rsync you will have to at least verify checksum of each volume transferred to destination (unless I miss my wits and rsync does verify checksums of files transferred, I just re-read rsync man and don't see verification - hopefully rsync expert will chime in and correct me if I'm wrong about rsync).
Anyway, that is what I would do.
Valeri
Thank you in advance
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Il 02/07/20 16:39, Valeri Galtsev ha scritto:
On 2020-07-02 08:28, Alessandro Baggi wrote:
Il 02/07/20 15:02, Valeri Galtsev ha scritto:
On 7/2/20 3:22 AM, Alessandro Baggi wrote:
Il 01/07/20 17:13, Leroy Tennison ha scritto:
I realize this shouldn't happen, the file is a tgz and isn't being modified while being transmitted. This has happened maybe three times this year and unfortunately I've just had to deal with it rather than invest the time to do the research.
Harriscomputer
Leroy Tennison Network Information/Cyber Security Sp
Hi Leroy,
I think that in my case I could not use a tgz archive. I'm speaking about full backups that reach 600/700GiB, compressing them and then rsync them could take so much time that it will be useless.
unless you use tape (of that high capacity), it is advantageous to restrict volume size to, say, 50GB. Then when you restore, search for specific files will be faster. And it will help your backup volumes transfers as well.
Valeri
Hi Valeri,
thank you for your suggestion.
Is bacula the right backup system when I need to replicate data offsite? There are other backup solution that simplify this process?
Bacula is great enterprise level open source backup system. I switched to its fork bareos at some point; I use bacula/bareos for at least a decade. And with this your extra requirement I still would stay with bareos (or bacula).
If I were to have two sets of backup: on site and off site, I would just set up separate bacula/bareos director and storage daemon(s) off site. Add to FDs (file daemons) extra instances of director - offsite one with different passwords for the sake of security. Then there will be a set of everything off site, not only a set of volumes. Of course, if you only have a set of volumes, but everything else has evaporated, you still will be able to restore everything, including database records by scanning set of volumes. Which will take forever. I would alternate dates of backups in offsite/onsite schedules, or define times of backups so that they do not overlap.
Another good news of this vs just rsyncing volumes is: bacula/bareos verifies checksum of every backed up file after receiving it. This will ensure consistency of files in remote volumes, for rsync you will have to at least verify checksum of each volume transferred to destination (unless I miss my wits and rsync does verify checksums of files transferred, I just re-read rsync man and don't see verification
- hopefully rsync expert will chime in and correct me if I'm wrong
about rsync).
Anyway, that is what I would do.
Valeri
Hi Valeri,
I'm in late but thank you for your suggestion.
Depending on the definition of offsite, you have a fundamental problem: either invest the time/effort compressing or take extra bandwidth, which is less costly? Hopefully a delta transfer makes sense in your situation and should save far more than compression would once the original copy is offsite.
________________________________ From: CentOS centos-bounces@centos.org on behalf of Valeri Galtsev galtsev@kicp.uchicago.edu Sent: Thursday, July 2, 2020 8:02 AM To: centos@centos.org centos@centos.org Subject: [EXTERNAL] Re: [CentOS] [OT] Bacula offsite replication
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Harriscomputer
Leroy Tennison Network Information/Cyber Security Specialist E: leroy@datavoiceint.com
[cid:Data-Voice-International-LOGO_aa3d1c6e-5cfb-451f-ba2c-af8059e69609.PNG]
2220 Bush Dr McKinney, Texas 75070 www.datavoiceint.comhttp://www..com
This message has been sent on behalf of a company that is part of the Harris Operating Group of Constellation Software Inc.
If you prefer not to be contacted by Harris Operating Group please notify ushttp://subscribe.harriscomputer.com/.
This message is intended exclusively for the individual or entity to which it is addressed. This communication may contain information that is proprietary, privileged or confidential or otherwise legally exempt from disclosure. If you are not the named addressee, you are not authorized to read, print, retain, copy or disseminate this message or any part of it. If you have received this message in error, please notify the sender immediately by e-mail and delete all copies of the message.
On 7/2/20 3:22 AM, Alessandro Baggi wrote:
Il 01/07/20 17:13, Leroy Tennison ha scritto:
I realize this shouldn't happen, the file is a tgz and isn't being modified while being transmitted. This has happened maybe three times this year and unfortunately I've just had to deal with it rather than invest the time to do the research.
Harriscomputer
Leroy Tennison Network Information/Cyber Security Sp
Hi Leroy,
I think that in my case I could not use a tgz archive. I'm speaking about full backups that reach 600/700GiB, compressing them and then rsync them could take so much time that it will be useless.
unless you use tape (of that high capacity), it is advantageous to restrict volume size to, say, 50GB. Then when you restore, search for specific files will be faster. And it will help your backup volumes transfers as well.
Valeri
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
-- ++++++++++++++++++++++++++++++++++++++++ Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247 ++++++++++++++++++++++++++++++++++++++++ _______________________________________________ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
-
Alessandro Baggi -
John Pierce -
Leon Fauster -
Leroy Tennison -
Valeri Galtsev