Hey there, I'm testing LVM snapshot to make backups and I've some questions about snapshot lvm.
In my test case I have httpd, postgresql db and a dataset on a VM.
Running a backup I follow this steps:
1) Stop httpd 2) Create lvm snapshot on the dataset 3) Backup database 4) restart httpd (to avoid more downtime) 5) mount the snapshot and execute backup 6) umount and remove the snapshot
I think that this could be fine (if not please correct me)
Now when bacula performs the backup what happen if bacula are copying a specified file and this specified file is modified?
Bacula will copy the right file using inode (this will copy the wrong file?) or copy entire file located in the snapshot?
Thanks in advance.
On 9/17/18 7:50 AM, Alessandro Baggi wrote:
Running a backup I follow this steps:
- Stop httpd
- Create lvm snapshot on the dataset
- Backup database
- restart httpd (to avoid more downtime)
- mount the snapshot and execute backup
- umount and remove the snapshot
I think that this could be fine (if not please correct me)
That doesn't look right. It should look more like 1) stop or freeze all of the services (httpd and database), 2) make the snapshot, 3) start or thaw all of the services, 4) mount the snapshot, 5) back up the data, 6) remove the snapshot.
Your sequence makes changes (step 3) after the snapshot is taken. In that case, the backup that you made will not be a part of the snapshot. It also prolongs the time that httpd is down unnecessarily.
Now when bacula performs the backup what happen if bacula are copying a specifed file and this specified file is modified?
If bacula is backing up the content of the snapshot, then changes made in the running services won't affect it. That's the purpose of the snapshot: it's static.
Are you using bacula's built-in snapshot support, or are you rolling your own?
Il 17/09/2018 22:12, Gordon Messmer ha scritto:
On 9/17/18 7:50 AM, Alessandro Baggi wrote:
Running a backup I follow this steps:
- Stop httpd
- Create lvm snapshot on the dataset
- Backup database
- restart httpd (to avoid more downtime)
- mount the snapshot and execute backup
- umount and remove the snapshot
I think that this could be fine (if not please correct me)
That doesn't look right. It should look more like 1) stop or freeze all of the services (httpd and database), 2) make the snapshot, 3) start or thaw all of the services, 4) mount the snapshot, 5) back up the data, 6) remove the snapshot.
About database setup I perform backups via pg_dump so how the snapshot affects pgsql database? What your suggestion I must perform database backup copying only filesystem file and not pgsql.sql database dump?
Your sequence makes changes (step 3) after the snapshot is taken. In that case, the backup that you made will not be a part of the snapshot. It also prolongs the time that httpd is down unnecessarily.
Now when bacula performs the backup what happen if bacula are copying a specifed file and this specified file is modified?
If bacula is backing up the content of the snapshot, then changes made in the running services won't affect it. That's the purpose of the snapshot: it's static.
Are you using bacula's built-in snapshot support, or are you rolling your own?
No I'm using pre/post job script where I have lvm commands to create and destroy snapshot volume.
On 9/17/18 11:38 PM, Alessandro Baggi wrote:
Il 17/09/2018 22:12, Gordon Messmer ha scritto:
That doesn't look right. It should look more like 1) stop or freeze all of the services (httpd and database), 2) make the snapshot, 3) start or thaw all of the services, 4) mount the snapshot, 5) back up the data, 6) remove the snapshot.
About database setup I perform backups via pg_dump so how the snapshot affects pgsql database? What your suggestion I must perform database backup copying only filesystem file and not pgsql.sql database dump?
If you want a plain-text dump of the DB, you can do that before the LVM snapshot sequence: 1) pg_dump, 2) stop or freeze all of the services (httpd and database), 3) make the snapshot, 4) start or thaw all of the services, 5) mount the snapshot, 6) back up the data, 7) remove the snapshot.
Typically, the reason you want to use snapshots for the backup is that you don't need to do pg_dump to get a consistent DB backup, though. pg_dump style backups are extremely slow to restore. If you freeze the DB, make a snapshot, and thaw, you can make a safe, consistent backup of the DB files directly, and restore in minimal time.
Are you using bacula's built-in snapshot support, or are you rolling your own?
No I'm using pre/post job script where I have lvm commands to create and destroy snapshot volume.
I really recommend using a ready-made process rather than rolling your own. Bacula has snapshot support. Alternately, my project can manage snapshots and handle freezing / thawing PostgreSQL services. I think it's a better option than Bacula's, but either is better than reinventing this wheel.
Il 18/09/2018 17:14, Gordon Messmer ha scritto:
On 9/17/18 11:38 PM, Alessandro Baggi wrote:
Il 17/09/2018 22:12, Gordon Messmer ha scritto:
That doesn't look right. It should look more like 1) stop or freeze all of the services (httpd and database), 2) make the snapshot, 3) start or thaw all of the services, 4) mount the snapshot, 5) back up the data, 6) remove the snapshot.
About database setup I perform backups via pg_dump so how the snapshot affects pgsql database? What your suggestion I must perform database backup copying only filesystem file and not pgsql.sql database dump?
If you want a plain-text dump of the DB, you can do that before the LVM snapshot sequence: 1) pg_dump, 2) stop or freeze all of the services (httpd and database), 3) make the snapshot, 4) start or thaw all of the services, 5) mount the snapshot, 6) back up the data, 7) remove the snapshot.
Typically, the reason you want to use snapshots for the backup is that you don't need to do pg_dump to get a consistent DB backup, though. pg_dump style backups are extremely slow to restore. If you freeze the DB, make a snapshot, and thaw, you can make a safe, consistent backup of the DB files directly, and restore in minimal time.
Are you using bacula's built-in snapshot support, or are you rolling your own?
No I'm using pre/post job script where I have lvm commands to create and destroy snapshot volume.
I really recommend using a ready-made process rather than rolling your own. Bacula has snapshot support. Alternately, my project can manage snapshots and handle freezing / thawing PostgreSQL services. I think it's a better option than Bacula's, but either is better than reinventing this wheel.
https://bitbucket.org/gordonmessmer/dragonsdawn-snapshot
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Thank you for the suggestion.
I don't know why I considered pg_dump better then filesystem backup. At this moment I prefer pg_dump because in this mode I can restore data on different version of postgresql. With filesystem dump I can restore only for a specific version. Is right?
I will give a try.
Thank you again for suggestions.
On Wed, 2018-09-19 at 08:55 +0200, Alessandro Baggi wrote:
Il 18/09/2018 17:14, Gordon Messmer ha scritto:
On 9/17/18 11:38 PM, Alessandro Baggi wrote:
Il 17/09/2018 22:12, Gordon Messmer ha scritto:
That doesn't look right. It should look more like 1) stop or freeze all of the services (httpd and database), 2) make the snapshot, 3) start or thaw all of the services, 4) mount the snapshot, 5) back up the data, 6) remove the snapshot.
About database setup I perform backups via pg_dump so how the snapshot affects pgsql database? What your suggestion I must perform database backup copying only filesystem file and not pgsql.sql database dump?
If you want a plain-text dump of the DB, you can do that before the LVM snapshot sequence: 1) pg_dump, 2) stop or freeze all of the services (httpd and database), 3) make the snapshot, 4) start or thaw all of the services, 5) mount the snapshot, 6) back up the data, 7) remove the snapshot.
Typically, the reason you want to use snapshots for the backup is that you don't need to do pg_dump to get a consistent DB backup, though. pg_dump style backups are extremely slow to restore. If you freeze the DB, make a snapshot, and thaw, you can make a safe, consistent backup of the DB files directly, and restore in minimal time.
Are you using bacula's built-in snapshot support, or are you rolling your own?
No I'm using pre/post job script where I have lvm commands to create and destroy snapshot volume.
I really recommend using a ready-made process rather than rolling your own. Bacula has snapshot support. Alternately, my project can manage snapshots and handle freezing / thawing PostgreSQL services. I think it's a better option than Bacula's, but either is better than reinventing this wheel.
https://bitbucket.org/gordonmessmer/dragonsdawn-snapshot
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Thank you for the suggestion.
I don't know why I considered pg_dump better then filesystem backup. At this moment I prefer pg_dump because in this mode I can restore data on different version of postgresql. With filesystem dump I can restore only for a specific version. Is right?
I will give a try.
Thank you again for suggestions.
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Hi,
an restore may need additional attention while restoring database files to a different version of postgres. However, while the versions does not differ that much (an official upgrade path exists, or no layout change was made between the versions) the files work just as if you updated the database using yum (check out spec file from source pkg to find out what happens on package upgrades, you may need to do upgrade steps manually).
This should only be a problem when you try to restore to a new major version of postgres. But in that case I would recommend an additional pg_dump backup to have a save fallback.
best regards
ps: I do not use postgres, please understand my testimony as not tested practically.
Il 19/09/2018 09:28, Lange, Markus ha scritto:
On Wed, 2018-09-19 at 08:55 +0200, Alessandro Baggi wrote:
Il 18/09/2018 17:14, Gordon Messmer ha scritto:
On 9/17/18 11:38 PM, Alessandro Baggi wrote:
Il 17/09/2018 22:12, Gordon Messmer ha scritto:
That doesn't look right. It should look more like 1) stop or freeze all of the services (httpd and database), 2) make the snapshot, 3) start or thaw all of the services, 4) mount the snapshot, 5) back up the data, 6) remove the snapshot.
About database setup I perform backups via pg_dump so how the snapshot affects pgsql database? What your suggestion I must perform database backup copying only filesystem file and not pgsql.sql database dump?
If you want a plain-text dump of the DB, you can do that before the LVM snapshot sequence: 1) pg_dump, 2) stop or freeze all of the services (httpd and database), 3) make the snapshot, 4) start or thaw all of the services, 5) mount the snapshot, 6) back up the data, 7) remove the snapshot.
Typically, the reason you want to use snapshots for the backup is that you don't need to do pg_dump to get a consistent DB backup, though. pg_dump style backups are extremely slow to restore. If you freeze the DB, make a snapshot, and thaw, you can make a safe, consistent backup of the DB files directly, and restore in minimal time.
Are you using bacula's built-in snapshot support, or are you rolling your own?
No I'm using pre/post job script where I have lvm commands to create and destroy snapshot volume.
I really recommend using a ready-made process rather than rolling your own. Bacula has snapshot support. Alternately, my project can manage snapshots and handle freezing / thawing PostgreSQL services. I think it's a better option than Bacula's, but either is better than reinventing this wheel.
https://bitbucket.org/gordonmessmer/dragonsdawn-snapshot
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Thank you for the suggestion.
I don't know why I considered pg_dump better then filesystem backup. At this moment I prefer pg_dump because in this mode I can restore data on different version of postgresql. With filesystem dump I can restore only for a specific version. Is right?
I will give a try.
Thank you again for suggestions.
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Hi,
an restore may need additional attention while restoring database files to a different version of postgres. However, while the versions does not differ that much (an official upgrade path exists, or no layout change was made between the versions) the files work just as if you updated the database using yum (check out spec file from source pkg to find out what happens on package upgrades, you may need to do upgrade steps manually).
This should only be a problem when you try to restore to a new major version of postgres. But in that case I would recommend an additional pg_dump backup to have a save fallback.
best regards
ps: I do not use postgres, please understand my testimony as not tested practically. _______________________________________________ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Thank you for the suggestion.
Best regards
On 9/18/18 11:55 PM, Alessandro Baggi wrote:
I don't know why I considered pg_dump better then filesystem backup. At this moment I prefer pg_dump because in this mode I can restore data on different version of postgresql. With filesystem dump I can restore only for a specific version. Is right?
That's right. pg_dump is an essential component of PostgreSQL upgrades, and it *can* be used as part of your backup process. On the up side: using pg_dump makes it possible to get consistent backups without filesystem snapshots. On the down side: using pg_dump is very slow, and makes it impossible to get incremental backups. For small databases, neither of those are terrible, but for large databases pg_dump is pretty awful.
With snapshots, you can back up the database files directly. It's reasonably fast, and you can typically improve backup time with incremental backups. You'd only need to pg_dump when you wanted to upgrade.
-
Alessandro Baggi -
Gordon Messmer -
Lange, Markus