Hi Everyone,
I need to keep 2 systems identical. Mostly e-mail directories, web directories, mysql, etc. The goal here is to have a 2nd system ready to go it the first one starts to exhibit hardware issues.
What are options to have this happen? I'm going out on a limb and thinking rsync but I haven't used it past just simple use cases.
Can anyone provide some insight for me?
CentOS 6.5.
Jason
On 10/9/2014 10:39 AM, Jason T. Slack-Moehrle wrote:
I need to keep 2 systems identical. Mostly e-mail directories, web directories, mysql, etc. The goal here is to have a 2nd system ready to go it the first one starts to exhibit hardware issues.
What are options to have this happen? I'm going out on a limb and thinking rsync but I haven't used it past just simple use cases.
Can anyone provide some insight for me?
drbd is the live replication system, this does block level replication of logical volumes. rsync doesn't handle changing-on-the-fly stuff very well, especially stuff like a database file system which is undergoing constant random write activity.
but note, the drbd slave copy can't be mounted until drbd is stopped, or all heck breaks loose.
Hi John,
On Thu, Oct 9, 2014 at 10:44 AM, John R Pierce pierce@hogranch.com wrote:
On 10/9/2014 10:39 AM, Jason T. Slack-Moehrle wrote:
I need to keep 2 systems identical. Mostly e-mail directories, web directories, mysql, etc. The goal here is to have a 2nd system ready to go it the first one starts to exhibit hardware issues.
What are options to have this happen? I'm going out on a limb and thinking rsync but I haven't used it past just simple use cases.
Can anyone provide some insight for me?
drbd is the live replication system, this does block level replication of logical volumes. rsync doesn't handle changing-on-the-fly stuff very well, especially stuff like a database file system which is undergoing constant random write activity.
but note, the drbd slave copy can't be mounted until drbd is stopped, or all heck breaks loose.
Just to clarify. What state is the slave in?
If master goes down, how does the slave become active? Just reboot it and let it come up? The wording "slave copy can't be mounted until drdb is stopped" leads me to believe the slave is in some alternate state to be receiving the blocks of data.
On 10/9/2014 11:57 AM, Jason T. Slack-Moehrle wrote:
Just to clarify. What state is the slave in?
If master goes down, how does the slave become active? Just reboot it and let it come up? The wording "slave copy can't be mounted until drdb is stopped" leads me to believe the slave is in some alternate state to be receiving the blocks of data.
the slave is up and running, but the file systems you're replicating are unmounted, and its services are stopped, so you could consider this to be a 'standby' state.
yes, to use drbd, its important that you put your email spools, databases, etc, on dedicated file system(s), NOT on the OS root file system. I generally use lvm for all this.
a cluster management package, such as the ones suggested by another poster, would take care of all this for you (once you have things setup properly), if the master fails, it would 'activate' the slave, switch its IP[*] over to be the 'production' system, and mount its file systems, starting its services (mysqld, postfix, etc) per your configuration.
[*] typically, you use THREE IP addresses for a HA cluster. a unique IP for each system, used only for management, and a 'service' IP used for the production accesses, which is held by the currently active system. when the master fails, the slave adopts this service IP.
Hi John,
On Thu, Oct 9, 2014 at 12:53 PM, John R Pierce pierce@hogranch.com wrote:
On 10/9/2014 11:57 AM, Jason T. Slack-Moehrle wrote:
Just to clarify. What state is the slave in?
If master goes down, how does the slave become active? Just reboot it and let it come up? The wording "slave copy can't be mounted until drdb is stopped" leads me to believe the slave is in some alternate state to be receiving the blocks of data.
the slave is up and running, but the file systems you're replicating are unmounted, and its services are stopped, so you could consider this to be a 'standby' state.
yes, to use drbd, its important that you put your email spools, databases, etc, on dedicated file system(s), NOT on the OS root file system. I generally use lvm for all this.
a cluster management package, such as the ones suggested by another poster, would take care of all this for you (once you have things setup properly), if the master fails, it would 'activate' the slave, switch its IP[*] over to be the 'production' system, and mount its file systems, starting its services (mysqld, postfix, etc) per your configuration.
[*] typically, you use THREE IP addresses for a HA cluster. a unique IP for each system, used only for management, and a 'service' IP used for the production accesses, which is held by the currently active system. when the master fails, the slave adopts this service IP.
Thank you for this info. This clears up a lot and is very helpful.
Jason
Most people forget that DRDB is a 'clustered' storage that creates a new block device on top of existing block devices, over the network.
There are issues like SPLIT BRAIN and a possible data inconsistency, should there be multiple network timeouts, congestions (since it is all network based) etc. This may not be the best solution for your problem!
On 10/9/2014 8:19 PM, Micky wrote:
Most people forget that DRDB is a 'clustered' storage that creates a new block device on top of existing block devices, over the network.
There are issues like SPLIT BRAIN and a possible data inconsistency, should there be multiple network timeouts, congestions (since it is all network based) etc. This may not be the best solution for your problem!
those are potential issues with ANY high availability clustering solution that need to be carefully engineered around. HA clusters commonly use fencing technology to prevent split brain, dedicated network adapters just for replication, and redundant connections for status monitoring, etc.
On Thu, Oct 9, 2014 at 11:43 PM, John R Pierce pierce@hogranch.com wrote:
Most people forget that DRDB is a 'clustered' storage that creates a new block device on top of existing block devices, over the network.
There are issues like SPLIT BRAIN and a possible data inconsistency, should there be multiple network timeouts, congestions (since it is all network based) etc. This may not be the best solution for your problem!
those are potential issues with ANY high availability clustering solution that need to be carefully engineered around. HA clusters commonly use fencing technology to prevent split brain, dedicated network adapters just for replication, and redundant connections for status monitoring, etc.
And you still need backups. Other things besides hardware failures can wipe your data and with mirrors/clusters, software or operator errors will take all the copies at once.
On 2014-10-09, Jason T. Slack-Moehrle slackmoehrle@gmail.com wrote:
I need to keep 2 systems identical. Mostly e-mail directories, web directories, mysql, etc. The goal here is to have a 2nd system ready to go it the first one starts to exhibit hardware issues.
What are options to have this happen? I'm going out on a limb and thinking rsync but I haven't used it past just simple use cases.
You can try lsyncd.
https://code.google.com/p/lsyncd/
It's basically a daemon monitoring inotify events and sending them to an rsync, so it can keep things up to date more easily. (I have never used it myself, so caveat emptor.)
--keith
On 10/9/2014 10:55 AM, Keith Keller wrote:
[lysncd is] basically a daemon monitoring inotify events and sending them to an rsync, so it can keep things up to date more easily. (I have never used it myself, so caveat emptor.)
rsync is only safe to use on something like a database server if you stop the database each time you run rsync, otherwise the files will get out of sync.
I suppose you could setup mysql replication, and exclude mysql from the lsyncd/rsync, that would be one approach.
On 2014-10-09, John R Pierce pierce@hogranch.com wrote:
On 10/9/2014 10:55 AM, Keith Keller wrote:
[lysncd is] basically a daemon monitoring inotify events and sending them to an rsync, so it can keep things up to date more easily. (I have never used it myself, so caveat emptor.)
rsync is only safe to use on something like a database server if you stop the database each time you run rsync, otherwise the files will get out of sync.
Ah, yes, I missed the OP's reference to mysql. Sorry about that!
I suppose you could setup mysql replication, and exclude mysql from the lsyncd/rsync, that would be one approach.
mysql/mariadb replication is a reasonable way to go, and probably has more eyes on it than lsyncd. MariaDB's somewhat new GTID replication might be more reliable than the old query/row/mixed replication scheme they had before.
--keith
On Thu, October 9, 2014 12:39 pm, Jason T. Slack-Moehrle wrote:
Hi Everyone,
I need to keep 2 systems identical. Mostly e-mail directories, web directories, mysql,
With databases you can not make sync of their data directories (on a filesystem level) while the daemon is running and accessing them in general. You may think of mysql redundant server or replication... Take a look here:
https://dev.mysql.com/doc/refman/5.0/en/replication.html
Valeri
etc. The goal here is to have a 2nd system ready to go it the first one starts to exhibit hardware issues.
What are options to have this happen? I'm going out on a limb and thinking rsync but I haven't used it past just simple use cases.
Can anyone provide some insight for me?
CentOS 6.5.
Jason _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
++++++++++++++++++++++++++++++++++++++++ Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247 ++++++++++++++++++++++++++++++++++++++++
On 10/9/2014 11:06 AM, Andrew Holway wrote:
Can anyone provide some insight for me?
This would be a perfect use case for ZFS snapshot send and receive!
Its not perfect if you don't want to lose all database transactions, new files, and email between the last update and the point of hardware failure.
drbd really is the best answer as its continuous in-order replication.
drbd really is the best answer as its continuous in-order replication.
Assuming of course that you dont need any kind of performance from your filesystem drdb is indeed the answer.
ZFS snapshots can however be run on a per second basis which is quite neat! What ever you choose to do you would probably want to use mysql binary replication anyway so having a completely sync filesystem replica is probably not too much of a problem.
I must point out that virtualisation made these kinds of problems mostly go away (assuming of course that you have cash for some kind of High Availability iSCSI device or something.
-- john r pierce 37N 122W somewhere on the middle of the left coast
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Am 09.10.2014 um 20:11 schrieb John R Pierce pierce@hogranch.com:
On 10/9/2014 11:06 AM, Andrew Holway wrote:
Can anyone provide some insight for me?
This would be a perfect use case for ZFS snapshot send and receive!
Its not perfect if you don't want to lose all database transactions, new files, and email between the last update and the point of hardware failure.
drbd really is the best answer as its continuous in-order replication.
+1
in conjunction with pacemaker and corosync it would also switch over when some hardware issues appears.
-- LF
-
Andrew Holway -
Jason T. Slack-Moehrle -
John R Pierce -
Keith Keller -
Leon Fauster -
Les Mikesell -
Micky -
Valeri Galtsev