I have a kvm virtual host running on what will become CentOS 6 with 12GB of memory and a Quad Xeon X5560 2.8Ghz . The store for virtual machines will be a software raid 6 array of 6 disks with an LVM layered on top. I'm not initially planning any major overcommitment of resources, though there could be a need for some overcommitment with a light workload on the guests.
In recent years people seem to configure a wide range of different swap allocations. I was thinking initially to spread swap across seperate non-raid partitions on 4 of these disks, but the downside of that is if I put 2gb on each disk, then I can only swap processes that will fit in 2gb swap space. Also, if one of the disks fails, I have to reboot if anything was swapped to that drive.
My questions are as follows:
1. What experience are others having with putting swap space on raid partitions? I was thinking about maybe swapping on a raid10 device, otherwise an LVM spanning multiple drives. 2. In practice, what kinds of swap allocation are people finding useful for a kvm virtual host of this size?
I definitely don't want a system that is so overcommited that performance is impacted, but if some overcommitment is reasonable for VM's that have light workload, then I consider that. I can increase system resources when that becomes necessary.
Nataraj
On Mon, Mar 14, 2011 at 5:24 PM, Nataraj incoming-centos@rjl.com wrote: [snip]
In recent years people seem to configure a wide range of different swap allocations. I was thinking initially to spread swap across seperate non-raid partitions on 4 of these disks, but the downside of that is if I put 2gb on each disk, then I can only swap processes that will fit in 2gb swap space. Also, if one of the disks fails, I have to reboot if anything was swapped to that drive.
I'm not certain that's entirely accurate... It's not so much "swap space" as "page space". I.e., there are pages being swapped out rather than entire processes.
on 14:24 Mon 14 Mar, Nataraj (incoming-centos@rjl.com) wrote:
I have a kvm virtual host running on what will become CentOS 6 with 12GB of memory and a Quad Xeon X5560 2.8Ghz . The store for virtual machines will be a software raid 6 array of 6 disks with an LVM layered on top. I'm not initially planning any major overcommitment of resources, though there could be a need for some overcommitment with a light workload on the guests.
In recent years people seem to configure a wide range of different swap allocations. I was thinking initially to spread swap across seperate non-raid partitions on 4 of these disks, but the downside of that is if I put 2gb on each disk, then I can only swap processes that will fit in 2gb swap space.
Incorrect.
Linux processes aren't "swapped" to disk (the entire process memory space), but are paged (given memory blocks are swapped out individually).
Swap allocated over multiple spindles is effectively striped (treated as one large RAID 0 partition).
If you've got SSD, you'll get even better swap performance.
For an excellent explanation of how Linux pages / handles memory:
http://sourcefrog.net/weblog/software/linux-kernel/swap.html
Also, if one of the disks fails, I have to reboot if anything was swapped to that drive.
My questions are as follows:
- What experience are others having with putting swap space on raid partitions? I was thinking about maybe swapping on a raid10 device, otherwise an LVM spanning multiple drives.
I'd allocate swap to the raw devices rather than the RAID devices, particularly if using SW RAID. In the case of HW RAID, it's a bit of a toss-up. Whatever's easier to manage.
- In practice, what kinds of swap allocation are people finding useful for a kvm virtual host of this size?
1-3x RAM is still my rule of thumb.
I definitely don't want a system that is so overcommited that performance is impacted, but if some overcommitment is reasonable for VM's that have light workload, then I consider that. I can increase system resources when that becomes necessary.
For this, you'll want to set the overcommit and swappiness kernel parameters. Amount of swap space is a secondary consideration. How much swap you /have/ and how much swap you're /doing/ are two different things.
On 03/14/11 3:02 PM, Dr. Ed Morbius wrote:
I'd allocate swap to the raw devices rather than the RAID devices,
why would you use RAID and not RAID your swap? the primary (many say, only) purpose of RAID is to maintain uptime, and having any swap device fail will ensure your system immediately crashes.
I *always* mirror my swap devices on any system that is using RAID.
On 03/14/2011 03:07 PM, John R Pierce wrote:
On 03/14/11 3:02 PM, Dr. Ed Morbius wrote:
I'd allocate swap to the raw devices rather than the RAID devices,
why would you use RAID and not RAID your swap? the primary (many say, only) purpose of RAID is to maintain uptime, and having any swap device fail will ensure your system immediately crashes.
I *always* mirror my swap devices on any system that is using RAID.
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Do you have any sense of whether this takes a big performance hit with software raid? (I was actually planning on hardware raid but the system got ordered incorrectly).
Thank You, Nataraj
On Mon, Mar 14, 2011 at 11:45 PM, Nataraj incoming-centos@rjl.com wrote:
Do you have any sense of whether this takes a big performance hit with software raid? (I was actually planning on hardware raid but the system got ordered incorrectly).
I have lots of machines running linux raid and swap on top of it, haven't noticed anything out of order so far; go for it.
On 03/14/11 4:45 PM, Nataraj wrote:
Do you have any sense of whether this takes a big performance hit with software raid? (I was actually planning on hardware raid but the system got ordered incorrectly).
if you have adequate memory for your workload, very little data gets written to swap, and what is written is done in the background so its of little importance to performance.
if your swap is mirrored, writes have to be done to both devices, while reads can be load balanced across them, so the write performance is equivalent to a single disk, while the read performance in aggregate may be twice as fast as a single drive.
if you DONT have adequate memory for your workload, your performance will suck pretty badly regardless of how fast the swap device(s) are.
On Mon, Mar 14, 2011 at 5:24 PM, Nataraj incoming-centos@rjl.com wrote:
I have a kvm virtual host running on what will become CentOS 6 with 12GB of memory and a Quad Xeon X5560 2.8Ghz . The store for virtual machines will be a software raid 6 array of 6 disks with an LVM layered on top. I'm not initially planning any major overcommitment of resources, though there could be a need for some overcommitment with a light workload on the guests.
In recent years people seem to configure a wide range of different swap allocations. I was thinking initially to spread swap across seperate non-raid partitions on 4 of these disks, but the downside of that is if I put 2gb on each disk, then I can only swap processes that will fit in 2gb swap space. Also, if one of the disks fails, I have to reboot if anything was swapped to that drive.
My questions are as follows:
What experience are others having with putting swap space on raid partitions? I was thinking about maybe swapping on a raid10 device, otherwise an LVM spanning multiple drives. In practice, what kinds of swap allocation are people finding useful for a kvm virtual host of this size?
I definitely don't want a system that is so overcommited that performance is impacted, but if some overcommitment is reasonable for VM's that have light workload, then I consider that. I can increase system resources when that becomes necessary.
Nataraj
I recommend against using any type of parity-based RAID (5 or 6) for any workload, period. RAID10 is the only way to go for everything these days. VMs often run into IO issues, and RAID[56] is going to make that problem worse. Also, RAID10 in software shouldn't have much performance impact at all, as opposed to RAID[56] which will have a big load in software.
I wouldn't worry as much about swap, as others have said. Money spent adding another device for swap would be better spent on more RAM. However, I usually put swap on the same device as everything else. It might not strictly be best practice, but again, if you're hitting swap, you have bigger problems.
-
Brian Mathis -
Dr. Ed Morbius -
John R Pierce -
Kwan Lowe -
Lucian -
Nataraj