Hi, I have a strange phenomenon that I cannot readily explain so I wonder if anyone here can shed a light on this.
The host system is a Dell r815 with 64 cores and 256G ram and has centos 6 installed. The five guests are also running centos 6 and are running as a hadoop cluster. The problem is that I see disk-io spikes in the vm's which then cause soft lockups in the guest but I also see hanging processes on the host as if the entire machine locks up for 30-60 seconds.
Now I know that having all cluster members running on the same system isn't efficient and that I cannot expect good performance but what I was not expecting is that a guest make host processes hang. Does anyone have an idea what the issue could be here or how I can find out what cause for this behavior is?
Regards, Dennis
I experienced similar issues when disk images of virtual machines were stored in qcow/qcow2 files instead of logical volumes (LVM). Using LVM gives you way better I/O performace than using qcow files.
Also very important: when you partition your disk drive(s) make sure that partitions are properly aligned to the physical allocation block size of the hard drive you use. Let's say your hard drive uses 4k sectors then every partition you create must start at a 4k boundary. If your partitions are mis-aligned then you'll get terrible disk I/O performace, just like the one you have described. For more info see, for example: http://www.thomas-krenn.com/en/wiki/Partition_Alignment
You can also tweak Linux to get better KVM performance. For more info you can check out these documents: http://pic.dhe.ibm.com/infocenter/lnxinfo/v3r0m0/topic/liaat/liaatbestpracti... http://www.novell.com/docrep/2013/05/kvm_virtualized_io_performance.pdf
Zoltan
On 2/20/2014 5:53 PM, Dennis Jacobfeuerborn wrote:
Hi, I have a strange phenomenon that I cannot readily explain so I wonder if anyone here can shed a light on this.
The host system is a Dell r815 with 64 cores and 256G ram and has centos 6 installed. The five guests are also running centos 6 and are running as a hadoop cluster. The problem is that I see disk-io spikes in the vm's which then cause soft lockups in the guest but I also see hanging processes on the host as if the entire machine locks up for 30-60 seconds.
Now I know that having all cluster members running on the same system isn't efficient and that I cannot expect good performance but what I was not expecting is that a guest make host processes hang. Does anyone have an idea what the issue could be here or how I can find out what cause for this behavior is?
Regards, Dennis _______________________________________________ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
What helped a lot for me is to increase the read ahead drastically on guests:
# Set read-ahead for optimal disk I/O blockdev --setra 16384 /dev/vda
I optained this value by repeatedly timing copies and this was, for my installation at least, optimal.
-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_- Eskimo North Linux Friendly Internet Access, Shell Accounts, and Hosting. Knowledgeable human assistance, not telephone trees or script readers. See our web site: http://www.eskimo.com/ (206) 812-0051 or (800) 246-6874.
On Thu, 20 Feb 2014, Zoltan Frombach wrote:
Date: Thu, 20 Feb 2014 19:01:10 +0100 From: Zoltan Frombach zoltan@frombach.com Reply-To: Discussion about the virtualization on CentOS centos-virt@centos.org To: Discussion about the virtualization on CentOS centos-virt@centos.org Subject: Re: [CentOS-virt] disk io in guests causes soft lockups in guests and host processes
I experienced similar issues when disk images of virtual machines were stored in qcow/qcow2 files instead of logical volumes (LVM). Using LVM gives you way better I/O performace than using qcow files.
Also very important: when you partition your disk drive(s) make sure that partitions are properly aligned to the physical allocation block size of the hard drive you use. Let's say your hard drive uses 4k sectors then every partition you create must start at a 4k boundary. If your partitions are mis-aligned then you'll get terrible disk I/O performace, just like the one you have described. For more info see, for example: http://www.thomas-krenn.com/en/wiki/Partition_Alignment
You can also tweak Linux to get better KVM performance. For more info you can check out these documents: http://pic.dhe.ibm.com/infocenter/lnxinfo/v3r0m0/topic/liaat/liaatbestpracti... http://www.novell.com/docrep/2013/05/kvm_virtualized_io_performance.pdf
Zoltan
On 2/20/2014 5:53 PM, Dennis Jacobfeuerborn wrote:
Hi, I have a strange phenomenon that I cannot readily explain so I wonder if anyone here can shed a light on this.
The host system is a Dell r815 with 64 cores and 256G ram and has centos 6 installed. The five guests are also running centos 6 and are running as a hadoop cluster. The problem is that I see disk-io spikes in the vm's which then cause soft lockups in the guest but I also see hanging processes on the host as if the entire machine locks up for 30-60 seconds.
Now I know that having all cluster members running on the same system isn't efficient and that I cannot expect good performance but what I was not expecting is that a guest make host processes hang. Does anyone have an idea what the issue could be here or how I can find out what cause for this behavior is?
Regards, Dennis _______________________________________________ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
-
Dennis Jacobfeuerborn -
Robert Dinse -
Zoltan Frombach