Guests pausing suddenly

List overview All Threads
Download

newer

older

proper way to snapshot

Mac virtualization over KVM

Peter Hopfgartner

29 Feb 2012 29 Feb '12

7:53 a.m.

We have a CentOS 6.2 server with KVM. That server hosts 2 virtual machines, both with Centos 6.2, too.

Regularly, one or both of the virtual machines pass to state "pause" without apparent reason. On resume, I do get have messages, like the following in /var/log/messages.

Feb 28 21:50:45 achernar fcoemon: Failed to connect to lldpad Feb 29 08:23:56 achernar kernel: sd 0:0:0:0: [sda] Unhandled error code Feb 29 08:23:56 achernar kernel: sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT Feb 29 08:23:56 achernar kernel: sd 0:0:0:0: [sda] CDB: Write(10): 2a 00 06 db 70 78 00 00 38 00 Feb 29 08:23:56 achernar kernel: end_request: I/O error, dev sda, sector 115044472 Feb 29 08:23:56 achernar kernel: Buffer I/O error on device dm-0, logical block 14252047 Feb 29 08:23:56 achernar kernel: lost page write due to I/O error on dm-0 Feb 29 08:23:56 achernar kernel: Buffer I/O error on device dm-0, logical block 14252048 Feb 29 08:23:56 achernar kernel: lost page write due to I/O error on dm-0 Feb 29 08:23:56 achernar kernel: Buffer I/O error on device dm-0, logical block 14252049 Feb 29 08:23:56 achernar kernel: lost page write due to I/O error on dm-0 Feb 29 08:23:56 achernar kernel: Buffer I/O error on device dm-0, logical block 14252050 Feb 29 08:23:56 achernar kernel: lost page write due to I/O error on dm-0 Feb 29 08:23:56 achernar kernel: Buffer I/O error on device dm-0, logical block 14252051 Feb 29 08:23:56 achernar kernel: lost page write due to I/O error on dm-0 Feb 29 08:23:56 achernar kernel: Buffer I/O error on device dm-0, logical block 14252052 Feb 29 08:23:56 achernar kernel: lost page write due to I/O error on dm-0 Feb 29 08:23:56 achernar kernel: Buffer I/O error on device dm-0, logical block 14252053 Feb 29 08:23:56 achernar kernel: lost page write due to I/O error on dm-0 Feb 29 08:23:57 achernar fcoemon: error 111 Connection refused

I could not find any sensible message on the pysical host, neither in /var/log/messages nor in /var/log/libvirt.

We do have an almost identical server, same hardware, same software which does not have this problem.

How could I proceed to better diagnose the cause of the troubles?

Regards,

-- Peter Hopfgartner web : www.r3-gis.com

Show replies by date

Wahyu Darmawan

1 Mar 1 Mar

12:02 a.m.

Hi Peter,

I saw from your log message that you've lost connection to the storage device which you use in the server machine. My suggestion is please examine the connection between server and storage. CMIIW.

Rgds, Wahyu

-----Original Message----- From: Peter Hopfgartner peter.hopfgartner@r3-gis.com Sender: centos-virt-bounces@centos.org Date: Wed, 29 Feb 2012 08:53:09 To: Discussion about the virtualization on CentOScentos-virt@centos.org Reply-To: Discussion about the virtualization on CentOS centos-virt@centos.org Subject: [CentOS-virt] Guests pausing suddenly

We have a CentOS 6.2 server with KVM. That server hosts 2 virtual machines, both with Centos 6.2, too.

Regularly, one or both of the virtual machines pass to state "pause" without apparent reason. On resume, I do get have messages, like the following in /var/log/messages.

I could not find any sensible message on the pysical host, neither in /var/log/messages nor in /var/log/libvirt.

We do have an almost identical server, same hardware, same software which does not have this problem.

How could I proceed to better diagnose the cause of the troubles?

Regards,

-- Peter Hopfgartner web : www.r3-gis.com _______________________________________________ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt

Peter Hopfgartner

9:12 a.m.

Hi Wahyu,

I guess that the warnings related to FCoE are not that important. I forgot to mention, that the images of the virtual machines are on the physical server.

Thanks & Regards,

Peter

On 03/01/2012 01:02 AM, Wahyu Darmawan wrote:

...

Hi Peter,

I saw from your log message that you've lost connection to the storage device which you use in the server machine. My suggestion is please examine the connection between server and storage. CMIIW.

Rgds, Wahyu

Powered by Telkomsel BlackBerry®

-----Original Message----- From: Peter Hopfgartnerpeter.hopfgartner@r3-gis.com Sender: centos-virt-bounces@centos.org Date: Wed, 29 Feb 2012 08:53:09 To: Discussion about the virtualization on CentOScentos-virt@centos.org Reply-To: Discussion about the virtualization on CentOS centos-virt@centos.org Subject: [CentOS-virt] Guests pausing suddenly

We have a CentOS 6.2 server with KVM. That server hosts 2 virtual machines, both with Centos 6.2, too.

Regularly, one or both of the virtual machines pass to state "pause" without apparent reason. On resume, I do get have messages, like the following in /var/log/messages.

Feb 28 21:50:45 achernar fcoemon: Failed to connect to lldpad Feb 29 08:23:56 achernar kernel: sd 0:0:0:0: [sda] Unhandled error code Feb 29 08:23:56 achernar kernel: sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT Feb 29 08:23:56 achernar kernel: sd 0:0:0:0: [sda] CDB: Write(10): 2a 00 06 db 70 78 00 00 38 00 Feb 29 08:23:56 achernar kernel: end_request: I/O error, dev sda, sector 115044472 Feb 29 08:23:56 achernar kernel: Buffer I/O error on device dm-0, logical block 14252047 Feb 29 08:23:56 achernar kernel: lost page write due to I/O error on dm-0 Feb 29 08:23:56 achernar kernel: Buffer I/O error on device dm-0, logical block 14252048 Feb 29 08:23:56 achernar kernel: lost page write due to I/O error on dm-0 Feb 29 08:23:56 achernar kernel: Buffer I/O error on device dm-0, logical block 14252049 Feb 29 08:23:56 achernar kernel: lost page write due to I/O error on dm-0 Feb 29 08:23:56 achernar kernel: Buffer I/O error on device dm-0, logical block 14252050 Feb 29 08:23:56 achernar kernel: lost page write due to I/O error on dm-0 Feb 29 08:23:56 achernar kernel: Buffer I/O error on device dm-0, logical block 14252051 Feb 29 08:23:56 achernar kernel: lost page write due to I/O error on dm-0 Feb 29 08:23:56 achernar kernel: Buffer I/O error on device dm-0, logical block 14252052 Feb 29 08:23:56 achernar kernel: lost page write due to I/O error on dm-0 Feb 29 08:23:56 achernar kernel: Buffer I/O error on device dm-0, logical block 14252053 Feb 29 08:23:56 achernar kernel: lost page write due to I/O error on dm-0 Feb 29 08:23:57 achernar fcoemon: error 111 Connection refused

I could not find any sensible message on the pysical host, neither in /var/log/messages nor in /var/log/libvirt.

We do have an almost identical server, same hardware, same software which does not have this problem.

How could I proceed to better diagnose the cause of the troubles?

Regards,

-- Peter Hopfgartner R3 GIS Srl - GmbH Via Johann Kravogl-Str. 2 I-39012 Meran/Merano (BZ) web : www.r3-gis.com mail : peter.hopfgartner@r3-gis.com phone: +39 0473 494949 fax : +39 0473 069902 ATTENZIONE! Le informazioni contenute nella presente e-mail e nei documenti eventualmente allegati sono confidenziali. La loro diffusione, distribuzione e/o riproduzione da parte di terzi, senza autorizzazione del mittente è vietata e può violare il D. Lgs. 196/2003. In caso di ricezione per errore, Vogliate immediatamente informare il mittente del messaggio e distruggere la e-mail. ACHTUNG! Die in dieser Nachricht oder in den beigelegten Dokumenten beinhalteten Informationen sind streng vertraulich. Ihre Verbreitung und/oder ihre Wiedergabe durch Dritte ist ohne Erlaubnis des Absenders verboten und verstößt gegen das Legislativdekret 196/2003. Sollten Sie diese Mitteilung irrtümlicherweise erhalten haben, bitten wir Sie uns umgehend zu informieren und anschließend die Mitteilung zu vernichten. WARNING! This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorised copying, disclosure or distribution of the material in this e-mail is strictly forbidden and could be against the law (D. Lgs. 196/2003)

Peter Hopfgartner

26 Apr 26 Apr

9:29 a.m.

The problem got slightly better when I upgraded all kernels, on host and guest, so that the "MTBF" went from 3-4 days to approx 50. Still, the problem is not solved, yet. A maybe stupid question: If the kernel in the guest sees an I/O error on sda, could this be a real error on the physical disk, even if there are no notices in the physical hosts log files, or is this more of a software problem?

As the next step, I'll try to update the physical servers firmware.

Any suggestion on this topic is welcome, even more then before.

Reagrds,

Peter

On 02/29/2012 08:53 AM, Peter Hopfgartner wrote:

...

We have a CentOS 6.2 server with KVM. That server hosts 2 virtual machines, both with Centos 6.2, too.

Regularly, one or both of the virtual machines pass to state "pause" without apparent reason. On resume, I do get have messages, like the following in /var/log/messages.

Feb 28 21:50:45 achernar fcoemon: Failed to connect to lldpad Feb 29 08:23:56 achernar kernel: sd 0:0:0:0: [sda] Unhandled error code Feb 29 08:23:56 achernar kernel: sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT Feb 29 08:23:56 achernar kernel: sd 0:0:0:0: [sda] CDB: Write(10): 2a 00 06 db 70 78 00 00 38 00 Feb 29 08:23:56 achernar kernel: end_request: I/O error, dev sda, sector 115044472 Feb 29 08:23:56 achernar kernel: Buffer I/O error on device dm-0, logical block 14252047 Feb 29 08:23:56 achernar kernel: lost page write due to I/O error on dm-0 Feb 29 08:23:56 achernar kernel: Buffer I/O error on device dm-0, logical block 14252048 Feb 29 08:23:56 achernar kernel: lost page write due to I/O error on dm-0 Feb 29 08:23:56 achernar kernel: Buffer I/O error on device dm-0, logical block 14252049 Feb 29 08:23:56 achernar kernel: lost page write due to I/O error on dm-0 Feb 29 08:23:56 achernar kernel: Buffer I/O error on device dm-0, logical block 14252050 Feb 29 08:23:56 achernar kernel: lost page write due to I/O error on dm-0 Feb 29 08:23:56 achernar kernel: Buffer I/O error on device dm-0, logical block 14252051 Feb 29 08:23:56 achernar kernel: lost page write due to I/O error on dm-0 Feb 29 08:23:56 achernar kernel: Buffer I/O error on device dm-0, logical block 14252052 Feb 29 08:23:56 achernar kernel: lost page write due to I/O error on dm-0 Feb 29 08:23:56 achernar kernel: Buffer I/O error on device dm-0, logical block 14252053 Feb 29 08:23:56 achernar kernel: lost page write due to I/O error on dm-0 Feb 29 08:23:57 achernar fcoemon: error 111 Connection refused

I could not find any sensible message on the pysical host, neither in /var/log/messages nor in /var/log/libvirt.

We do have an almost identical server, same hardware, same software which does not have this problem.

How could I proceed to better diagnose the cause of the troubles?

Regards,

-- Peter Hopfgartner web : http://www.r3-gis.com

Benjamin Franz

1:32 p.m.

On 04/26/2012 02:29 AM, Peter Hopfgartner wrote:

...

The problem got slightly better when I upgraded all kernels, on host and guest, so that the "MTBF" went from 3-4 days to approx 50. Still, the problem is not solved, yet. A maybe stupid question: If the kernel in the guest sees an I/O error on sda, could this be a real error on the physical disk, even if there are no notices in the physical hosts log files, or is this more of a software problem?

As the next step, I'll try to update the physical servers firmware.

Any suggestion on this topic is welcome, even more then before.

This could be being caused by failing areas on the underlaying disk drive. Particularly if you are using consumer grade hard drives instead of enterprise drives. The most relevant difference here is that consumer grade drives can try for up to a couple of minutes to read a bad sector and might eventually succeed if the error isn't too egregious while an enterprise drive will just quickly report the sector as unreadable and move on.

I would install smartmontools on the physical server and check the SMART status of the drive after running a 'long' test.

-- Benjamin Franz

Peter Hopfgartner

27 Apr 27 Apr

7:03 a.m.

On 04/26/2012 03:32 PM, Benjamin Franz wrote:

...

On 04/26/2012 02:29 AM, Peter Hopfgartner wrote:

...
The problem got slightly better when I upgraded all kernels, on host and guest, so that the "MTBF" went from 3-4 days to approx 50. Still, the problem is not solved, yet. A maybe stupid question: If the kernel in the guest sees an I/O error on sda, could this be a real error on the physical disk, even if there are no notices in the physical hosts log files, or is this more of a software problem?

As the next step, I'll try to update the physical servers firmware.

Any suggestion on this topic is welcome, even more then before.

This could be being caused by failing areas on the underlaying disk drive. Particularly if you are using consumer grade hard drives instead of enterprise drives. The most relevant difference here is that consumer grade drives can try for up to a couple of minutes to read a bad sector and might eventually succeed if the error isn't too egregious while an enterprise drive will just quickly report the sector as unreadable and move on.

Hallo Benjamin

thanks for your reply.

Isn't it strange, that the log entries are only on the guest VMs, not on the physical server? I'm not able to give an answer on this, due to my inexperience on this topic. Can I go and call the Dell assistance and tell them to handle me 2 new disks, since it is reasonable clear that one of those disks placed in server is flawed? Anyway, the machine is a Dell R410 Poweredge server with a hardware RAID PERC H200 Adapter and 2 600 GB SAS disks in RAID 1. The "twin" to this machine, that we purched together with this one, does not show the same behaviour. Anyway, it has a lighter load.

...

I would install smartmontools on the physical server and check the SMART status of the drive after running a 'long' test.

After some googling I've found how to do this with this RAID controller:

Device: SEAGATE ST3600057SS Version: ES64 Serial number: xxxxxxx Device type: disk Transport protocol: SAS Local Time is: Fri Apr 27 08:52:50 2012 CEST Device supports SMART and is Enabled Temperature Warning Disabled or Not Supported SMART Health Status: OK

Current Drive Temperature: 37 C Drive Trip Temperature: 68 C Elements in grown defect list: 0 Vendor (Seagate) cache information Blocks sent to initiator = 1806551142 Blocks received from initiator = 1325078948 Blocks read from cache and sent to initiator = 281977973 Number of read and write commands whose size <= segment size = 82709392 Number of read and write commands whose size > segment size = 183965 Vendor (Seagate/Hitachi) factory information number of hours powered up = 5526.70 number of minutes until next internal SMART test = 47

Error counter log: Errors Corrected by Total Correction Gigabytes Total ECC rereads/ errors algorithm processed uncorrected fast | delayed rewrites corrected invocations [10^9 bytes] errors read: 77855443 0 0 77855443 77855443 5989.204 0 write: 0 0 0 0 0 66665.246 0 verify: 35799949 0 0 35799949 35799949 3727.548 0

Non-medium error count: 3

SMART Self-test log Num Test Status segment LifeTime LBA_first_err [SK ASC ASQ] Description number (hours) # 1 Background long Completed 16 3 - [- - -] # 2 Background long Completed 16 1 - [- - -] # 3 Background short Completed 16 0 - [- - -]

Long (extended) Self Test duration: 6400 seconds [106.7 minutes]

Device: SEAGATE ST3600057SS Version: ES64 Serial number: xxxxxxx Device type: disk Transport protocol: SAS Local Time is: Fri Apr 27 08:57:10 2012 CEST Device supports SMART and is Enabled Temperature Warning Disabled or Not Supported SMART Health Status: OK

Current Drive Temperature: 36 C Drive Trip Temperature: 68 C Elements in grown defect list: 0 Vendor (Seagate) cache information Blocks sent to initiator = 2858579160 Blocks received from initiator = 163698761 Blocks read from cache and sent to initiator = 3391810210 Number of read and write commands whose size <= segment size = 97415598 Number of read and write commands whose size > segment size = 183976 Vendor (Seagate/Hitachi) factory information number of hours powered up = 5526.82 number of minutes until next internal SMART test = 40

Error counter log: Errors Corrected by Total Correction Gigabytes Total ECC rereads/ errors algorithm processed uncorrected fast | delayed rewrites corrected invocations [10^9 bytes] errors read: 265118506 1 0 265118507 265118507 50649.094 0 write: 0 0 0 0 0 66071.078 0 verify: 19656379 0 0 19656379 19656379 3586.762 0

Non-medium error count: 22

Long (extended) Self Test duration: 6400 seconds [106.7 minutes]

How do I interpret these nunbers? To me, they look quite good.

Thanks,

Peter

-- Peter Hopfgartner web : http://www.r3-gis.com

4553

Age (days ago)

4611

Last active (days ago)

virt@lists.centos.org

5 comments

3 participants

tags (0)

participants (3)

Benjamin Franz
Peter Hopfgartner
Wahyu Darmawan