Why is my load stuck around 3?
top - 01:19:55 up 146 days, 5:53, 2 users, load average: 3.00, 2.95, 2.52 Tasks: 121 total, 1 running, 120 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0% us, 0.3% sy, 0.0% ni, 99.7% id, 0.0% wa, 0.0% hi, 0.0% si Mem: 1928300k total, 1911640k used, 16660k free, 10760k buffers Swap: 2031608k total, 224k used, 2031384k free, 1561196k cached
top - 01:20:22 up 146 days, 5:54, 2 users, load average: 3.06, 2.97, 2.54 Tasks: 121 total, 1 running, 120 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0% us, 0.2% sy, 0.0% ni, 99.8% id, 0.0% wa, 0.0% hi, 0.0% si Mem: 1928300k total, 1911192k used, 17108k free, 10816k buffers Swap: 2031608k total, 224k used, 2031384k free, 1561276k cached
top - 01:21:14 up 146 days, 5:54, 2 users, load average: 3.02, 2.97, 2.56 Tasks: 121 total, 1 running, 120 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0% us, 0.0% sy, 0.0% ni, 100.0% id, 0.0% wa, 0.0% hi, 0.0% si Mem: 1928300k total, 1911448k used, 16852k free, 10912k buffers Swap: 2031608k total, 224k used, 2031384k free, 1561180k cached
$ cat /proc/cpuinfo processor : 0 vendor_id : AuthenticAMD cpu family : 15 model : 75 model name : AMD Athlon(tm) 64 X2 Dual Core Processor 4200+ stepping : 2 cpu MHz : 1800.178 cache size : 512 KB physical id : 0 siblings : 2 core id : 0 cpu cores : 2 fpu : yes fpu_exception : yes cpuid level : 1 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext lm 3dnowext 3dnow pni cx16 bogomips : 3603.04 TLB size : 1088 4K pages clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: ts fid vid ttp [4] [5]
processor : 1 vendor_id : AuthenticAMD cpu family : 15 model : 75 model name : AMD Athlon(tm) 64 X2 Dual Core Processor 4200+ stepping : 2 cpu MHz : 1800.178 cache size : 512 KB physical id : 0 siblings : 2 core id : 1 cpu cores : 2 fpu : yes fpu_exception : yes cpuid level : 1 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext lm 3dnowext 3dnow pni cx16 bogomips : 3603.04 TLB size : 1088 4K pages clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: ts fid vid ttp [4] [5]
$ cat /proc/meminfo MemTotal: 1928300 kB MemFree: 17292 kB Buffers: 10984 kB Cached: 1561244 kB SwapCached: 0 kB Active: 1254600 kB Inactive: 410104 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 1928300 kB LowFree: 17292 kB SwapTotal: 2031608 kB SwapFree: 2031384 kB Dirty: 96 kB Writeback: 0 kB Mapped: 1222324 kB Slab: 151796 kB CommitLimit: 2995756 kB Committed_AS: 568768 kB PageTables: 7200 kB VmallocTotal: 536870911 kB VmallocUsed: 264072 kB VmallocChunk: 536606367 kB HugePages_Total: 0 HugePages_Free: 0 Hugepagesize: 2048 kB
-- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- - - - Jason Pyeron PD Inc. http://www.pdinc.us - - Principal Consultant 10 West 24th Street #100 - - +1 (443) 269-1555 x333 Baltimore, Maryland 21218 - - - -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- This message is copyright PD Inc, subject to license 20080407P00.
On 2009-02-06 07:24, Jason Pyeron wrote:
Why is my load stuck around 3?
top - 01:19:55 up 146 days, 5:53, 2 users, load average: 3.00, 2.95, 2.52 Tasks: 121 total, 1 running, 120 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0% us, 0.3% sy, 0.0% ni, 99.7% id, 0.0% wa, 0.0% hi, 0.0% si Mem: 1928300k total, 1911640k used, 16660k free, 10760k buffers Swap: 2031608k total, 224k used, 2031384k free, 1561196k cached
Do you have any Java process running?
On 2009-02-06 10:11, Paul Bijnens wrote:
On 2009-02-06 07:24, Jason Pyeron wrote:
Why is my load stuck around 3?
top - 01:19:55 up 146 days, 5:53, 2 users, load average: 3.00, 2.95, 2.52 Tasks: 121 total, 1 running, 120 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0% us, 0.3% sy, 0.0% ni, 99.7% id, 0.0% wa, 0.0% hi, 0.0% si Mem: 1928300k total, 1911640k used, 16660k free, 10760k buffers Swap: 2031608k total, 224k used, 2031384k free, 1561196k cached
Do you have any Java process running?
And following up to myself... I see this phenonomon on most hosts that run a java process but I'm still lacking some decent explanation why java would exhibit such behaviour. Any experts?
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Paul Bijnens Sent: Friday, February 06, 2009 9:25 To: CentOS mailing list Subject: Re: [CentOS] Deciphering top's data
On 2009-02-06 10:11, Paul Bijnens wrote:
On 2009-02-06 07:24, Jason Pyeron wrote:
Why is my load stuck around 3?
top - 01:19:55 up 146 days, 5:53, 2 users, load
average: 3.00, 2.95, 2.52
Tasks: 121 total, 1 running, 120 sleeping, 0 stopped,
0 zombie
Cpu(s): 0.0% us, 0.3% sy, 0.0% ni, 99.7% id, 0.0% wa,
0.0% hi, 0.0% si
Mem: 1928300k total, 1911640k used, 16660k free,
10760k buffers
Swap: 2031608k total, 224k used, 2031384k free,
1561196k cached
Do you have any Java process running?
And following up to myself... I see this phenonomon on most hosts that run a java process but I'm still lacking some decent explanation why java would exhibit such behaviour. Any experts?
No Java processes running. Now that I have more info and some sleep, I most recently shut down 3 VMWare instances, they were listed as zombies so I ignored them. This morning I cam back to see:
top - 09:48:56 up 146 days, 14:22, 2 users, load average: 3.00, 3.00, 3.00 Tasks: 105 total, 1 running, 101 sleeping, 0 stopped, 3 zombie Cpu(s): 0.0% us, 0.0% sy, 0.0% ni, 99.7% id, 0.0% wa, 0.2% hi, 0.0% si Mem: 1928300k total, 1911376k used, 16924k free, 182208k buffers Swap: 2031608k total, 224k used, 2031384k free, 1139440k cached
So I issued a reboot.
It hung on stoping iptables.
I had to power cycle the machine.
Machine details: Linux ------------- 2.6.9-55.0.2.ELsmp #1 SMP Tue Jun 26 14:14:47 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux CentOS release 4.6 (Final)
-- Paul Bijnens, Xplanation Technology Services Tel +32 16 397.525 Interleuvenlaan 86, B-3001 Leuven, BELGIUM Fax +32 16 397.552
- I think I've got the hang of it now: exit, ^D, ^C, ^, ^Z,
^Q, ^^, *
- quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye,
/bye, ~., *
- stop, end, ^]c, +++ ATH, disconnect, halt, abort,
hangup, KJOB, *
- ^X^X, :D::D, kill -9 1, kill -1 $$, shutdown, init 0,
Alt-F4, *
- Alt-f-e, Ctrl-Alt-Del, Alt-SysRq-reisub, Stop-A,
AltGr-NumLock, ... *
- ... "Are you sure?" ... YES ... Phew ... I'm out *
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
-- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- - - - Jason Pyeron PD Inc. http://www.pdinc.us - - Principal Consultant 10 West 24th Street #100 - - +1 (443) 269-1555 x333 Baltimore, Maryland 21218 - - - -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- This message is copyright PD Inc, subject to license 20080407P00.
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Jason Pyeron Sent: Friday, February 06, 2009 10:13 To: 'CentOS mailing list' Subject: Re: [CentOS] Deciphering top's data
So I issued a reboot.
It hung on stoping iptables.
I had to power cycle the machine.
Machine details: Linux ------------- 2.6.9-55.0.2.ELsmp #1 SMP Tue Jun 26 14:14:47 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux CentOS release 4.6 (Final)
Here we go again.
UID PID PPID C STIME TTY STAT TIME CMD root 4066 1 4 10:01 ? Ds 1:13 /usr/lib/vmware/bin/vmware-vmx -C /var/lib/vmware/VMs/proxy/RHEL.vmx
top - 10:33:59 up 33 min, 2 users, load average: 1.06, 0.67, 0.40 Tasks: 118 total, 1 running, 117 sleeping, 0 stopped, 0 zombie Cpu(s): 0.5% us, 2.0% sy, 0.0% ni, 97.3% id, 0.0% wa, 0.0% hi, 0.2% si Mem: 1928300k total, 1318688k used, 609612k free, 47648k buffers Swap: 2031608k total, 0k used, 2031608k free, 1046132k cached
Ug. This production system.... They are going to have my head.
-- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- - - - Jason Pyeron PD Inc. http://www.pdinc.us - - Principal Consultant 10 West 24th Street #100 - - +1 (443) 269-1555 x333 Baltimore, Maryland 21218 - - - -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- This message is copyright PD Inc, subject to license 20080407P00.
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Jason Pyeron Sent: Friday, February 06, 2009 10:40 To: 'CentOS mailing list' Subject: Re: [CentOS] Deciphering top's data
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Jason Pyeron Sent: Friday, February 06, 2009 10:13 To: 'CentOS mailing list' Subject: Re: [CentOS] Deciphering top's data
So I issued a reboot.
It hung on stoping iptables.
I had to power cycle the machine.
Machine details: Linux ------------- 2.6.9-55.0.2.ELsmp #1 SMP Tue Jun 26 14:14:47 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux CentOS release 4.6 (Final)
Here we go again.
UID PID PPID C STIME TTY STAT TIME CMD root 4066 1 4 10:01 ? Ds 1:13 /usr/lib/vmware/bin/vmware-vmx -C /var/lib/vmware/VMs/proxy/RHEL.vmx
top - 10:33:59 up 33 min, 2 users, load average: 1.06, 0.67, 0.40 Tasks: 118 total, 1 running, 117 sleeping, 0 stopped, 0 zombie Cpu(s): 0.5% us, 2.0% sy, 0.0% ni, 97.3% id, 0.0% wa, 0.0% hi, 0.2% si Mem: 1928300k total, 1318688k used, 609612k free, 47648k buffers Swap: 2031608k total, 0k used, 2031608k free, 1046132k cached
Ug. This production system.... They are going to have my head.
$ lsof -p 4066 COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME vmware-vm 4066 root cwd DIR 253,0 4096 14483669 /var/lib/vmware/Virtual Machines/proxy vmware-vm 4066 root rtd DIR 253,0 4096 2 / vmware-vm 4066 root txt REG 253,0 4385956 4096669 /usr/lib/vmware/bin/vmware-vmx vmware-vm 4066 root mem REG 253,0 112168 17826955 /lib/ld-2.3.4.so vmware-vm 4066 root mem REG 253,0 1529120 17826956 /lib/tls/libc-2.3.4.so vmware-vm 4066 root mem REG 253,0 16732 17826957 /lib/libdl-2.3.4.so vmware-vm 4066 root mem REG 253,0 63624 3909886 /usr/lib/libz.so.1.2.1.2 vmware-vm 4066 root mem REG 253,0 213772 17826958 /lib/tls/libm-2.3.4.so vmware-vm 4066 root mem REG 253,0 107800 17828754 /lib/tls/libpthread-2.3.4.so vmware-vm 4066 root mem REG 253,0 53476 3912514 /usr/X11R6/lib/libXext.so.6.4 vmware-vm 4066 root mem REG 253,0 32720 3912517 /usr/X11R6/lib/libSM.so.6.0 vmware-vm 4066 root mem REG 253,0 86520 3912516 /usr/X11R6/lib/libICE.so.6.3 vmware-vm 4066 root mem REG 253,0 326684 3912518 /usr/X11R6/lib/libXt.so.6.0 vmware-vm 4066 root mem REG 253,0 913020 3905116 /usr/X11R6/lib/libX11.so.6.2 vmware-vm 4066 root mem REG 253,0 29668 3912519 /usr/X11R6/lib/libXrender.so.1.2.2 vmware-vm 4066 root mem REG 253,0 18596 3912515 /usr/X11R6/lib/libXtst.so.6.1 vmware-vm 4066 root mem REG 253,0 38624 17825931 /lib/libgcc_s-3.4.6-20060404.so.1 vmware-vm 4066 root mem CHR 1,1 526 /dev/mem vmware-vm 4066 root mem REG 253,0 79376 17825849 /lib/libresolv-2.3.4.so vmware-vm 4066 root mem REG 253,0 22516 17825834 /lib/libnss_dns-2.3.4.so vmware-vm 4066 root mem REG 253,0 47404 17825837 /lib/libnss_files-2.3.4.so vmware-vm 4066 root mem REG 253,0 189908 4128894 /usr/lib/vmware/lib/libssl.so.0.9.7/libssl.so.0.9.7 vmware-vm 4066 root mem REG 253,0 1022860 4128781 /usr/lib/vmware/lib/libcrypto.so.0.9.7/libcrypto.so.0.9.7 vmware-vm 4066 root mem REG 253,0 21544 3932925 /usr/lib/gconv/gconv-modules.cache vmware-vm 4066 root 0u CHR 1,3 2335 /dev/null vmware-vm 4066 root 1w FIFO 0,7 8833 pipe vmware-vm 4066 root 2w FIFO 0,7 8833 pipe vmware-vm 4066 root 3r FIFO 0,7 8848 pipe vmware-vm 4066 root 4w FIFO 0,7 8848 pipe vmware-vm 4066 root 5r FIFO 0,7 8849 pipe vmware-vm 4066 root 6w FIFO 0,7 8849 pipe vmware-vm 4066 root 7r FIFO 0,7 8854 pipe vmware-vm 4066 root 8w FIFO 0,7 8854 pipe vmware-vm 4066 root 9r FIFO 0,7 8855 pipe vmware-vm 4066 root 10w FIFO 0,7 8855 pipe vmware-vm 4066 root 11r FIFO 0,7 8856 pipe vmware-vm 4066 root 12w FIFO 0,7 8856 pipe vmware-vm 4066 root 13r FIFO 0,7 8857 pipe vmware-vm 4066 root 14w FIFO 0,7 8857 pipe vmware-vm 4066 root 15r FIFO 0,7 8858 pipe vmware-vm 4066 root 16w FIFO 0,7 8858 pipe vmware-vm 4066 root 17u REG 253,0 20542 17514528 /var/lib/vmware/Virtual Machines/proxy/vmware.log vmware-vm 4066 root 18r FIFO 0,7 8865 pipe vmware-vm 4066 root 19w FIFO 0,7 8865 pipe vmware-vm 4066 root 20r FIFO 0,7 8866 pipe vmware-vm 4066 root 21w FIFO 0,7 8866 pipe vmware-vm 4066 root 22r FIFO 0,7 8873 pipe vmware-vm 4066 root 23w FIFO 0,7 8873 pipe vmware-vm 4066 root 24r FIFO 0,7 8875 pipe vmware-vm 4066 root 25w FIFO 0,7 8875 pipe vmware-vm 4066 root 26u unix 0x00000100735bf080 8917 /var/run/vmware/root/4066/vmx-vmdb-fd vmware-vm 4066 root 27u unix 0x000001007359e6c0 8920 socket vmware-vm 4066 root 28r FIFO 0,7 35880 pipe vmware-vm 4066 root 29r FIFO 0,7 35873 pipe vmware-vm 4066 root 30r FIFO 0,7 35882 pipe vmware-vm 4066 root 31w FIFO 0,7 35882 pipe vmware-vm 4066 root 32r FIFO 0,7 8933 pipe vmware-vm 4066 root 34u unix 0x0000010073616640 8883 /var/run/vmware/root/4066/servercontrol-fd vmware-vm 4066 root 35u unix 0x0000010073616c40 8885 /var/run/vmware/root/4066/control-fd vmware-vm 4066 root 36u unix 0x00000100735bf980 8887 /var/run/vmware/root/4066/ui-fd vmware-vm 4066 root 37u CHR 10,165 7872 /dev/vmmon vmware-vm 4066 root 38r FIFO 0,7 8889 pipe vmware-vm 4066 root 39w FIFO 0,7 8889 pipe vmware-vm 4066 root 40r FIFO 0,7 8889 pipe vmware-vm 4066 root 41r FIFO 0,7 9050 pipe vmware-vm 4066 root 42w FIFO 0,7 9050 pipe vmware-vm 4066 root 43u unix 0x00000100735bf380 8891 /var/run/vmware/root/4066/mks-fd vmware-vm 4066 root 45u CHR 10,165 7872 /dev/vmmon vmware-vm 4066 root 46w FIFO 0,7 8933 pipe vmware-vm 4066 root 47r FIFO 0,7 8934 pipe vmware-vm 4066 root 48w FIFO 0,7 8934 pipe vmware-vm 4066 root 49r FIFO 0,7 8935 pipe vmware-vm 4066 root 50w FIFO 0,7 8935 pipe vmware-vm 4066 root 51r FIFO 0,7 8936 pipe vmware-vm 4066 root 52w FIFO 0,7 8936 pipe vmware-vm 4066 root 53u unix 0x0000010073585680 9057 /var/run/vmware/root/4066/remoteDevice-fd vmware-vm 4066 root 166u unix 0x00000100735bfc80 8949 socket vmware-vm 4066 root 181u CHR 10,165 7872 /dev/vmmon vmware-vm 4066 root 182u CHR 10,165 7872 /dev/vmmon vmware-vm 4066 root 185r FIFO 0,7 35880 pipe vmware-vm 4066 root 186w FIFO 0,7 35880 pipe
Output from sysrq t: (I don't know how to read this.)
SysRq : Show State
sibling task PC pid father child younger older init S 000000000000000b 0 1 0 2 (NOTLB) 0000010076451d78 0000000000000002 0000010076451da8 ffffffff801d4bf5 ffffffffffffffff ffffffff00000000 000000d000000000 0000000100000246 00000100764237f0 00000000000007e1 Call Trace:<ffffffff801d4bf5>{selinux_inode_permission+0} <ffffffff80140090>{__mod_timer+293} <ffffffff8030cbbb>{schedule_timeout+367} <ffffffff80140aba>{process_timeout+0} <ffffffff8018c4c3>{do_select+939} <ffffffff8018c05d>{__pollwait+0} <ffffffff8018c842>{sys_select+820} <ffffffff8011026a>{system_call+126}
..... SNIP ....
acpid S 000001007325bac0 0 4038 1 4066 4028 (NOTLB) 000001006b4bdd78 0000000000000006 000001006c32b300 000000000000007a 0000010072c374c0 000000006b4bdd30 0000010001020a80 000000000000000e 000001006ba1b030 000000000001f24e Call Trace:<ffffffff8030cb2c>{schedule_timeout+224} <ffffffff8018c4c3>{do_select+939} <ffffffff8018c05d>{__pollwait+0} <ffffffff8018c842>{sys_select+820} <ffffffff801940e8>{dnotify_parent+34} <ffffffff8011026a>{system_call+126}
vmware-vmx D 0000000000000000 0 4066 1 4090 4038 (NOTLB) 000001006b6079f8 0000000000000006 ffffffffffffffff ffffffff00000069 0000000000000002 00000000a02349c6 0000010001028a80 00000001a023ba48 000001007300e030 000000000016a770 Call Trace:<ffffffffa022bc9c>{:vmmon:LinuxDriverError+32} <ffffffff8030c2dd>{wait_for_completion+167} <ffffffff801341cc>{default_wake_function+0} <ffffffff80142abf>{group_send_sig_info+108} <ffffffff801341cc>{default_wake_function+0} <ffffffff80142b4f>{kill_proc_info+71} <ffffffffa022e739>{:vmmon:HostIF_SetFastClockRate+353} <ffffffffa022ff29>{:vmmon:Vmx86_SetHostClockRate+181} <ffffffffa022f530>{:vmmon:Vmx86FreeAllVMResources+16} <ffffffffa022fc9a>{:vmmon:Vmx86_ReleaseVM+62} <ffffffffa022b20d>{:vmmon:LinuxDriver_Ioctl+249} <ffffffff801628b0>{slab_destroy+124} <ffffffff801629f6>{free_block+285} <ffffffffa006e726>{:jbd:do_get_write_access+1529} <ffffffffa006e726>{:jbd:do_get_write_access+1529} <ffffffffa006e726>{:jbd:do_get_write_access+1529} <ffffffff8017e310>{__getblk+42} <ffffffff8015ab70>{find_get_page+65} <ffffffffa006de73>{:jbd:__journal_file_buffer+384} <ffffffffa006e726>{:jbd:do_get_write_access+1529} <ffffffffa006de73>{:jbd:__journal_file_buffer+384} <ffffffff80134273>{__wake_up+54} <ffffffffa006dad8>{:jbd:journal_stop+574} <ffffffff8018fcd1>{dput+56} <ffffffff8017aca1>{__fput+253} <ffffffff8030cd57>{__down_read+52} <ffffffffa022a059>{:vmmon:LinuxDriver_Ioctl32_Handler+57} <ffffffff8019f7e4>{compat_sys_ioctl+235} <ffffffff80126257>{cstar_do_call+27}
cupsd S 0000000000000004 0 4090 1 4167 4066 (NOTLB) 000001006afe1d78 0000000000000006 ffffffff803d7680 0000000000000000 00000100736db180 000001006afe1d30 000000d000000000 0000000000000246 000001006af71030 000000000000052b Call Trace:<ffffffff80140090>{__mod_timer+293} <ffffffff8030cbbb>{schedule_timeout+367} <ffffffff80140aba>{process_timeout+0} <ffffffff8018c4c3>{do_select+939} <ffffffff8018c05d>{__pollwait+0} <ffffffff8018c842>{sys_select+820} <ffffffff8018fd56>{dput+189} <ffffffff8011026a>{system_call+126}
..... SNIP ....
$ rpm -qa | grep -i vmw VMware-server-1.0.2-39867
-- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- - - - Jason Pyeron PD Inc. http://www.pdinc.us - - Principal Consultant 10 West 24th Street #100 - - +1 (443) 269-1555 x333 Baltimore, Maryland 21218 - - - -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- This message is copyright PD Inc, subject to license 20080407P00.
On Fri, Feb 06, 2009 at 10:58:29AM -0500, Jason Pyeron wrote:
..... SNIP ....
$ rpm -qa | grep -i vmw VMware-server-1.0.2-39867
definetely worth an update [tru@sillage2 ~]$ grep -i vmw /var/log/rpmpkgs VMware-server-1.0.8-126538.i386.rpm
Tru
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Tru Huynh Sent: Friday, February 06, 2009 11:23 To: CentOS mailing list Subject: Re: [CentOS] VM Ware guest shutdown hangs with high loadin uninterruptible sleep was RE: Deciphering top's data
On Fri, Feb 06, 2009 at 10:58:29AM -0500, Jason Pyeron wrote:
..... SNIP ....
$ rpm -qa | grep -i vmw VMware-server-1.0.2-39867
definetely worth an update [tru@sillage2 ~]$ grep -i vmw /var/log/rpmpkgs VMware-server-1.0.8-126538.i386.rpm
$ ps -f 8112 UID PID PPID C STIME TTY STAT TIME CMD root 8112 1 17 11:49 ? Ds 0:51 /usr/lib/vmware/bin/vmware-vmx -C /var/lib/vmware/VMs/proxy/RHEL.vmx
$ uname -a Linux ------------ 2.6.9-78.0.13.ELsmp #1 SMP Wed Jan 14 15:55:36 EST 2009 x86_64 x86_64 x86_64 GNU/Linux
$ rpm -qa | grep -i vmw VMware-server-1.0.8-126538
top - 11:53:14 up 10 min, 1 user, load average: 0.81, 0.47, 0.29 Tasks: 117 total, 1 running, 116 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0% us, 0.2% sy, 0.0% ni, 99.8% id, 0.0% wa, 0.0% hi, 0.0% si Mem: 1928236k total, 846208k used, 1082028k free, 41000k buffers Swap: 2031608k total, 0k used, 2031608k free, 568960k cached
Tru
Tru Huynh (mirrors, CentOS-3 i386/x86_64 Package Maintenance) http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xBEFA581B
Whats next?
-- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- - - - Jason Pyeron PD Inc. http://www.pdinc.us - - Principal Consultant 10 West 24th Street #100 - - +1 (443) 269-1555 x333 Baltimore, Maryland 21218 - - - -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- This message is copyright PD Inc, subject to license 20080407P00.
On Fri, Feb 06, 2009 at 11:57:15AM -0500, Jason Pyeron wrote:
$ ps -f 8112 UID PID PPID C STIME TTY STAT TIME CMD root 8112 1 17 11:49 ? Ds 0:51 /usr/lib/vmware/bin/vmware-vmx -C /var/lib/vmware/VMs/proxy/RHEL.vmx
have you applied the recommended vmware boot flags for your vm?
other ideas but this is becoming off topic for CentOS since it's seems to be a vmware server issue for a RHEL guest.. - moved vmware guest and a vmware console question left unanswered. - damaged vmdk - anything on the vmware logs? ...
Tru
On Fri, 2009-02-06 at 18:17 +0100, Tru Huynh wrote:
On Fri, Feb 06, 2009 at 11:57:15AM -0500, Jason Pyeron wrote:
$ ps -f 8112 UID PID PPID C STIME TTY STAT TIME CMD root 8112 1 17 11:49 ? Ds 0:51 /usr/lib/vmware/bin/vmware-vmx -C /var/lib/vmware/VMs/proxy/RHEL.vmx
have you applied the recommended vmware boot flags for your vm?
other ideas but this is becoming off topic for CentOS since it's seems to be a vmware server issue for a RHEL guest..
- moved vmware guest and a vmware console question left unanswered.
- damaged vmdk
- anything on the vmware logs?
...
Tru
------ And the only other reason I can think of why it is happening is your possibly running Multipath but that Issue is supposed to be fixed in version 4.5? Rh released an update to Multipathd to fix this. Should have trickled down to centos as well.
JohnSanley
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of JohnS Sent: Friday, February 06, 2009 12:52 To: CentOS mailing list Subject: Re: [CentOS] VM Ware guest shutdown hangs withhigh loadin uninterruptible sleep was RE: Deciphering top's data
On Fri, 2009-02-06 at 18:17 +0100, Tru Huynh wrote:
On Fri, Feb 06, 2009 at 11:57:15AM -0500, Jason Pyeron wrote:
$ ps -f 8112 UID PID PPID C STIME TTY STAT TIME CMD root 8112 1 17 11:49 ? Ds 0:51 /usr/lib/vmware/bin/vmware-vmx -C
/var/lib/vmware/VMs/proxy/RHEL.vmx
have you applied the recommended vmware boot flags for your vm?
other ideas but this is becoming off topic for CentOS since
it's seems
to be a vmware server issue for a RHEL guest..
- moved vmware guest and a vmware console question left unanswered.
- damaged vmdk
- anything on the vmware logs?
...
Tru
And the only other reason I can think of why it is happening is your possibly running Multipath but that Issue is supposed to be fixed in version 4.5? Rh released an update to Multipathd to fix this. Should have trickled down to centos as well.
How do I check this?
JohnSanley
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
-- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- - - - Jason Pyeron PD Inc. http://www.pdinc.us - - Principal Consultant 10 West 24th Street #100 - - +1 (443) 269-1555 x333 Baltimore, Maryland 21218 - - - -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- This message is copyright PD Inc, subject to license 20080407P00.
On Fri, 2009-02-06 at 14:46 -0500, Jason Pyeron wrote:
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of JohnS Sent: Friday, February 06, 2009 12:52 To: CentOS mailing list Subject: Re: [CentOS] VM Ware guest shutdown hangs withhigh loadin uninterruptible sleep was RE: Deciphering top's data
On Fri, 2009-02-06 at 18:17 +0100, Tru Huynh wrote:
On Fri, Feb 06, 2009 at 11:57:15AM -0500, Jason Pyeron wrote:
$ ps -f 8112 UID PID PPID C STIME TTY STAT TIME CMD root 8112 1 17 11:49 ? Ds 0:51 /usr/lib/vmware/bin/vmware-vmx -C
/var/lib/vmware/VMs/proxy/RHEL.vmx
have you applied the recommended vmware boot flags for your vm?
other ideas but this is becoming off topic for CentOS since
it's seems
to be a vmware server issue for a RHEL guest..
- moved vmware guest and a vmware console question left unanswered.
- damaged vmdk
- anything on the vmware logs?
...
Tru
And the only other reason I can think of why it is happening is your possibly running Multipath but that Issue is supposed to be fixed in version 4.5? Rh released an update to Multipathd to fix this. Should have trickled down to centos as well.
How do I check this?
-------- rpm -qa | grep mapper Will show you your current version. The fix started in 4.5 and for version 5 release 1 of RHEL. That being said does not mean it is in CentOS. Centos 4.7 should have it as below. device-mapper-multipath-0.4.5-31.el4.i386.rpm device-mapper-1.02.25-2.el4.i386.rpm They are up to date with ftp.redhat!
What you are describing only was supposed to happen on ESX and not VMWare Server so you may be totally into a different problem. LIKE, running out ram? A 64 bit system takes more ram to actually allocate ram than a 32 bit system. Check /var/log/vmwarev/mware-serverd.log ? Check your disks for bad sectors (local disks). Memory Errors. Bad firmware on the Raid Controller or controller going bad under heavy loads. Loading images to boot over the network? Network Latency? You need some kind of log output from the raid controller. I take it it is a pretty high end machine with all those images so you should have some kind of monitoring process like Dell OMSA on it to get that info. Top and vmstat is not going to show you what the problem is. That's fantasy world. All that shows is memory allocation, cpu usage etc etc. You need some log files to ponder over and look at possible hardware. I am just assuming it is a hardware prob since you updated and because the multipath problem is not supposed to affect vmw-server.
Ohh how you check that? It is called Red Hat Bugzilla and keeping up with the technology you support like you should be doing.
JohnStanley
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Tru Huynh Sent: Friday, February 06, 2009 12:17 To: CentOS mailing list Subject: Re: [CentOS] VM Ware guest shutdown hangs with highloadin uninterruptible sleep was RE: Deciphering top's data
On Fri, Feb 06, 2009 at 11:57:15AM -0500, Jason Pyeron wrote:
$ ps -f 8112 UID PID PPID C STIME TTY STAT TIME CMD root 8112 1 17 11:49 ? Ds 0:51 /usr/lib/vmware/bin/vmware-vmx -C /var/lib/vmware/VMs/proxy/RHEL.vmx
have you applied the recommended vmware boot flags for your vm?
other ideas but this is becoming off topic for CentOS since it's seems to be a vmware server issue for a RHEL guest..
Windows 2000 pro, server, centos 3, 4 i386 guests, and even the linux boot cd manifest the same issues.
- moved vmware guest and a vmware console question left unanswered.
Since the problems with the production box this morning, the exchange server vm has been relocated, it has no problems on the new host.
- damaged vmdk
Does not appear to be so
- anything on the vmware logs?
Nothing useful or unusual that I see.
...
Tru
Tru Huynh (mirrors, CentOS-3 i386/x86_64 Package Maintenance) http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xBEFA581B
-- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- - - - Jason Pyeron PD Inc. http://www.pdinc.us - - Principal Consultant 10 West 24th Street #100 - - +1 (443) 269-1555 x333 Baltimore, Maryland 21218 - - - -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- This message is copyright PD Inc, subject to license 20080407P00.
On Fri, Feb 06, 2009 at 10:13:02AM -0500, Jason Pyeron wrote:
...
I had to power cycle the machine.
Machine details: Linux ------------- 2.6.9-55.0.2.ELsmp #1 SMP Tue Jun 26 14:14:47 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux CentOS release 4.6 (Final)
current is 4.7 with kernel-smp-2.6.9-78.0.13.EL.x86_64
You might consider upgrading and rebooting...
Tru
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Tru Huynh Sent: Friday, February 06, 2009 11:05 AM To: CentOS mailing list Subject: Re: [CentOS] Deciphering top's data
On Fri, Feb 06, 2009 at 10:13:02AM -0500, Jason Pyeron wrote:
...
I had to power cycle the machine.
Machine details: Linux ------------- 2.6.9-55.0.2.ELsmp #1 SMP Tue Jun 26
14:14:47 EDT 2007
x86_64 x86_64 x86_64 GNU/Linux CentOS release 4.6 (Final)
current is 4.7 with kernel-smp-2.6.9-78.0.13.EL.x86_64
You might consider upgrading and rebooting...
Tru
---
Upgrade VmWare, also it is antique in age with numerous problems. Build the vmware kernel modules against the newer updated SMP Kernel. You will need the other *kernel* rpms as well.
JohnStanley
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of John Sent: Friday, February 06, 2009 11:39 To: 'CentOS mailing list' Subject: Re: [CentOS] Deciphering top's data
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Tru Huynh Sent: Friday, February 06, 2009 11:05 AM To: CentOS mailing list Subject: Re: [CentOS] Deciphering top's data
On Fri, Feb 06, 2009 at 10:13:02AM -0500, Jason Pyeron wrote:
...
I had to power cycle the machine.
Machine details: Linux ------------- 2.6.9-55.0.2.ELsmp #1 SMP Tue Jun 26
14:14:47 EDT 2007
x86_64 x86_64 x86_64 GNU/Linux CentOS release 4.6 (Final)
current is 4.7 with kernel-smp-2.6.9-78.0.13.EL.x86_64
You might consider upgrading and rebooting...
Tru
Upgrade VmWare, also it is antique in age with numerous problems. Build the vmware kernel modules against the newer updated SMP Kernel. You will need the other *kernel* rpms as well.
In progress...
JohnStanley
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
-- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- - - - Jason Pyeron PD Inc. http://www.pdinc.us - - Principal Consultant 10 West 24th Street #100 - - +1 (443) 269-1555 x333 Baltimore, Maryland 21218 - - - -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- This message is copyright PD Inc, subject to license 20080407P00.
-
Jason Pyeron -
John -
JohnS
-
Paul Bijnens -
Tru Huynh