I have a server holding a 2.5-4.0 load level consistently, where should I start debugging it?
top - 13:32:01 up 167 days, 5:45, 6 users, load average: 2.87, 1.83, 1.51 Tasks: 111 total, 1 running, 110 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0% us, 0.0% sy, 0.2% ni, 99.8% id, 0.0% wa, 0.0% hi, 0.0% si Mem: 895728k total, 871996k used, 23732k free, 190612k buffers Swap: 1802232k total, 216k used, 1802016k free, 63776k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1 root 16 0 4756 556 460 S 0 0.1 0:35.34 init 2 root RT 0 0 0 0 S 0 0.0 0:02.83 migration/0 3 root 34 19 0 0 0 S 0 0.0 0:27.27 ksoftirqd/0 4 root RT 0 0 0 0 S 0 0.0 0:05.03 migration/1 5 root 34 19 0 0 0 S 0 0.0 0:22.88 ksoftirqd/1 6 root 5 -10 0 0 0 S 0 0.0 0:03.15 events/0 7 root 5 -10 0 0 0 S 0 0.0 0:01.98 events/1 8 root 5 -10 0 0 0 S 0 0.0 0:00.00 khelper 9 root 5 -10 0 0 0 S 0 0.0 0:00.00 kblockd/0 10 root 5 -10 0 0 0 S 0 0.0 0:00.00 kblockd/1 11 root 15 0 0 0 0 S 0 0.0 0:00.00 khubd 66 root 15 0 0 0 0 S 0 0.0 0:16.14 pdflush 67 root 15 0 0 0 0 S 0 0.0 2:30.44 kswapd0 68 root 15 -10 0 0 0 S 0 0.0 0:00.00 aio/0 69 root 5 -10 0 0 0 S 0 0.0 0:00.00 aio/1 213 root 22 0 0 0 0 S 0 0.0 0:00.00 kseriod 450 root 5 -10 0 0 0 S 0 0.0 0:00.00 ata/0 451 root 5 -10 0 0 0 S 0 0.0 0:00.00 ata/1 452 root 6 -10 0 0 0 S 0 0.0 0:00.00 ata_aux 456 root 15 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_0 457 root 15 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_1 465 root 15 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_2 466 root 15 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_3 494 root 15 0 0 0 0 S 0 0.0 4:03.36 kjournald 542 root 16 0 24192 596 416 S 0 0.1 0:36.55 ypbind 595 root 15 0 0 0 0 S 0 0.0 0:00.00 rpciod 596 root 18 0 0 0 0 S 0 0.0 0:00.00 lockd 775 apache 15 0 185m 40m 3628 S 0 4.6 0:00.59 httpd 1832 root 5 -10 0 0 0 S 0 0.0 0:00.00 kedac 1904 root 16 0 10828 1072 856 S 0 0.1 0:00.00 pppd 1973 root 6 -10 0 0 0 S 0 0.0 0:00.00 kauditd 1998 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_4 1999 root 15 0 0 0 0 S 0 0.0 3:10.72 usb-storage 2176 root 15 0 0 0 0 S 0 0.0 0:00.01 kjournald 3070 root 15 0 3632 552 440 S 0 0.1 1:08.11 syslogd 3074 root 15 0 2540 372 296 S 0 0.0 0:00.04 klogd 3087 root 16 0 2552 300 200 S 0 0.0 0:30.06 irqbalance 3155 root 15 0 2916 488 256 S 0 0.1 0:00.97 smartd 3215 root 15 0 8716 776 612 S 0 0.1 0:00.00 xinetd 3293 root 16 0 40836 2672 984 S 0 0.3 1:07.23 sendmail 3303 smmsp 16 0 33504 2108 812 S 0 0.2 0:00.99 sendmail 3313 root 16 0 4180 324 244 S 0 0.0 0:00.00 gpm 3323 root 16 0 175m 35m 9064 S 0 4.0 0:13.24 httpd 3332 root 16 0 57092 976 560 S 0 0.1 0:50.01 crond 3353 xfs 16 0 10020 1472 756 S 0 0.2 0:00.17 xfs
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- - - - Jason Pyeron PD Inc. http://www.pdinc.us - - Principal Consultant 10 West 24th Street #100 - - +1 (443) 269-1555 x333 Baltimore, Maryland 21218 - - - -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, purge the message from your system and notify the sender immediately. Any other use of the email by you is prohibited.
Perhaps you could run iotop find out if there is a process chewing up your hard disks. You aren't just limited to top, you can also use vmstat and iostat in this situation.
Jason Pyeron wrote:
I have a server holding a 2.5-4.0 load level consistently, where should I start debugging it?
top - 13:32:01 up 167 days, 5:45, 6 users, load average: 2.87, 1.83, 1.51 Tasks: 111 total, 1 running, 110 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0% us, 0.0% sy, 0.2% ni, 99.8% id, 0.0% wa, 0.0% hi, 0.0% si Mem: 895728k total, 871996k used, 23732k free, 190612k buffers Swap: 1802232k total, 216k used, 1802016k free, 63776k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1 root 16 0 4756 556 460 S 0 0.1 0:35.34 init 2 root RT 0 0 0 0 S 0 0.0 0:02.83 migration/0 3 root 34 19 0 0 0 S 0 0.0 0:27.27 ksoftirqd/0 4 root RT 0 0 0 0 S 0 0.0 0:05.03 migration/1 5 root 34 19 0 0 0 S 0 0.0 0:22.88 ksoftirqd/1 6 root 5 -10 0 0 0 S 0 0.0 0:03.15 events/0 7 root 5 -10 0 0 0 S 0 0.0 0:01.98 events/1 8 root 5 -10 0 0 0 S 0 0.0 0:00.00 khelper 9 root 5 -10 0 0 0 S 0 0.0 0:00.00 kblockd/0 10 root 5 -10 0 0 0 S 0 0.0 0:00.00 kblockd/1 11 root 15 0 0 0 0 S 0 0.0 0:00.00 khubd 66 root 15 0 0 0 0 S 0 0.0 0:16.14 pdflush 67 root 15 0 0 0 0 S 0 0.0 2:30.44 kswapd0 68 root 15 -10 0 0 0 S 0 0.0 0:00.00 aio/0 69 root 5 -10 0 0 0 S 0 0.0 0:00.00 aio/1 213 root 22 0 0 0 0 S 0 0.0 0:00.00 kseriod 450 root 5 -10 0 0 0 S 0 0.0 0:00.00 ata/0 451 root 5 -10 0 0 0 S 0 0.0 0:00.00 ata/1 452 root 6 -10 0 0 0 S 0 0.0 0:00.00 ata_aux 456 root 15 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_0 457 root 15 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_1 465 root 15 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_2 466 root 15 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_3 494 root 15 0 0 0 0 S 0 0.0 4:03.36 kjournald 542 root 16 0 24192 596 416 S 0 0.1 0:36.55 ypbind 595 root 15 0 0 0 0 S 0 0.0 0:00.00 rpciod 596 root 18 0 0 0 0 S 0 0.0 0:00.00 lockd 775 apache 15 0 185m 40m 3628 S 0 4.6 0:00.59 httpd 1832 root 5 -10 0 0 0 S 0 0.0 0:00.00 kedac 1904 root 16 0 10828 1072 856 S 0 0.1 0:00.00 pppd 1973 root 6 -10 0 0 0 S 0 0.0 0:00.00 kauditd 1998 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_4 1999 root 15 0 0 0 0 S 0 0.0 3:10.72 usb-storage 2176 root 15 0 0 0 0 S 0 0.0 0:00.01 kjournald 3070 root 15 0 3632 552 440 S 0 0.1 1:08.11 syslogd 3074 root 15 0 2540 372 296 S 0 0.0 0:00.04 klogd 3087 root 16 0 2552 300 200 S 0 0.0 0:30.06 irqbalance 3155 root 15 0 2916 488 256 S 0 0.1 0:00.97 smartd 3215 root 15 0 8716 776 612 S 0 0.1 0:00.00 xinetd 3293 root 16 0 40836 2672 984 S 0 0.3 1:07.23 sendmail 3303 smmsp 16 0 33504 2108 812 S 0 0.2 0:00.99 sendmail 3313 root 16 0 4180 324 244 S 0 0.0 0:00.00 gpm 3323 root 16 0 175m 35m 9064 S 0 4.0 0:13.24 httpd 3332 root 16 0 57092 976 560 S 0 0.1 0:50.01 crond 3353 xfs 16 0 10020 1472 756 S 0 0.2 0:00.17 xfs
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
-- Jason Pyeron PD Inc. http://www.pdinc.us -
- Principal Consultant 10 West 24th Street #100 -
- +1 (443) 269-1555 x333 Baltimore, Maryland 21218 -
--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, purge the message from your system and notify the sender immediately. Any other use of the email by you is prohibited.
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On Fri, Apr 25, 2008 at 1:46 PM, Kevin Faulkner kevlar.kernel@gmail.com wrote:
Perhaps you could run iotop find out if there is a process chewing up your hard disks. You aren't just limited to top, you can also use vmstat and iostat in this situation.
I also recommend dstat (http://dag.wieers.com/home-made/dstat/). It shows informations from several sources, so it's possible to see if you have CPU peaks and I/O peaks at the same time.
Filipe
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Kevin Faulkner Sent: Friday, April 25, 2008 1:46 PM To: CentOS mailing list Subject: Re: [CentOS] load level?
Perhaps you could run iotop find out if there is a process chewing up your hard disks. You aren't just limited to top, you can also use vmstat and iostat in this situation.
After stoping httpd, sendmail and mysql:
# uptime 15:48:02 up 167 days, 9:20, 5 users, load average: 0.52, 0.55, 0.92
# vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 196 144668 153800 202860 0 0 3 9 1 2 13 12 75 0
init(1)-+-atd(3370) |-dbus-daemon-1(3392) |-events/0(6)-+-aio/0(68) | |-aio/1(69) | |-kauditd(1973) | |-kblockd/0(9) | |-kblockd/1(10) | |-khelper(8) | `-pdflush(66) |-events/1(7)-+-ata/0(450) | |-ata/1(451) | |-ata_aux(452) | |-kedac(1832) | `-pdflush(21556) |-gpm(3313) |-hald(3402) |-irqbalance(3087) |-khubd(11) |-kjournald(494) |-kjournald(2176) |-klogd(3074) |-kseriod(213) |-ksoftirqd/0(3) |-ksoftirqd/1(5) |-kswapd0(67) |-lockd(596) |-login(3509)---bash(19184) |-migration/0(2) |-migration/1(4) |-mingetty(3510) |-mingetty(3511) |-mingetty(3512) |-mingetty(3513) |-mingetty(3514) |-minilogd(20220) |-minilogd(22648) |-portmap(26404) |-rpciod(595) |-scsi_eh_0(456) |-scsi_eh_1(457) |-scsi_eh_2(465) |-scsi_eh_3(466) |-scsi_eh_4(1998) |-smartd(3155) |-sshd(21029)-+-sshd(11899)---bash(11911)-+-pstree(14050) | | `-tail(12125) | |-sshd(12900)---sshd(12902)---bash(12913) | |-sshd(13882)---bash(13900) | `-sshd(23839)---bash(23841) |-syslogd(3070) |-udevd(7922) |-usb-storage(1999) |-xfs(3353) |-xinetd(3215) `-ypbind(14008)
# up2date --whatprovides iostat # up2date --whatprovides iotop #
Where do I get iostat / iotop?
Starting sendmail back up puts me in the high 1's to low 2's
Jason Pyeron wrote:
I have a server holding a 2.5-4.0 load level consistently, where should
I
start debugging it?
top - 13:32:01 up 167 days, 5:45, 6 users, load average: 2.87, 1.83,
1.51
Tasks: 111 total, 1 running, 110 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0% us, 0.0% sy, 0.2% ni, 99.8% id, 0.0% wa, 0.0% hi,
0.0% si
Mem: 895728k total, 871996k used, 23732k free, 190612k buffers Swap: 1802232k total, 216k used, 1802016k free, 63776k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1 root 16 0 4756 556 460 S 0 0.1 0:35.34 init 2 root RT 0 0 0 0 S 0 0.0 0:02.83 migration/0 3 root 34 19 0 0 0 S 0 0.0 0:27.27 ksoftirqd/0 4 root RT 0 0 0 0 S 0 0.0 0:05.03 migration/1 5 root 34 19 0 0 0 S 0 0.0 0:22.88 ksoftirqd/1 6 root 5 -10 0 0 0 S 0 0.0 0:03.15 events/0 7 root 5 -10 0 0 0 S 0 0.0 0:01.98 events/1 8 root 5 -10 0 0 0 S 0 0.0 0:00.00 khelper 9 root 5 -10 0 0 0 S 0 0.0 0:00.00 kblockd/0 10 root 5 -10 0 0 0 S 0 0.0 0:00.00 kblockd/1 11 root 15 0 0 0 0 S 0 0.0 0:00.00 khubd 66 root 15 0 0 0 0 S 0 0.0 0:16.14 pdflush 67 root 15 0 0 0 0 S 0 0.0 2:30.44 kswapd0 68 root 15 -10 0 0 0 S 0 0.0 0:00.00 aio/0 69 root 5 -10 0 0 0 S 0 0.0 0:00.00 aio/1 213 root 22 0 0 0 0 S 0 0.0 0:00.00 kseriod 450 root 5 -10 0 0 0 S 0 0.0 0:00.00 ata/0 451 root 5 -10 0 0 0 S 0 0.0 0:00.00 ata/1 452 root 6 -10 0 0 0 S 0 0.0 0:00.00 ata_aux 456 root 15 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_0 457 root 15 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_1 465 root 15 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_2 466 root 15 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_3 494 root 15 0 0 0 0 S 0 0.0 4:03.36 kjournald 542 root 16 0 24192 596 416 S 0 0.1 0:36.55 ypbind 595 root 15 0 0 0 0 S 0 0.0 0:00.00 rpciod 596 root 18 0 0 0 0 S 0 0.0 0:00.00 lockd 775 apache 15 0 185m 40m 3628 S 0 4.6 0:00.59 httpd 1832 root 5 -10 0 0 0 S 0 0.0 0:00.00 kedac 1904 root 16 0 10828 1072 856 S 0 0.1 0:00.00 pppd 1973 root 6 -10 0 0 0 S 0 0.0 0:00.00 kauditd 1998 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_4 1999 root 15 0 0 0 0 S 0 0.0 3:10.72 usb-storage 2176 root 15 0 0 0 0 S 0 0.0 0:00.01 kjournald 3070 root 15 0 3632 552 440 S 0 0.1 1:08.11 syslogd 3074 root 15 0 2540 372 296 S 0 0.0 0:00.04 klogd 3087 root 16 0 2552 300 200 S 0 0.0 0:30.06 irqbalance 3155 root 15 0 2916 488 256 S 0 0.1 0:00.97 smartd 3215 root 15 0 8716 776 612 S 0 0.1 0:00.00 xinetd 3293 root 16 0 40836 2672 984 S 0 0.3 1:07.23 sendmail 3303 smmsp 16 0 33504 2108 812 S 0 0.2 0:00.99 sendmail 3313 root 16 0 4180 324 244 S 0 0.0 0:00.00 gpm 3323 root 16 0 175m 35m 9064 S 0 4.0 0:13.24 httpd 3332 root 16 0 57092 976 560 S 0 0.1 0:50.01 crond 3353 xfs 16 0 10020 1472 756 S 0 0.2 0:00.17 xfs
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
-- Jason Pyeron PD Inc. http://www.pdinc.us -
- Principal Consultant 10 West 24th Street #100 -
- +1 (443) 269-1555 x333 Baltimore, Maryland 21218 -
--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, purge the message from your system and notify the sender immediately. Any other use of the email by you is prohibited.
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- - - - Jason Pyeron PD Inc. http://www.pdinc.us - - Principal Consultant 10 West 24th Street #100 - - +1 (443) 269-1555 x333 Baltimore, Maryland 21218 - - - -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, purge the message from your system and notify the sender immediately. Any other use of the email by you is prohibited.
Jason Pyeron wrote:
-----Original Message-----
Starting sendmail back up puts me in the high 1's to low 2's
Are you getting lots of large-ish emails? maybe sendmail is working with some sort of anti-virus/spam tool?
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Milton Calnek Sent: Friday, April 25, 2008 3:56 PM To: CentOS mailing list Subject: Re: [CentOS] load level?
Jason Pyeron wrote:
-----Original Message-----
Starting sendmail back up puts me in the high 1's to low 2's
Nope it is behind a firewall, no port 25 access.
mailq is 55k (logwatch)
Are you getting lots of large-ish emails? maybe sendmail is working with some sort of anti-virus/spam tool?
-- Milton Calnek BSc, A/Slt(Ret.) milton@calnek.com 306-717-8737
-- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- - - - Jason Pyeron PD Inc. http://www.pdinc.us - - Principal Consultant 10 West 24th Street #100 - - +1 (443) 269-1555 x333 Baltimore, Maryland 21218 - - - -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, purge the message from your system and notify the sender immediately. Any other use of the email by you is prohibited.
Jason Pyeron wrote:
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Milton Calnek Sent: Friday, April 25, 2008 3:56 PM To: CentOS mailing list Subject: Re: [CentOS] load level?
Jason Pyeron wrote:
-----Original Message-----
Starting sendmail back up puts me in the high 1's to low 2's
Nope it is behind a firewall, no port 25 access.
mailq is 55k (logwatch)
(55 kilobytes? or 55,000 messages?? Just curious; the former is nothing of particular interest...the latter is a concern.)
Sounds like sendmail is hanging up on DNS lookups or other network I/O. This wont slow your system down necessarily, it will just inflate your load averages. The system load is just a measure of how many processes are in the run queue averaged over the preceding 1/5/15 minutes.
So if sendmail (and other processes) are waiting for slow DNS responses, the process will sit in the run queue until it either times out, or gets a response. Additionally, if sendmail is trying to connect to unresponsive SMTP hosts, it will sit with a process in the run queue until the remote SMTP server either responds or times out. The fact the load dropped and then spiked again with the shutdown/restart of sendmail seems to lend credibility to my hypothesis.
Check your DNS settings and also check to see if you are logging FQDN's in Apache as this will add significantly to your load average if your DNS is a little flaky. If you're worried about security, you only need to allow UDP/53 for DNS lookups in 99.99% of cases - it would be very unlikely that a DNS RR exceeds the maximum size of a UDP packet. To be completely covered also open TCP/53 *to* your DNS server(s) and allow connections that are related/established.
Lastly, make sure you reject (as opposed to "drop") any egress traffic you don't want going out. This way the local processes wont sit around waiting as they will get an ICMP connection refused/RST/FIN or whatever and terminate the connection immediately, clean up and get out of the run queue. If you silently drop egress packets at your edge, the local process will have no idea what happened to it's SYN packet etc, and will just sit around in the run queue (inflating your load averages) waiting for a reply that will never come, until it times out.
HTH,
James
Jason Pyeron wrote:
Kevin Faulkner wrote:
Perhaps you could run iotop find out if there is a process chewing up your hard disks. You aren't just limited to top, you can also use vmstat and iostat in this situation.
After stoping httpd, sendmail and mysql:
# uptime 15:48:02 up 167 days, 9:20, 5 users, load average: 0.52, 0.55, 0.92
# vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 196 144668 153800 202860 0 0 3 9 1 2 13 12 75 0
init(1)-+-atd(3370) |-dbus-daemon-1(3392) |-events/0(6)-+-aio/0(68) | |-aio/1(69) | |-kauditd(1973) | |-kblockd/0(9) | |-kblockd/1(10) | |-khelper(8) | `-pdflush(66) |-events/1(7)-+-ata/0(450) | |-ata/1(451) | |-ata_aux(452) | |-kedac(1832) | `-pdflush(21556) |-gpm(3313) |-hald(3402) |-irqbalance(3087) |-khubd(11) |-kjournald(494) |-kjournald(2176) |-klogd(3074) |-kseriod(213) |-ksoftirqd/0(3) |-ksoftirqd/1(5) |-kswapd0(67) |-lockd(596) |-login(3509)---bash(19184) |-migration/0(2) |-migration/1(4) |-mingetty(3510) |-mingetty(3511) |-mingetty(3512) |-mingetty(3513) |-mingetty(3514) |-minilogd(20220) |-minilogd(22648) |-portmap(26404) |-rpciod(595) |-scsi_eh_0(456) |-scsi_eh_1(457) |-scsi_eh_2(465) |-scsi_eh_3(466) |-scsi_eh_4(1998) |-smartd(3155) |-sshd(21029)-+-sshd(11899)---bash(11911)-+-pstree(14050) | | `-tail(12125) | |-sshd(12900)---sshd(12902)---bash(12913) | |-sshd(13882)---bash(13900) | `-sshd(23839)---bash(23841) |-syslogd(3070) |-udevd(7922) |-usb-storage(1999) |-xfs(3353) |-xinetd(3215) `-ypbind(14008)
Nice process tree, what generated it?
# up2date --whatprovides iostat # up2date --whatprovides iotop
Try 'yum provides iostat'
Where do I get iostat / iotop?
They're in the sysstat package
Starting sendmail back up puts me in the high 1's to low 2's
I would think about isolating email then, especially if it's handling it unfiltered from the Internet.
-Ross
______________________________________________________________________ This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender and permanently delete the original and any copy or printout thereof.
On Fri, 25 Apr 2008, Ross S. W. Walker wrote:
init(1)-+-atd(3370) |-dbus-daemon-1(3392) |-events/0(6)-+-aio/0(68) | |-aio/1(69) [...snip...] |-xfs(3353) |-xinetd(3215) `-ypbind(14008)
Nice process tree, what generated it?
pstree, part of the psmisc package.
Paul Heinlein wrote:
On Fri, 25 Apr 2008, Ross S. W. Walker wrote:
init(1)-+-atd(3370) |-dbus-daemon-1(3392) |-events/0(6)-+-aio/0(68) | |-aio/1(69) [...snip...] |-xfs(3353) |-xinetd(3215) `-ypbind(14008)
Nice process tree, what generated it?
pstree, part of the psmisc package.
Thanks, good to know.
-Ross
______________________________________________________________________ This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender and permanently delete the original and any copy or printout thereof.
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Ross S. W. Walker Sent: Friday, April 25, 2008 4:05 PM To: CentOS mailing list Subject: RE: [CentOS] load level?
Jason Pyeron wrote:
Kevin Faulkner wrote:
Perhaps you could run iotop find out if there is a process chewing up your hard disks. You aren't just limited to top, you can also use vmstat and iostat in this situation.
They're in the sysstat package
Starting sendmail back up puts me in the high 1's to low 2's
I would think about isolating email then, especially if it's handling it unfiltered from the Internet.
After 15 minutes it dropped down to .75, now with httpd its back up
[root@devserver21 ~]# iostat Linux 2.6.9-55.0.12.ELsmp (devserver21.internal.pdinc.us) 04/25/2008
avg-cpu: %user %nice %sys %iowait %idle 13.31 0.00 11.66 0.12 74.91
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn hdc 0.00 0.00 0.00 424 0 sda 2.44 12.81 34.34 116132748 311250630 sda1 0.00 0.00 0.00 8138 2214 sda2 5.34 12.81 34.34 116124346 311248368 dm-0 5.34 12.80 34.32 115985770 311090800 dm-1 0.00 0.02 0.02 137856 157616
not sure what I am looking at.
-Ross
This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender and permanently delete the original and any copy or printout thereof.
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- - - - Jason Pyeron PD Inc. http://www.pdinc.us - - Principal Consultant 10 West 24th Street #100 - - +1 (443) 269-1555 x333 Baltimore, Maryland 21218 - - - -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, purge the message from your system and notify the sender immediately. Any other use of the email by you is prohibited.
Where do I get iostat / iotop?
Starting sendmail back up puts me in the high 1's to low 2's
My next course of action would be to start sendmail in debug level 0.1 and check some logs. Find out exactly sendmail is trying to do, maybe lsof to see if it is getting stuck on something. I am pretty certain (from the information provided) that its IO that is bringing your system to its knees. iotop is available on pbone.net and as for iostat # yum install sysstat. But I think you really you have found your problem child.
-
Filipe Brandenburger -
James Gray -
Jason Pyeron -
Kevin Faulkner -
Les Mikesell -
Milton Calnek -
Paul Heinlein -
Ross S. W. Walker