Hey! I have a strange condition in one of the servers that I don't where to start looking. I login to the server via SSH (cant doit any other way) and anything that I type is slow HTTP sessions timeout waiting for screen redraw. So, the server is acting "slow".
server is bare metal. no virtual services. no alarms in the disk raid
note: server was restarted because of power failure.
Some outputs from this server that is a mail server: [root@correo ~]# top top - 09:54:43 up 23:51, 2 users, load average: 0.18, 0.23, 0.28 Tasks: 210 total, 1 running, 209 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.2 us, 0.1 sy, 0.0 ni, 99.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 32606084 total, 25106412 free, 5932244 used, 1567428 buff/cache KiB Swap: 16449532 total, 16449532 free, 0 used. 26282624 avail Mem
**iostat** [root@correo ~]# iostat -y 5 Linux 3.10.0-1062.12.1.el7.x86_64 (correo.binal.ac.pa) 07/03/2020 _x86_64_ (4 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle 0.05 0.00 0.05 0.05 0.00 99.85
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn sda 0.00 0.00 0.00 0 0 dm-0 0.00 0.00 0.00 0 0 dm-1 0.00 0.00 0.00 0 0
avg-cpu: %user %nice %system %iowait %steal %idle 0.05 0.00 0.05 0.05 0.00 99.85
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn sda 21.40 0.00 169.60 0 848 dm-0 21.40 0.00 169.60 0 848 dm-1 0.00 0.00 0.00 0 0
avg-cpu: %user %nice %system %iowait %steal %idle 0.60 0.00 0.05 0.45 0.00 98.90
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn sda 1.20 16.80 0.00 84 0 dm-0 1.20 16.80 0.00 84 0 dm-1 0.00 0.00 0.00 0 0
avg-cpu: %user %nice %system %iowait %steal %idle 0.05 0.00 0.00 0.05 0.00 99.90
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn sda 8.00 0.00 100.20 0 501 dm-0 9.00 0.00 100.20 0 501 dm-1 0.00 0.00 0.00 0 0
avg-cpu: %user %nice %system %iowait %steal %idle 0.45 0.00 0.35 0.05 0.00 99.15
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn sda 1.00 0.80 3.20 4 16 dm-0 1.00 0.80 3.20 4 16 dm-1 0.00 0.00 0.00 0 0
**dstop** [root@correo ~]# dstat -cd --disk-util --disk-tps ----total-cpu-usage---- -dsk/total- sda- -dsk/total- usr sys idl wai hiq siq| read writ|util|reads writs 1 0 99 0 0 0| 20k 17k|0.14| 1 1 0 0 100 0 0 0| 0 0 | 0| 0 0 0 0 100 0 0 0| 0 0 | 0| 0 0 0 0 100 0 0 0| 0 0 | 0| 0 0 0 0 100 0 0 0| 0 0 | 0| 0 0 0 0 100 0 0 0| 0 0 | 0| 0 0 4 0 84 11 0 0|2512k 228k|52.3| 123 2 31 4 58 7 0 0|1912k 1026k|38.1| 132 23 0 0 99 0 0 0| 0 0 | 0| 0 0 1 0 99 1 0 0|4096B 3819k|22.5| 1 270 0 0 100 0 0 0| 0 0 | 0| 0 0 13 1 83 4 0 0| 148k 2304k|15.3| 18 214 1 0 98 1 0 0| 140k 499k|9.70| 14 8 26 5 69 0 0 0| 0 1260k|1.30| 0 46 56 7 38 0 0 0| 0 204k|0.30| 0 12 14 11 75 0 0 0| 0 0 | 0| 0 0 22 10 68 0 0 0| 0 0 | 0| 0 0 16 10 71 3 0 0| 192k 37k|14.0| 12 2 0 0 100 0 0 0| 0 0 | 0| 0 0 0 0 100 0 0 0| 0 152k| 0| 0 2 0 0 100 0 0 0| 0 0 | 0| 0 0 1 1 98 1 0 0| 16k 2569k|14.8| 1 207 1 1 98 0 0 0|4096B 0 |1.10| 1 0 1 0 99 0 0 0| 0 0 | 0| 0 0 0 0 100 0 0 0| 0 600k|1.30| 0 51 2 0 98 0 0 0| 0 0 | 0| 0 0 4 0 96 0 0 0| 0 0 | 0| 0 0 0 0 100 0 0 0| 0 0 | 0| 0 0
It was found that the software NIC team created in Centos was having issues due to a failing network cable. The team was going berserk with up/down changes.
On Fri, Jul 3, 2020 at 10:12 AM Erick Perez - Quadrian Enterprises < eperez@quadrianweb.com> wrote:
Hey! I have a strange condition in one of the servers that I don't where to start looking. I login to the server via SSH (cant doit any other way) and anything that I type is slow HTTP sessions timeout waiting for screen redraw. So, the server is acting "slow".
server is bare metal. no virtual services. no alarms in the disk raid
note: server was restarted because of power failure.
Some outputs from this server that is a mail server: [root@correo ~]# top top - 09:54:43 up 23:51, 2 users, load average: 0.18, 0.23, 0.28 Tasks: 210 total, 1 running, 209 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.2 us, 0.1 sy, 0.0 ni, 99.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 32606084 total, 25106412 free, 5932244 used, 1567428 buff/cache KiB Swap: 16449532 total, 16449532 free, 0 used. 26282624 avail Mem
**iostat** [root@correo ~]# iostat -y 5 Linux 3.10.0-1062.12.1.el7.x86_64 (correo.binal.ac.pa) 07/03/2020 _x86_64_ (4 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle 0.05 0.00 0.05 0.05 0.00 99.85
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn sda 0.00 0.00 0.00 0 0 dm-0 0.00 0.00 0.00 0 0 dm-1 0.00 0.00 0.00 0 0
avg-cpu: %user %nice %system %iowait %steal %idle 0.05 0.00 0.05 0.05 0.00 99.85
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn sda 21.40 0.00 169.60 0 848 dm-0 21.40 0.00 169.60 0 848 dm-1 0.00 0.00 0.00 0 0
avg-cpu: %user %nice %system %iowait %steal %idle 0.60 0.00 0.05 0.45 0.00 98.90
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn sda 1.20 16.80 0.00 84 0 dm-0 1.20 16.80 0.00 84 0 dm-1 0.00 0.00 0.00 0 0
avg-cpu: %user %nice %system %iowait %steal %idle 0.05 0.00 0.00 0.05 0.00 99.90
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn sda 8.00 0.00 100.20 0 501 dm-0 9.00 0.00 100.20 0 501 dm-1 0.00 0.00 0.00 0 0
avg-cpu: %user %nice %system %iowait %steal %idle 0.45 0.00 0.35 0.05 0.00 99.15
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn sda 1.00 0.80 3.20 4 16 dm-0 1.00 0.80 3.20 4 16 dm-1 0.00 0.00 0.00 0 0
**dstop** [root@correo ~]# dstat -cd --disk-util --disk-tps ----total-cpu-usage---- -dsk/total- sda- -dsk/total- usr sys idl wai hiq siq| read writ|util|reads writs 1 0 99 0 0 0| 20k 17k|0.14| 1 1 0 0 100 0 0 0| 0 0 | 0| 0 0 0 0 100 0 0 0| 0 0 | 0| 0 0 0 0 100 0 0 0| 0 0 | 0| 0 0 0 0 100 0 0 0| 0 0 | 0| 0 0 0 0 100 0 0 0| 0 0 | 0| 0 0 4 0 84 11 0 0|2512k 228k|52.3| 123 2 31 4 58 7 0 0|1912k 1026k|38.1| 132 23 0 0 99 0 0 0| 0 0 | 0| 0 0 1 0 99 1 0 0|4096B 3819k|22.5| 1 270 0 0 100 0 0 0| 0 0 | 0| 0 0 13 1 83 4 0 0| 148k 2304k|15.3| 18 214 1 0 98 1 0 0| 140k 499k|9.70| 14 8 26 5 69 0 0 0| 0 1260k|1.30| 0 46 56 7 38 0 0 0| 0 204k|0.30| 0 12 14 11 75 0 0 0| 0 0 | 0| 0 0 22 10 68 0 0 0| 0 0 | 0| 0 0 16 10 71 3 0 0| 192k 37k|14.0| 12 2 0 0 100 0 0 0| 0 0 | 0| 0 0 0 0 100 0 0 0| 0 152k| 0| 0 2 0 0 100 0 0 0| 0 0 | 0| 0 0 1 1 98 1 0 0| 16k 2569k|14.8| 1 207 1 1 98 0 0 0|4096B 0 |1.10| 1 0 1 0 99 0 0 0| 0 0 | 0| 0 0 0 0 100 0 0 0| 0 600k|1.30| 0 51 2 0 98 0 0 0| 0 0 | 0| 0 0 4 0 96 0 0 0| 0 0 | 0| 0 0 0 0 100 0 0 0| 0 0 | 0| 0 0
--
Erick Perez
Hi Erick,
what was the value of 'si' in top ?
Best Regards, Strahil Nikolov
На 3 юли 2020 г. 18:48:30 GMT+03:00, Erick Perez - Quadrian Enterprises eperez@quadrianweb.com написа:
It was found that the software NIC team created in Centos was having issues due to a failing network cable. The team was going berserk with up/down changes.
On Fri, Jul 3, 2020 at 10:12 AM Erick Perez - Quadrian Enterprises < eperez@quadrianweb.com> wrote:
Hey! I have a strange condition in one of the servers that I don't where
to
start looking. I login to the server via SSH (cant doit any other way) and anything
that
I type is slow HTTP sessions timeout waiting for screen redraw. So, the server is
acting
"slow".
server is bare metal. no virtual services. no alarms in the disk raid
note: server was restarted because of power failure.
Some outputs from this server that is a mail server: [root@correo ~]# top top - 09:54:43 up 23:51, 2 users, load average: 0.18, 0.23, 0.28 Tasks: 210 total, 1 running, 209 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.2 us, 0.1 sy, 0.0 ni, 99.8 id, 0.0 wa, 0.0 hi, 0.0
si,
0.0 st KiB Mem : 32606084 total, 25106412 free, 5932244 used, 1567428
buff/cache
KiB Swap: 16449532 total, 16449532 free, 0 used. 26282624
avail Mem
**iostat** [root@correo ~]# iostat -y 5 Linux 3.10.0-1062.12.1.el7.x86_64 (correo.binal.ac.pa) 07/03/2020 _x86_64_ (4 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle 0.05 0.00 0.05 0.05 0.00 99.85
Device: tps kB_read/s kB_wrtn/s kB_read
kB_wrtn
sda 0.00 0.00 0.00 0
0
dm-0 0.00 0.00 0.00 0
0
dm-1 0.00 0.00 0.00 0
0
avg-cpu: %user %nice %system %iowait %steal %idle 0.05 0.00 0.05 0.05 0.00 99.85
Device: tps kB_read/s kB_wrtn/s kB_read
kB_wrtn
sda 21.40 0.00 169.60 0
848
dm-0 21.40 0.00 169.60 0
848
dm-1 0.00 0.00 0.00 0
0
avg-cpu: %user %nice %system %iowait %steal %idle 0.60 0.00 0.05 0.45 0.00 98.90
Device: tps kB_read/s kB_wrtn/s kB_read
kB_wrtn
sda 1.20 16.80 0.00 84
0
dm-0 1.20 16.80 0.00 84
0
dm-1 0.00 0.00 0.00 0
0
avg-cpu: %user %nice %system %iowait %steal %idle 0.05 0.00 0.00 0.05 0.00 99.90
Device: tps kB_read/s kB_wrtn/s kB_read
kB_wrtn
sda 8.00 0.00 100.20 0
501
dm-0 9.00 0.00 100.20 0
501
dm-1 0.00 0.00 0.00 0
0
avg-cpu: %user %nice %system %iowait %steal %idle 0.45 0.00 0.35 0.05 0.00 99.15
Device: tps kB_read/s kB_wrtn/s kB_read
kB_wrtn
sda 1.00 0.80 3.20 4
16
dm-0 1.00 0.80 3.20 4
16
dm-1 0.00 0.00 0.00 0
0
**dstop** [root@correo ~]# dstat -cd --disk-util --disk-tps ----total-cpu-usage---- -dsk/total- sda- -dsk/total- usr sys idl wai hiq siq| read writ|util|reads writs 1 0 99 0 0 0| 20k 17k|0.14| 1 1 0 0 100 0 0 0| 0 0 | 0| 0 0 0 0 100 0 0 0| 0 0 | 0| 0 0 0 0 100 0 0 0| 0 0 | 0| 0 0 0 0 100 0 0 0| 0 0 | 0| 0 0 0 0 100 0 0 0| 0 0 | 0| 0 0 4 0 84 11 0 0|2512k 228k|52.3| 123 2 31 4 58 7 0 0|1912k 1026k|38.1| 132 23 0 0 99 0 0 0| 0 0 | 0| 0 0 1 0 99 1 0 0|4096B 3819k|22.5| 1 270 0 0 100 0 0 0| 0 0 | 0| 0 0 13 1 83 4 0 0| 148k 2304k|15.3| 18 214 1 0 98 1 0 0| 140k 499k|9.70| 14 8 26 5 69 0 0 0| 0 1260k|1.30| 0 46 56 7 38 0 0 0| 0 204k|0.30| 0 12 14 11 75 0 0 0| 0 0 | 0| 0 0 22 10 68 0 0 0| 0 0 | 0| 0 0 16 10 71 3 0 0| 192k 37k|14.0| 12 2 0 0 100 0 0 0| 0 0 | 0| 0 0 0 0 100 0 0 0| 0 152k| 0| 0 2 0 0 100 0 0 0| 0 0 | 0| 0 0 1 1 98 1 0 0| 16k 2569k|14.8| 1 207 1 1 98 0 0 0|4096B 0 |1.10| 1 0 1 0 99 0 0 0| 0 0 | 0| 0 0 0 0 100 0 0 0| 0 600k|1.30| 0 51 2 0 98 0 0 0| 0 0 | 0| 0 0 4 0 96 0 0 0| 0 0 | 0| 0 0 0 0 100 0 0 0| 0 0 | 0| 0 0
--
Erick Perez
"si / software interrupts" value was 0.0 and right now with all working fine, continues to be 0.0
On Fri, Jul 3, 2020 at 11:23 AM Strahil Nikolov hunter86_bg@yahoo.com wrote:
Hi Erick,
what was the value of 'si' in top ?
Best Regards, Strahil Nikolov
На 3 юли 2020 г. 18:48:30 GMT+03:00, Erick Perez - Quadrian Enterprises < eperez@quadrianweb.com> написа:
It was found that the software NIC team created in Centos was having issues due to a failing network cable. The team was going berserk with up/down changes.
On Fri, Jul 3, 2020 at 10:12 AM Erick Perez - Quadrian Enterprises < eperez@quadrianweb.com> wrote:
Hey! I have a strange condition in one of the servers that I don't where
to
start looking. I login to the server via SSH (cant doit any other way) and anything
that
I type is slow HTTP sessions timeout waiting for screen redraw. So, the server is
acting
"slow".
server is bare metal. no virtual services. no alarms in the disk raid
note: server was restarted because of power failure.
Some outputs from this server that is a mail server: [root@correo ~]# top top - 09:54:43 up 23:51, 2 users, load average: 0.18, 0.23, 0.28 Tasks: 210 total, 1 running, 209 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.2 us, 0.1 sy, 0.0 ni, 99.8 id, 0.0 wa, 0.0 hi, 0.0
si,
0.0 st KiB Mem : 32606084 total, 25106412 free, 5932244 used, 1567428
buff/cache
KiB Swap: 16449532 total, 16449532 free, 0 used. 26282624
avail Mem
**iostat** [root@correo ~]# iostat -y 5 Linux 3.10.0-1062.12.1.el7.x86_64 (correo.binal.ac.pa) 07/03/2020 _x86_64_ (4 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle 0.05 0.00 0.05 0.05 0.00 99.85
Device: tps kB_read/s kB_wrtn/s kB_read
kB_wrtn
sda 0.00 0.00 0.00 0
0
dm-0 0.00 0.00 0.00 0
0
dm-1 0.00 0.00 0.00 0
0
avg-cpu: %user %nice %system %iowait %steal %idle 0.05 0.00 0.05 0.05 0.00 99.85
Device: tps kB_read/s kB_wrtn/s kB_read
kB_wrtn
sda 21.40 0.00 169.60 0
848
dm-0 21.40 0.00 169.60 0
848
dm-1 0.00 0.00 0.00 0
0
avg-cpu: %user %nice %system %iowait %steal %idle 0.60 0.00 0.05 0.45 0.00 98.90
Device: tps kB_read/s kB_wrtn/s kB_read
kB_wrtn
sda 1.20 16.80 0.00 84
0
dm-0 1.20 16.80 0.00 84
0
dm-1 0.00 0.00 0.00 0
0
avg-cpu: %user %nice %system %iowait %steal %idle 0.05 0.00 0.00 0.05 0.00 99.90
Device: tps kB_read/s kB_wrtn/s kB_read
kB_wrtn
sda 8.00 0.00 100.20 0
501
dm-0 9.00 0.00 100.20 0
501
dm-1 0.00 0.00 0.00 0
0
avg-cpu: %user %nice %system %iowait %steal %idle 0.45 0.00 0.35 0.05 0.00 99.15
Device: tps kB_read/s kB_wrtn/s kB_read
kB_wrtn
sda 1.00 0.80 3.20 4
16
dm-0 1.00 0.80 3.20 4
16
dm-1 0.00 0.00 0.00 0
0
**dstop** [root@correo ~]# dstat -cd --disk-util --disk-tps ----total-cpu-usage---- -dsk/total- sda- -dsk/total- usr sys idl wai hiq siq| read writ|util|reads writs 1 0 99 0 0 0| 20k 17k|0.14| 1 1 0 0 100 0 0 0| 0 0 | 0| 0 0 0 0 100 0 0 0| 0 0 | 0| 0 0 0 0 100 0 0 0| 0 0 | 0| 0 0 0 0 100 0 0 0| 0 0 | 0| 0 0 0 0 100 0 0 0| 0 0 | 0| 0 0 4 0 84 11 0 0|2512k 228k|52.3| 123 2 31 4 58 7 0 0|1912k 1026k|38.1| 132 23 0 0 99 0 0 0| 0 0 | 0| 0 0 1 0 99 1 0 0|4096B 3819k|22.5| 1 270 0 0 100 0 0 0| 0 0 | 0| 0 0 13 1 83 4 0 0| 148k 2304k|15.3| 18 214 1 0 98 1 0 0| 140k 499k|9.70| 14 8 26 5 69 0 0 0| 0 1260k|1.30| 0 46 56 7 38 0 0 0| 0 204k|0.30| 0 12 14 11 75 0 0 0| 0 0 | 0| 0 0 22 10 68 0 0 0| 0 0 | 0| 0 0 16 10 71 3 0 0| 192k 37k|14.0| 12 2 0 0 100 0 0 0| 0 0 | 0| 0 0 0 0 100 0 0 0| 0 152k| 0| 0 2 0 0 100 0 0 0| 0 0 | 0| 0 0 1 1 98 1 0 0| 16k 2569k|14.8| 1 207 1 1 98 0 0 0|4096B 0 |1.10| 1 0 1 0 99 0 0 0| 0 0 | 0| 0 0 0 0 100 0 0 0| 0 600k|1.30| 0 51 2 0 98 0 0 0| 0 0 | 0| 0 0 4 0 96 0 0 0| 0 0 | 0| 0 0 0 0 100 0 0 0| 0 0 | 0| 0 0
--
Erick Perez
-
Erick Perez - Quadrian Enterprises -
Strahil Nikolov