Hi,
recently I noticed, that one of our webservers is using swap space, while there is plenty of physical ram available.
free -m total used free shared buffers cached Mem: 8118 2014 6103 0 85 261 -/+ buffers/cache: 1667 6450 Swap: 8197 77 8119
It's not that much, but why?
Any ideas how to debug that? Thanks for any hint and suggestion.
/Götz
Hi Götz,
On Dec 10, 2013, at 15:01 , Götz Reinicke - IT Koordinator goetz.reinicke@filmakademie.de wrote:
Hi,
recently I noticed, that one of our webservers is using swap space, while there is plenty of physical ram available.
free -m total used free shared buffers cached Mem: 8118 2014 6103 0 85 261 -/+ buffers/cache: 1667 6450 Swap: 8197 77 8119
It's not that much, but why?
my first idea would be that RAM usage was higher temporarily, which led to some pages being swapped out, and those pages haven't been used since so they were never swapped in again.
Any ideas how to debug that? Thanks for any hint and suggestion.
/proc/*/smaps is a starting point. It's fairly easy to write a little script that adds the values in the Swap: lines for each process and calculates swap usage per process.
Bests,
Peter.
recently I noticed, that one of our webservers is using swap space, while there is plenty of physical ram available.
free -m total used free shared buffers cached Mem: 8118 2014 6103 0 85 261 -/+ buffers/cache: 1667 6450 Swap: 8197 77 8119
It's not that much, but why?
Any ideas how to debug that? Thanks for any hint and suggestion.
#!/bin/bash # Get current swap usage for all running processes # Erik Ljungstrom 27/05/2011 # Modified by Mikko Rantalainen 2012-08-09 # Pipe the output to "sort -nk3" to get sorted output SUM=0 OVERALL=0 for DIR in `find /proc/ -maxdepth 1 -type d -regex "^/proc/[0-9]+"` do PID=`echo $DIR | cut -d / -f 3` PROGNAME=`ps -p $PID -o comm --no-headers` for SWAP in `grep Swap $DIR/smaps 2>/dev/null | awk '{ print $2 }'` do let SUM=$SUM+$SWAP done if (( $SUM > 0 )); then echo "PID=$PID swapped $SUM KB ($PROGNAME)" fi let OVERALL=$OVERALL+$SUM SUM=0 done echo "Overall swap used: $OVERALL KB"
On 12/10/2013 6:01 AM, Götz Reinicke - IT Koordinator wrote:
recently I noticed, that one of our webservers is using swap space, while there is plenty of physical ram available.
free -m total used free shared buffers cached Mem: 8118 2014 6103 0 85 261 -/+ buffers/cache: 1667 6450 Swap: 8197 77 8119
It's not that much, but why?
during idle time, dirty pages will be written to swap so they can then be discarded if needed. ignore it, it means nothing
Am 10.12.13 17:18, schrieb John R Pierce:
On 12/10/2013 6:01 AM, Götz Reinicke - IT Koordinator wrote:
recently I noticed, that one of our webservers is using swap space, while there is plenty of physical ram available.
free -m total used free shared buffers cached Mem: 8118 2014 6103 0 85 261 -/+ buffers/cache: 1667 6450 Swap: 8197 77 8119
It's not that much, but why?
during idle time, dirty pages will be written to swap so they can then be discarded if needed. ignore it, it means nothing
Hi thanks to all feedbacks, I'f found that the httpd is eating up swap; currently about 500MB while 4GB RAM are still free.
I'll look into that.
Regard . Götz
On 12/11/2013 07:10 AM, Götz Reinicke - IT Koordinator wrote:
Am 10.12.13 17:18, schrieb John R Pierce:
On 12/10/2013 6:01 AM, Götz Reinicke - IT Koordinator wrote:
recently I noticed, that one of our webservers is using swap space, while there is plenty of physical ram available.
free -m total used free shared buffers cached Mem: 8118 2014 6103 0 85 261 -/+ buffers/cache: 1667 6450 Swap: 8197 77 8119
It's not that much, but why?
during idle time, dirty pages will be written to swap so they can then be discarded if needed. ignore it, it means nothing
Hi thanks to all feedbacks, I'f found that the httpd is eating up swap; currently about 500MB while 4GB RAM are still free.
I wouldn't worry about it. It's quite normal for a long-running process like httpd to have a few pages that are used once during startup and never referenced again. Eventually, the kernel moves them out to swap in preference for some extra buffer/cache space. Why you would prefer keeping long- unreferenced pages in RAM is unclear. Of course if it is just a question of why one server is behaving differently from the others, that is a different matter.
--On Tuesday, December 10, 2013 08:18:09 AM -0800 John R Pierce pierce@hogranch.com wrote:
during idle time, dirty pages will be written to swap so they can then be discarded if needed. ignore it, it means nothing
Agreed. If you want to see if paging is actually an issue, run "vmstat -5" and ignore the first line of output. If, over time, you're seeing consistently high values for the si and so columns, then you have something to investigate. If they're usually low or zero, then ignore your swap usage.
Same thing goes for the "free" values in top and vmstat; having low free memory counts in a long running kernel is normal and can be ignored. (Too many people, especially coming from the windows world, get wrapped around the handle about this.)
If you want to maintain a historical record of memory and other performance indicators that you can use for investigative purposes after the fact, see the man page for sar(1).
Devin
Hi,
I have a smiliar problem on a CentOS 6.4 KVM host.
The host has 32 GB memory. virt-top shows: Mem: 30208 MB (30208 MB by guests) vm.swappiness = 0
Guests couldn't use more than 30,2 GB memory!
# uname -r 2.6.32-358.14.1.el6.x86_64
# uptime 13:48:16 up 132 days, 1:55, 3 users, load average: 3.10, 2.45, 1.16
# sysctl vm.swappiness vm.swappiness = 0
# free -m total used free shared buffers cached Mem: 32081 31784 296 0 206 2635 -/+ buffers/cache: 28943 3137 Swap: 16111 3220 12891
Swap usage:
PID=1877 swapped 415092 KB (qemu-kvm) PID=1925 swapped 47680 KB (qemu-kvm) PID=2012 swapped 37188 KB (qemu-kvm) PID=2114 swapped 555560 KB (qemu-kvm) PID=2154 swapped 191832 KB (qemu-kvm) PID=5299 swapped 341108 KB (qemu-kvm) PID=11564 swapped 327620 KB (qemu-kvm) PID=15383 swapped 218360 KB (qemu-kvm) PID=16299 swapped 11280 KB (qemu-kvm) PID=20391 swapped 946656 KB (qemu-kvm) PID=30963 swapped 23040 KB (qemu-kvm) PID=31248 swapped 169680 KB (qemu-kvm) Overall swap used: 3285096 KB
What's going wrong? vm.swappiness ist set to 0 and the system has enough free memory without need of swapping?
-- Chris
2013/12/12 Devin Reade gdr@gno.org
--On Tuesday, December 10, 2013 08:18:09 AM -0800 John R Pierce pierce@hogranch.com wrote:
during idle time, dirty pages will be written to swap so they can then be discarded if needed. ignore it, it means nothing
Agreed. If you want to see if paging is actually an issue, run "vmstat -5" and ignore the first line of output. If, over time, you're seeing consistently high values for the si and so columns, then you have something to investigate. If they're usually low or zero, then ignore your swap usage.
Same thing goes for the "free" values in top and vmstat; having low free memory counts in a long running kernel is normal and can be ignored. (Too many people, especially coming from the windows world, get wrapped around the handle about this.)
If you want to maintain a historical record of memory and other performance indicators that you can use for investigative purposes after the fact, see the man page for sar(1).
Devin
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
# free -m total used free shared buffers cached Mem: 32081 31784 296 0 206 2635 -/+ buffers/cache: 28943 3137 Swap: 16111 3220 12891
free memory without need of swapping?
Not really. The values at the time of that snapshot show that you've just exceeded memory (swap used (3220) > free (3137)). However what you can't see, from this, is other periods of peak load. Maybe you have overnight processing going on that causes extra memory requirements at that time?
You might be able to tell, from "sar" output or similar.
Maybe you had an extra VM running temporarily that has since been shut down?
There could be many reasons for a temporary increase in memory usage.
Once a page has been swapped out then the kernel won't normally swap it back in unless it's needed again. Efficiency; non-requested pages can happily stay on swap and leave RAM free for real activity :-)
Having pages in swap is not indicative of a problem; what's more important is the level of swap _activity_. See "vmstat" output, for example, to determine how much swap activity is occuring. If that's zero then you're not throwing new pages out to swap.
From: Stephen Harris lists@spuddy.org
Having pages in swap is not indicative of a problem; what's more important is the level of swap _activity_. See "vmstat" output, for example, to determine how much swap activity is occuring. If that's zero then you're not throwing new pages out to swap.
Maybe unrelated but we ran into some NUMA memory allocation per cpu problem with our mysql servers. The swap would be used (up to 100%) while there was plenty of free RAM. Prefixing mysqld with "numactl --interleave all" seemed to fix it...
JD
-
Chris -
Devin Reade -
Götz Reinicke - IT Koordinator -
John Doe -
John R Pierce -
Peter Eckel -
Robert Nichols -
Stephen Harris -
Александр Кириллов