Hi,
I have a centos 5.6 xen vps which loses network connectivity once in a while with following error.
========================================= -bash-3.2# ping 8.8.8.8 PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data. ping: sendmsg: No buffer space available ping: sendmsg: No buffer space available ping: sendmsg: No buffer space available ping: sendmsg: No buffer space available =========================================
All my investigation so far led me to believe that it is because skbuff cache getting full.
=========================================================================
PROC-SLABINFO skbuff_fclone_cache 227 308 512 7 1 : tunables 54 27 8 : slabdata 44 44 0 skbuff_head_cache 1574 1650 256 15 1 : tunables 120 60 8 : slabdata 110 110 0
SLAB-TOP Active / Total Objects (% used) : 2140910 / 2200115 (97.3%) Active / Total Slabs (% used) : 139160 / 139182 (100.0%) Active / Total Caches (% used) : 88 / 136 (64.7%) Active / Total Size (% used) : 512788.94K / 520252.14K (98.6%) Minimum / Average / Maximum Object : 0.02K / 0.24K / 128.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 664000 620290 93% 0.09K 16600 40 66400K buffer_head 409950 408396 99% 0.21K 22775 18 91100K dentry_cache 343056 340307 99% 0.08K 7147 48 28588K selinux_inode_security 338590 336756 99% 0.74K 67718 5 270872K ext3_inode_cache 143665 143363 99% 0.06K 2435 59 9740K size-64 99540 99407 99% 0.25K 6636 15 26544K size-256 96450 96447 99% 0.12K 3215 30 12860K size-128 60858 60858 100% 0.52K 8694 7 34776K radix_tree_node 12420 11088 89% 0.16K 540 23 2160K vm_area_struct 5895 4185 70% 0.25K 393 15 1572K filp 4816 3355 69% 0.03K 43 112 172K size-32 2904 2810 96% 0.09K 66 44 264K sysfs_dir_cache 2058 1937 94% 0.58K 343 6 1372K proc_inode_cache 1728 1215 70% 0.02K 12 144 48K anon_vma 1650 1590 96% 0.25K 110 15 440K skbuff_head_cache 1498 1493 99% 2.00K 749 2 2996K size-2048 1050 1032 98% 0.55K 150 7 600K inode_cache 792 767 96% 1.00K 198 4 792K size-1024 649 298 45% 0.06K 11 59 44K pid 600 227 37% 0.09K 15 40 60K journal_head 590 298 50% 0.06K 10 59 40K delayacct_cache 496 424 85% 0.50K 62 8 248K size-512 413 156 37% 0.06K 7 59 28K fs_cache 404 44 10% 0.02K 2 202 8K biovec-1 390 293 75% 0.12K 13 30 52K bio 327 327 100% 4.00K 327 1 1308K size-4096 320 190 59% 0.38K 32 10 128K ip_dst_cache 308 227 73% 0.50K 44 7 176K skbuff_fclone_cache 258 247 95% 0.62K 43 6 172K sock_inode_cache 254 254 100% 1.84K 127 2 508K task_struct 252 225 89% 0.81K 28 9 224K signal_cache 240 203 84% 0.73K 48 5 192K shmem_inode_cache 204 204 100% 2.06K 68 3 544K sighand_cache 202 4 1% 0.02K 1 202 4K revoke_table 195 194 99% 0.75K 39 5 156K UDP 159 77 48% 0.07K 3 53 12K eventpoll_pwq 145 139 95% 0.75K 29 5 116K files_cache 144 41 28% 0.02K 1 144 4K journal_handle 140 140 100% 0.88K 35 4 140K mm_struct 140 77 55% 0.19K 7 20 28K eventpoll_epi 135 135 100% 2.12K 135 1 540K kmem_cache 121 45 37% 0.69K 11 11 88K UNIX 119 114 95% 0.52K 17 7 68K idr_layer_cache 118 41 34% 0.06K 2 59 8K blkdev_ioc 112 32 28% 0.03K 1 112 4K tcp_bind_bucket 110 56 50% 0.17K 5 22 20K file_lock_cache 106 35 33% 0.07K 2 53 8K avc_node 105 98 93% 1.50K 21 5 168K TCP 105 100 95% 1.04K 15 7 120K bio_map_info 92 1 1% 0.04K 1 92 4K dnotify_cache 80 18 22% 0.19K 4 20 16K tw_sock_TCP 70 44 62% 0.27K 5 14 20K blkdev_requests 59 23 38% 0.06K 1 59 4K biovec-4 59 13 22% 0.06K 1 59 4K fib6_nodes 59 11 18% 0.06K 1 59 4K ip_fib_hash 59 11 18% 0.06K 1 59 4K ip_fib_alias 53 53 100% 0.07K 1 53 4K taskstats_cache 53 1 1% 0.07K 1 53 4K inotify_watch_cache 48 48 100% 0.16K 2 24 8K sigqueue 48 8 16% 0.08K 1 48 4K crq_pool 45 27 60% 0.25K 3 15 12K mnt_cache 45 29 64% 0.25K 3 15 12K dquot 45 32 71% 0.25K 3 15 12K sgpool-8 40 19 47% 0.19K 2 20 8K key_jar 32 32 100% 0.50K 4 8 16K sgpool-16 32 32 100% 1.00K 8 4 32K sgpool-32 32 32 100% 2.00K 16 2 64K sgpool-64 32 32 100% 4.00K 32 1 128K sgpool-128 31 31 100% 8.00K 31 1 248K size-8192 30 27 90% 1.54K 6 5 48K blkdev_queue 30 14 46% 0.12K 1 30 4K request_sock_TCP 30 3 10% 0.12K 1 30 4K inet_peer_cache
FREE-M total used free shared buffers cached Mem: 8192 4821 3370 0 500 2793 -/+ buffers/cache: 1527 6664 Swap: 8191 0 8191
PROC-MEMINFO MemTotal: 8388608 kB MemFree: 3451384 kB Buffers: 512352 kB Cached: 2860580 kB SwapCached: 0 kB Active: 2971812 kB Inactive: 1178908 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 8388608 kB LowFree: 3451384 kB SwapTotal: 8388600 kB SwapFree: 8388600 kB Dirty: 644 kB Writeback: 0 kB AnonPages: 777524 kB Mapped: 24816 kB Slab: 557716 kB PageTables: 16300 kB NFS_Unstable: 0 kB Bounce: 0 kB CommitLimit: 12582904 kB Committed_AS: 1624128 kB VmallocTotal: 34359738367 kB VmallocUsed: 4556 kB VmallocChunk: 34359733771 kB ===========================================================
Can any one provide any insights into a possible solution of this issue.
-- Kind Regards, Sherin