[CentOS] Memcache timeouts?

Fri Mar 22 14:37:23 UTC 2013
Rafał Radecki <radecki.rafal at gmail.com>

Hi All.

I am currently using memcache daemon in version 1.4.15 on one of my
servers. The second one is executing php scripts which use mentioned
memcache daemon (php53-pecl-memcache-2.2.5).

Memcache server:
CentOS release 6.3 (Final)
2.6.32-279.5.2.el6.centos.plus.x86_64
uptime 184 days

PHP server:
CentOS release 6.3 (Final)
2.6.32-279.19.1.el6.centos.plus.x86_64
uptime 43 days

The memcache server gets about:
- 250Mb/s traffic
- 4500 connections/s

On PHP server I get sometimes timeouts, ~20-30 daily. Both servers are
not swapping, they have free memory, cpu is ~40% used max. But I have
found some errors in netstat -s:

Memcache server:
TCP
7251 failed connection attempts (incrementing)
56447 connection resets received (incrementing)
TcpExt
36 packets pruned from receive queue because of socket buffer overrun
(not incrementing)
2820 packets collapsed in receive queue due to low socket buffer (not
incrementing)
459479 connections reset due to unexpected data (incrementing)

PHP server:
TCP
540 failed connection attempts (incrementing)
381066 connection resets received (incrementing)
TcpExt
7194 packets pruned from receive queue because of socket buffer
overrun (incrementing)
250104 packets collapsed in receive queue due to low socket buffer
(incrementing)
2447931 connections reset due to unexpected data (incrementing)

I have tuned tcp/ip a bit:

net.ipv4.tcp_fin_timeout = 5
net.ipv4.tcp_tw_reuse = 1
net.core.somaxconn = 1024
net.ipv4.tcp_max_syn_backlog = 4196
net.core.netdev_max_backlog = 4196
net.ipv4.tcp_sack = 0
net.ipv4.ip_local_port_range = 8192 65534

net.core.rmem_max = 8388608
net.core.wmem_max = 8388608
net.ipv4.tcp_rmem = 4096 65536 8388608
net.ipv4.tcp_wmem = 4096 65536 8388608

and set txqueuelen for interfaces (1Gb/s) to 5000.

but where to look next? May it be so that the php client version is
buggy? Any other tips?

Best regards,
Rafal.