php-5.3.3-3.el6_2.6.x86_64 2.6.32-220.4.2.el6.x86_64
After some testing we moved our production PHP intranet site from an old server to a new CentOS 6.2 instance running the mentioned versions. At first it worked well and then user's started to experience a hung site. Upon some digging it appears that the PHP module is getting stuck in a race condition regarding the sessions file -
[root@yannigan-orange fd]# strace -p 22607 Process 22607 attached - interrupt to quit flock(53, LOCK_EX^C <unfinished ...>
where file handle 53 is
... lr-x------. 1 root root 64 Feb 17 15:24 52 -> /opt/informix/msg/en_us/0333/cli.iem lrwx------. 1 root root 64 Feb 17 15:24 53 -> /var/lib/php/session/sess_ulgsl9joplobb9o4fue3n2e1k0 l-wx------. 1 root root 64 Feb 17 15:24 6 -> pipe:[660246] ....
The PHP session file. Originally I had a tmpfs mounted for the session files. I removed that and used the underlying filesystem. Same issue. I disabled APC. Same issue.
Switching the the memcache session handler resolved the issue, although that may be less optimal. But this seems like a PHP bug.
Tips, comments, pointers?
Backtrace of a hung httpd - #0 0x00007fa175af9be7 in flock () from /lib64/libc.so.6 #1 0x00007fa16cb59dfc in ?? () from /etc/httpd/modules/libphp5.so #2 0x00007fa16cb5a047 in ?? () from /etc/httpd/modules/libphp5.so #3 0x00007fa16cb56e4c in php_session_start () from /etc/httpd/modules/libphp5.so #4 0x00007fa16cb57559 in ?? () from /etc/httpd/modules/libphp5.so #5 0x00007fa16cc9afe8 in ?? () from /etc/httpd/modules/libphp5.so #6 0x00007fa16cc72320 in execute () from /etc/httpd/modules/libphp5.so #7 0x00007fa16cc4ca5d in zend_execute_scripts () from /etc/httpd/modules/libphp5.so #8 0x00007fa16cbfacf8 in php_execute_script () from /etc/httpd/modules/libphp5.so #9 0x00007fa16ccd59a5 in ?? () from /etc/httpd/modules/libphp5.so #10 0x00007fa17751da60 in ap_run_handler () #11 0x00007fa17752131e in ap_invoke_handler () #12 0x00007fa17752c990 in ap_process_request () #13 0x00007fa177529858 in ?? () #14 0x00007fa177525568 in ap_run_process_connection () #15 0x00007fa177531767 in ?? () #16 0x00007fa177531a7a in ?? () #17 0x00007fa177531dab in ap_mpm_run () #18 0x00007fa177509900 in main ()
http://debuginfo.centos.org/6/x86_64/
My CentOS server has php-5.3.3-3.el6_2.6.x86_64 but it appears the latest debuginfo package is php-debuginfo-5.3.3-3.el6_2.5.x86_64.rpm
yum doesn't want to install this due to "Package does not match intended download" [yes, I did the obligatory "clean metadata"].
Are debug info packages not current? Does the .6. vs. .5. matter?
On Fri, 2012-02-17 at 16:51 -0500, Adam Tauno Williams wrote:
php-5.3.3-3.el6_2.6.x86_64 2.6.32-220.4.2.el6.x86_64
After some testing we moved our production PHP intranet site from an old server to a new CentOS 6.2 instance running the mentioned versions. At first it worked well and then user's started to experience a hung site. Upon some digging it appears that the PHP module is getting stuck in a race condition regarding the sessions file -
[root@yannigan-orange fd]# strace -p 22607 Process 22607 attached - interrupt to quit flock(53, LOCK_EX^C <unfinished ...>
where file handle 53 is
... lr-x------. 1 root root 64 Feb 17 15:24 52 -> /opt/informix/msg/en_us/0333/cli.iem lrwx------. 1 root root 64 Feb 17 15:24 53 -> /var/lib/php/session/sess_ulgsl9joplobb9o4fue3n2e1k0 l-wx------. 1 root root 64 Feb 17 15:24 6 -> pipe:[660246] ....
The PHP session file. Originally I had a tmpfs mounted for the session files. I removed that and used the underlying filesystem. Same issue. I disabled APC. Same issue.
Switching the the memcache session handler resolved the issue, although that may be less optimal. But this seems like a PHP bug.
Tips, comments, pointers?
Backtrace of a hung httpd - #0 0x00007fa175af9be7 in flock () from /lib64/libc.so.6 #1 0x00007fa16cb59dfc in ?? () from /etc/httpd/modules/libphp5.so #2 0x00007fa16cb5a047 in ?? () from /etc/httpd/modules/libphp5.so #3 0x00007fa16cb56e4c in php_session_start () from /etc/httpd/modules/libphp5.so #4 0x00007fa16cb57559 in ?? () from /etc/httpd/modules/libphp5.so #5 0x00007fa16cc9afe8 in ?? () from /etc/httpd/modules/libphp5.so #6 0x00007fa16cc72320 in execute () from /etc/httpd/modules/libphp5.so #7 0x00007fa16cc4ca5d in zend_execute_scripts () from /etc/httpd/modules/libphp5.so #8 0x00007fa16cbfacf8 in php_execute_script () from /etc/httpd/modules/libphp5.so #9 0x00007fa16ccd59a5 in ?? () from /etc/httpd/modules/libphp5.so #10 0x00007fa17751da60 in ap_run_handler () #11 0x00007fa17752131e in ap_invoke_handler () #12 0x00007fa17752c990 in ap_process_request () #13 0x00007fa177529858 in ?? () #14 0x00007fa177525568 in ap_run_process_connection () #15 0x00007fa177531767 in ?? () #16 0x00007fa177531a7a in ?? () #17 0x00007fa177531dab in ap_mpm_run () #18 0x00007fa177509900 in main ()
-
Adam Tauno Williams