Observing deadlock between SIPhandler and SDL Router in CUCM. Looks like this is fixed in glibc 2.17.162 version, but we see the issue still
https://bugzilla.redhat.com/show_bug.cgi?id=906468
Need help to identify the root cause and add a preventive fix.
Here is the error messages:
*********************************************************************************************************************************************************************************************************************************** gdb) thread 40 [Switching to thread 40 (Thread 0xaca90b40 (LWP 5935))] #3 0xf67fdbd1 in malloc () from /usr/lib/libc.so.6 (gdb) bt #0 0xf775a430 in __kernel_vsyscall () #1 0xf688d462 in __lll_lock_wait_private () from /usr/lib/libc.so.6 #2 0xf68006c0 in _L_lock_14246 () from /usr/lib/libc.so.6 #3 0xf67fdbd1 in malloc () from /usr/lib/libc.so.6 #4 0xf6a68907 in operator new(unsigned int) () from /usr/local/cm/lib/libstlport.so.5.2 #5 0x0936c3f9 in operator() (this=0xaca8dd70) at /view/BLD-cm_12_5_1-cct-ccm-d/vob/ccm/Common/Include/Utilities/Member.hpp:52 #6 c_str (this=0xaca8dd70) at /view/BLD-cm_12_5_1-cct-ccm-d/vob/ccm/Common/Include/CallManager/TDCLCpShares.hpp:862 #7 assign (len=12, digits=0xaca8de34 "+19570676705", this=0xaca8dd70) at /view/BLD-cm_12_5_1-cct-ccm-d/vob/ccm/Common/Include/CallManager/TDCLCpShares.hpp:1252 #8 CdPNumStringVar::CdPNumStringVar (this=0xaca8dd70, digits=0xaca8de34 "+19570676705", len=12) at /view/BLD-cm_12_5_1-cct-ccm-d/vob/ccm/Common/Include/CallManager/TDCLCpShares.hpp:846 #9 0x092efdf8 in SIPCdpc::processSIPSetupMsg (this=this@entry=0x7de0c8e0, sipContainerWrapper=sipContainerWrapper@entry=0x8e966e78) at ProcessSIPCdpc.cpp:2142 #10 0x0934086d in SIPCdpc::processSIPSetupInd (this=this@entry=0x7de0c8e0) at ProcessSIPCdpc.cpp:3300 #11 0x0934ec08 in SIPCdpc::inCall_waitRSVPRes_PolicyAndCACRegisterRes (this=0x7de0c8e0, s= Name : "PolicyAndCACRegisterRes", Sender : spy:PID (8,100,152,1) "ReservationMgr", Receiver : spy:PID (8,100,180,995813) "SIPCdpc", Priority : kNormalPriority) at ProcessSIPCdpc.cpp:26471 #12 0x09e69c7b in SdlProcessBase::inputSignal (this=0x7de0c8e0, rSignal=0x807d8d80, traceType=SdlSystemLog::SignalRouterThread, highPriority=0, normalPriority=0, lowPriority=0, veryLowPriority=0, lazyPriority=0, dbUpdatePriority=0) at SdlProcessBase.cpp:417 #13 0x09e6ce8a in SdlRouter::callProcess (this=this@entry=0xcfff0e0, _sdlSignal=_sdlSignal@entry=0x807d8d80, _deleteSignal=@0xaca8fdb3: true, _traceType=_traceType@entry=SdlSystemLog::SignalRouterThread, _hp=0, _np=0, _lp=0, _vlp=0, _lzp=0, _dbp=0) at SdlRouter.cpp:257 #14 0x09e6d60c in SdlRouter::scheduler (this=this@entry=0xcfff0e0) at SdlRouter.cpp:164 #15 0x09e6dcad in SdlRouter::schedulerInit (sdlRouter=0xcfff0e0) at SdlRouter.cpp:107 #16 0xf69beb3c in start_thread () from /usr/lib/libpthread.so.0 #17 0xf687f44e in clone () from /usr/lib/libc.so.6
(gdb) thread 1 [Switching to thread 1 (Thread 0xa9136b40 (LWP 5964))] #0 0xf67fb876 in _int_malloc () from /usr/lib/libc.so.6 (gdb) bt #0 0xf67fb876 in _int_malloc () from /usr/lib/libc.so.6 #1 0xf67fdbda in malloc () from /usr/lib/libc.so.6 #2 0xf6a68907 in operator new(unsigned int) () from /usr/local/cm/lib/libstlport.so.5.2 #3 0x09cd4ce2 in sip_get_sipspi_message () at sip_common_spi.cpp:20059 #4 0x09cf94d1 in sipTransportPostSendMessage (msg=0x8a20a150, gcb=gcb@entry=0x8015d208, tcb_context=..., connId=24, destAddr=0x8015d228, destPort=5061, transport=SIP_TRANSPORT_TLS_OVER_TCP, emit_debugs=1 '\001', ip_sig_tos=0 '\000') at sip_common_transport.cpp:3032 #5 0x09cf9946 in sipTransportLogicSendMsg (gcb=gcb@entry=0x8015d208, sw_transport=sw_transport@entry=0 '\000') at sip_common_transport.cpp:2367 #6 0x09cfab6f in sipSPITransportSendMessage (msg=msg@entry=0x8a20a150, context=context@entry=0x8015d208, addr=addr@entry=0xa91355dc, port=port@entry=5061, sentBy_port=sentBy_port@entry=0, is_request=is_request@entry=1 '\001', transport=transport@entry=SIP_TRANSPORT_TLS_OVER_TCP, sw_transport=0 '\000', func=func@entry=0x9cb0080 <sentAck(_sipGenericCB*)>, tcb_context=...) at sip_common_transport.cpp:3323 #7 0x09cbe638 in sipSPISendAck (ccb=0x8015d208, flags=flags@entry=0, func=0x9cb0080 <sentAck(_sipGenericCB*)>) at sip_common_send_msg.cpp:2992 #8 0x09ccc910 in sipSPISendAckForMidCallInvite (ccb=0x8015d208, flag=flag@entry=0) at sip_common_spi.cpp:8745 #9 0x09ccdb37 in sact_sent_mid_invite_new_message_response (ccb=0x8015d208, response=0x89305d58) at sip_common_spi.cpp:9278 #10 0x09cda815 in ccsip_spi_process_event (freeEvent=0xa91359df "\001", event=0x7ca43348) at sip_common_spi.cpp:19333 #11 ccsip_process_sipspi_queue_event (event=event@entry=0x7ca43348) at sip_common_spi.cpp:19997 #12 0x093ef758 in SIPHandler::wait_SIPNormalizeRes (this=0xac1c05e8, s= Name : "SIPNormalizeRes", Sender : spy:PID (8,100,184,1) "SIPNormalization", Receiver : spy:PID (8,100,183,1) "SIPHandler", Priority : kNormalPriority) at ProcessSIPHandler.cpp:1683 #13 0x09e69c7b in SdlProcessBase::inputSignal (this=this@entry=0xac1c05e8, rSignal=rSignal@entry=0x7d2f8a78, traceType=SdlSystemLog::SignalThreadedWithPriorities, highPriority=0, normalPriority=0, lowPriority=0, veryLowPriority=0, lazyPriority=0, dbUpdatePriority=0) at SdlProcessBase.cpp:417 #14 0x09e8d2fd in SdlThreadedProcess::threadQueueReader (this=this@entry=0xac1c05e8) at SdlThreadedProcess.cpp:110 #15 0x09e8d56d in SdlThreadedProcess::threadQueueReaderInit (sdlThreadedProcess=0xac1c05e8) at SdlThreadedProcess.cpp:75 #16 0xf69beb3c in start_thread () from /usr/lib/libpthread.so.0 #17 0xf687f44e in clone () from /usr/lib/libc.so.6
BOth the threads are allocated memory at the same time , eventually resulting in deadlock.
-Thanks