Observing deadlock between SIPhandler and SDL Router in CUCM. - devel

25 Jan 2019


      Observing deadlock between SIPhandler and SDL Router in CUCM.
Looks like this is fixed in glibc 2.17.162 version, but we see the issue still
https://bugzilla.redhat.com/show_bug.cgi?id=906468
Need help to identify the  root cause and add a preventive fix.
Here is the error messages:
***********************************************************************************************************************************************************************************************************************************
 gdb) thread 40
[Switching to thread 40 (Thread 0xaca90b40 (LWP 5935))]
#3 0xf67fdbd1 in malloc () from /usr/lib/libc.so.6
(gdb) bt
#0 0xf775a430 in __kernel_vsyscall ()
#1 0xf688d462 in __lll_lock_wait_private () from /usr/lib/libc.so.6
#2 0xf68006c0 in _L_lock_14246 () from /usr/lib/libc.so.6
#3 0xf67fdbd1 in malloc () from /usr/lib/libc.so.6
#4 0xf6a68907 in operator new(unsigned int) () from /usr/local/cm/lib/libstlport.so.5.2
#5 0x0936c3f9 in operator() (this=0xaca8dd70) at /view/BLD-cm_12_5_1-cct-ccm-d/vob/ccm/Common/Include/Utilities/Member.hpp:52
#6 c_str (this=0xaca8dd70) at /view/BLD-cm_12_5_1-cct-ccm-d/vob/ccm/Common/Include/CallManager/TDCLCpShares.hpp:862
#7 assign (len=12, digits=0xaca8de34 "+19570676705", this=0xaca8dd70) at /view/BLD-cm_12_5_1-cct-ccm-d/vob/ccm/Common/Include/CallManager/TDCLCpShares.hpp:1252
#8 CdPNumStringVar::CdPNumStringVar (this=0xaca8dd70, digits=0xaca8de34 "+19570676705", len=12) at /view/BLD-cm_12_5_1-cct-ccm-d/vob/ccm/Common/Include/CallManager/TDCLCpShares.hpp:846
#9 0x092efdf8 in SIPCdpc::processSIPSetupMsg (this=this@entry=0x7de0c8e0, sipContainerWrapper=sipContainerWrapper@entry=0x8e966e78) at ProcessSIPCdpc.cpp:2142
#10 0x0934086d in SIPCdpc::processSIPSetupInd (this=this@entry=0x7de0c8e0) at ProcessSIPCdpc.cpp:3300
#11 0x0934ec08 in SIPCdpc::inCall_waitRSVPRes_PolicyAndCACRegisterRes (this=0x7de0c8e0, s=
Name : "PolicyAndCACRegisterRes",
Sender : spy:PID (8,100,152,1) "ReservationMgr",
Receiver : spy:PID (8,100,180,995813) "SIPCdpc",
Priority : kNormalPriority) at ProcessSIPCdpc.cpp:26471
#12 0x09e69c7b in SdlProcessBase::inputSignal (this=0x7de0c8e0, rSignal=0x807d8d80, traceType=SdlSystemLog::SignalRouterThread, highPriority=0, normalPriority=0, lowPriority=0, veryLowPriority=0, lazyPriority=0,
dbUpdatePriority=0) at SdlProcessBase.cpp:417
#13 0x09e6ce8a in SdlRouter::callProcess (this=this@entry=0xcfff0e0, _sdlSignal=_sdlSignal@entry=0x807d8d80, _deleteSignal=@0xaca8fdb3: true, _traceType=_traceType@entry=SdlSystemLog::SignalRouterThread, _hp=0,
_np=0, _lp=0, _vlp=0, _lzp=0, _dbp=0) at SdlRouter.cpp:257
#14 0x09e6d60c in SdlRouter::scheduler (this=this@entry=0xcfff0e0) at SdlRouter.cpp:164
#15 0x09e6dcad in SdlRouter::schedulerInit (sdlRouter=0xcfff0e0) at SdlRouter.cpp:107
#16 0xf69beb3c in start_thread () from /usr/lib/libpthread.so.0
#17 0xf687f44e in clone () from /usr/lib/libc.so.6
(gdb) thread 1
[Switching to thread 1 (Thread 0xa9136b40 (LWP 5964))]
#0 0xf67fb876 in _int_malloc () from /usr/lib/libc.so.6
(gdb) bt
#0 0xf67fb876 in _int_malloc () from /usr/lib/libc.so.6
#1 0xf67fdbda in malloc () from /usr/lib/libc.so.6
#2 0xf6a68907 in operator new(unsigned int) () from /usr/local/cm/lib/libstlport.so.5.2
#3 0x09cd4ce2 in sip_get_sipspi_message () at sip_common_spi.cpp:20059
#4 0x09cf94d1 in sipTransportPostSendMessage (msg=0x8a20a150, gcb=gcb@entry=0x8015d208, tcb_context=..., connId=24, destAddr=0x8015d228, destPort=5061, transport=SIP_TRANSPORT_TLS_OVER_TCP, emit_debugs=1 '\001',
ip_sig_tos=0 '\000') at sip_common_transport.cpp:3032
#5 0x09cf9946 in sipTransportLogicSendMsg (gcb=gcb@entry=0x8015d208, sw_transport=sw_transport@entry=0 '\000') at sip_common_transport.cpp:2367
#6 0x09cfab6f in sipSPITransportSendMessage (msg=msg@entry=0x8a20a150, context=context@entry=0x8015d208, addr=addr@entry=0xa91355dc, port=port@entry=5061, sentBy_port=sentBy_port@entry=0,
is_request=is_request@entry=1 '\001', transport=transport@entry=SIP_TRANSPORT_TLS_OVER_TCP, sw_transport=0 '\000', func=func@entry=0x9cb0080 <sentAck(_sipGenericCB*)>, tcb_context=...)
at sip_common_transport.cpp:3323
#7 0x09cbe638 in sipSPISendAck (ccb=0x8015d208, flags=flags@entry=0, func=0x9cb0080 <sentAck(_sipGenericCB*)>) at sip_common_send_msg.cpp:2992
#8 0x09ccc910 in sipSPISendAckForMidCallInvite (ccb=0x8015d208, flag=flag@entry=0) at sip_common_spi.cpp:8745
#9 0x09ccdb37 in sact_sent_mid_invite_new_message_response (ccb=0x8015d208, response=0x89305d58) at sip_common_spi.cpp:9278
#10 0x09cda815 in ccsip_spi_process_event (freeEvent=0xa91359df "\001", event=0x7ca43348) at sip_common_spi.cpp:19333
#11 ccsip_process_sipspi_queue_event (event=event@entry=0x7ca43348) at sip_common_spi.cpp:19997
#12 0x093ef758 in SIPHandler::wait_SIPNormalizeRes (this=0xac1c05e8, s=
Name : "SIPNormalizeRes",
Sender : spy:PID (8,100,184,1) "SIPNormalization",
Receiver : spy:PID (8,100,183,1) "SIPHandler",
Priority : kNormalPriority) at ProcessSIPHandler.cpp:1683
#13 0x09e69c7b in SdlProcessBase::inputSignal (this=this@entry=0xac1c05e8, rSignal=rSignal@entry=0x7d2f8a78, traceType=SdlSystemLog::SignalThreadedWithPriorities, highPriority=0, normalPriority=0, lowPriority=0,
veryLowPriority=0, lazyPriority=0, dbUpdatePriority=0) at SdlProcessBase.cpp:417
#14 0x09e8d2fd in SdlThreadedProcess::threadQueueReader (this=this@entry=0xac1c05e8) at SdlThreadedProcess.cpp:110
#15 0x09e8d56d in SdlThreadedProcess::threadQueueReaderInit (sdlThreadedProcess=0xac1c05e8) at SdlThreadedProcess.cpp:75
#16 0xf69beb3c in start_thread () from /usr/lib/libpthread.so.0
#17 0xf687f44e in clone () from /usr/lib/libc.so.6
BOth the threads are allocated memory at the same time , eventually resulting in deadlock.
-Thanks