<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; " class=""><div style=""><br class="Apple-interchange-newline" style=""><blockquote type="cite"><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0; ">As mentioned before, IO could give such strange results. I suggest launching<span class="Apple-converted-space"> </span><br class="" style="">dstat with logging to a file, and analyzing the file afterwards.<br class="" style=""></span></blockquote></div><br class="" style=""><div class="" style="">Thanks, much appreciated! </div><div class="" style=""><br class="webkit-block-placeholder"></div><div class="" style="">This has yielded some interesting data, which I'll attempt to include a few seconds before and after one of these events occurred.</div><div class="" style=""><br></div><div class="" style=""><span class="Apple-tab-span" style="white-space:pre"> </span>system interrupts per second: Note the ~200x jump to almost 200,000 interrupts per second.</div><div class="" style=""><span class="Apple-tab-span" style="white-space:pre"> </span>2907</div><div class="" style=""><span class="Apple-tab-span" style="white-space:pre"> </span>6714</div><div class="" style=""><span class="Apple-tab-span" style="white-space:pre"> </span>1371</div><div class="" style=""><span class="Apple-tab-span" style="white-space:pre"> </span>194218</div><div class="" style=""><span class="Apple-tab-span" style="white-space:pre"> </span>2456</div><div class="" style=""><span class="Apple-tab-span" style="white-space:pre"> </span>2907<br class="webkit-block-placeholder"></div><div class="" style=""><br></div><div class="" style=""><span class="Apple-tab-span" style="white-space:pre"> </span>network received: <span class="Apple-style-span" style="border-collapse: collapse; ">Note the network received ramps up over 5 seconds, peaks at ~50x background, and ramps back down in about 3 seconds. The peak is from the same sample as the 200x sample above.</span></div><div class="" style=""><span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-style-span" style="border-collapse: collapse; ">108784</span></div><div class="" style=""><span class="Apple-style-span" style="border-collapse: collapse;"><span class="Apple-tab-span" style="white-space:pre"> </span>389794</span></div><div class="" style=""><span class="Apple-style-span" style="border-collapse: collapse;"><span class="Apple-tab-span" style="white-space:pre"> </span>1070850</span></div><div class="" style=""><span class="Apple-style-span" style="border-collapse: collapse;"><span class="Apple-tab-span" style="white-space:pre"> </span>4843956</span></div><div class="" style=""><span class="Apple-style-span" style="border-collapse: collapse;"><span class="Apple-tab-span" style="white-space:pre"> </span>352226</span></div><div class="" style=""><span class="Apple-style-span" style="border-collapse: collapse;"><span class="Apple-tab-span" style="white-space:pre"> </span>353102</span></div><div class="" style=""><span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-style-span" style="border-collapse: collapse; ">96392</span></div><div class="" style=""><span class="Apple-style-span" style="border-collapse: collapse;"><br class="webkit-block-placeholder"></span></div><div class="" style=""><span class="Apple-style-span" style="border-collapse: collapse;"><br class="webkit-block-placeholder"></span></div><div class="" style=""><span class="Apple-style-span" style="border-collapse: collapse;">Everything else looks sane -- there's enough ram, nothing's being swapped out, etc. This is on a private-network server that has a load balancer in front of it, so if it's network related, it wouldn't be misdirected random bits.</span></div><div class="" style=""><span class="Apple-style-span" style="border-collapse: collapse;"><br class="webkit-block-placeholder"></span></div><div class="" style=""><span class="Apple-style-span" style="border-collapse: collapse;">Has anyone seen this sort of behavior before? What was the cause? What should I do to figure out how to keep the load averages from flipping out of control? </span></div><div class="" style=""><span class="Apple-style-span" style="border-collapse: collapse;"><br class="webkit-block-placeholder"></span></div><div class="" style=""><span class="Apple-style-span" style="border-collapse: collapse;">(This isn't something as lame as a counter rolling over somewhere internal to the kernel, is it? Wouldn't think so, but thought to ask. Running <span class="Apple-style-span" style="border-collapse: separate; "><span class="Apple-style-span" style="border-collapse: collapse;">2.6.18-8.1.8.el5. We could reboot to run </span><span class="Apple-style-span" style="border-collapse: collapse; "><span class="Apple-style-span" style="border-collapse: separate; "><span class="Apple-style-span" style="border-collapse: collapse;">2.6.18-8.1.15 if that'd be a potential fix.</span><span class="Apple-style-span" style="border-collapse: collapse; ">)</span></span></span></span></span></div><div class="" style=""><span class="Apple-style-span" style="border-collapse: collapse;"><br class="webkit-block-placeholder"></span></div><div class="" style=""><span class="Apple-style-span" style="border-collapse: collapse;">Thanks for any insight!</span></div><div class="" style=""><span class="Apple-style-span" style="border-collapse: collapse;"><br class="webkit-block-placeholder"></span></div><div class="" style=""><span class="Apple-style-span" style="border-collapse: collapse;">best,</span></div><div class="" style=""><span class="Apple-style-span" style="border-collapse: collapse;">Jeff</span></div><div class="" style=""><span class="Apple-style-span" style="border-collapse: collapse;"><br class="webkit-block-placeholder"></span></div><div class="" style=""><span class="Apple-style-span" style="border-collapse: collapse;"><br class="webkit-block-placeholder"></span></div><div class="" style=""><br class="webkit-block-placeholder"></div><div class="" style=""><br class="webkit-block-placeholder" style=""></div><div class="" style="">
<table border="0" cellpadding="0" cellspacing="0" width="600" style="border-collapse: collapse; position: static; z-index: auto; ">
<!--StartFragment-->
<col width="75" span="8">
<tbody><tr height="13">
<td height="13" colspan="2" width="150">total cpu usage</td>
<td width="75">dsk/total</td>
<td width="75"></td>
<td width="75">net/total</td>
<td width="75"></td>
<td width="75">system</td>
<td width="75"></td>
</tr>
<tr height="13">
<td height="13">usr</td>
<td>sys</td>
<td>read</td>
<td>writ</td>
<td>recv</td>
<td>send</td>
<td>int</td>
<td>csw</td>
</tr>
<tr height="13">
<td height="13" align="right" x:num="10.5">10.5</td>
<td align="right" x:num="3.25">3.25</td>
<td align="right" x:num="0.0">0</td>
<td align="right" x:num="409600.0">409600</td>
<td align="right" x:num="108784.0">108784</td>
<td align="right" x:num="72286.0">72286</td>
<td align="right" x:num="2907.0">2907</td>
<td align="right" x:num="20376.0">20376</td>
</tr>
<tr height="13">
<td height="13" align="right" x:num="3.99">3.99</td>
<td align="right" x:num="2.993">2.993</td>
<td align="right" x:num="0.0">0</td>
<td align="right" x:num="319488.0">319488</td>
<td align="right" x:num="389794.0">389794</td>
<td align="right" x:num="661170.0">661170</td>
<td align="right" x:num="6714.0">6714</td>
<td align="right" x:num="23941.0">23941</td>
</tr>
<tr height="13">
<td height="13" align="right" x:num="0.25">0.25</td>
<td align="right" x:num="0.25">0.25</td>
<td align="right" x:num="0.0">0</td>
<td align="right" x:num="720896.0">720896</td>
<td align="right" x:num="1.07085E6">1070850</td>
<td align="right" x:num="1.18972E6">1189720</td>
<td align="right" x:num="1371.0">1371</td>
<td align="right" x:num="16648.0">16648</td>
</tr>
<tr height="13">
<td height="13" align="right" x:num="9.166999999999999">9.167</td>
<td align="right" x:num="90.44199999999999">90.442</td>
<td align="right" x:num="12288.0">12288</td>
<td align="right" x:num="1.122304E6">1122304</td>
<td align="right" x:num="4.843956E6">4843956</td>
<td align="right" x:num="399998.0">399998</td>
<td align="right" x:num="194218.0">194218</td>
<td align="right" x:num="55433.0">55433</td>
</tr>
<tr height="13">
<td height="13" align="right" x:num="56.93099999999999">56.931</td>
<td align="right" x:num="16.832">16.832</td>
<td align="right" x:num="0.0">0</td>
<td align="right" x:num="1.273856E6">1273856</td>
<td align="right" x:num="352226.0">352226</td>
<td align="right" x:num="334506.0">334506</td>
<td align="right" x:num="2456.0">2456</td>
<td align="right" x:num="12844.0">12844</td>
</tr>
<tr height="13">
<td height="13" align="right" x:num="46.24999999999999">46.25</td>
<td align="right" x:num="20.0">20</td>
<td align="right" x:num="0.0">0</td>
<td align="right" x:num="454656.0">454656</td>
<td align="right" x:num="353102.0">353102</td>
<td align="right" x:num="384496.0">384496</td>
<td align="right" x:num="2907.0">2907</td>
<td align="right" x:num="20631.0">20631</td>
</tr>
<tr height="13">
<td height="13" align="right" x:num="24.25">24.25</td>
<td align="right" x:num="1.25">1.25</td>
<td align="right" x:num="0.0">0</td>
<td align="right" x:num="3.260416E6">3260416</td>
<td align="right" x:num="96392.0">96392</td>
<td align="right" x:num="72316.0">72316</td>
<td align="right" x:num="1342.0">1342</td>
<td align="right" x:num="17307.0">17307</td>
</tr>
<tr height="13">
<td height="13" align="right" x:num="23.25">23.25</td>
<td align="right" x:num="2.25">2.25</td>
<td align="right" x:num="0.0">0</td>
<td align="right" x:num="610304.0">610304</td>
<td align="right" x:num="91086.0">91086</td>
<td align="right" x:num="71194.0">71194</td>
<td align="right" x:num="1458.0">1458</td>
<td align="right" x:num="17584.0">17584</td>
</tr>
<tr height="13">
<td height="13" align="right" x:num="10.973">10.973</td>
<td align="right" x:num="1.496">1.496</td>
<td align="right" x:num="0.0">0</td>
<td align="right" x:num="0.0">0</td>
<td align="right" x:num="84192.0">84192</td>
<td align="right" x:num="46276.0">46276</td>
<td align="right" x:num="1349.0">1349</td>
<td align="right" x:num="18135.0">18135</td>
</tr>
<tr height="13">
<td height="13" align="right" x:num="0.0">0</td>
<td align="right" x:num="0.0">0</td>
<td align="right" x:num="0.0">0</td>
<td align="right" x:num="94208.0">94208</td>
<td align="right" x:num="71892.0">71892</td>
<td align="right" x:num="33304.0">33304</td>
<td align="right" x:num="1220.0">1220</td>
<td align="right" x:num="16979.0">16979</td>
</tr>
<tr height="13">
<td height="13" align="right" x:num="0.25">0.25</td>
<td align="right" x:num="0.25">0.25</td>
<td align="right" x:num="0.0">0</td>
<td align="right" x:num="126976.0">126976</td>
<td align="right" x:num="71184.0">71184</td>
<td align="right" x:num="47576.0">47576</td>
<td align="right" x:num="1268.0">1268</td>
<td align="right" x:num="16973.0">16973</td>
</tr>
<!--EndFragment-->
</tbody></table>
</div></body></html>