I've been getting LOTS of messages like the below in the daily log, and from all indications, it appears to all be related to the cpu; the machine is just over a year old, and was the old vortex.wa4phy;net server from the downtown co-lo site. Aside from huge log files, and lots of other fluff, numerous problems of other nature have started cropping up. Anyone have any suggestions as to what to do besides make a boat anchor out of it? It's all greek to me, so I'm totally at the mercy of those folks who understand the bits and bites (lit) that have run amok in things. Any suggestions as a possible fault other than the cpu just becoming more toasty brown? :)
Thanks... Sam
Begin log snip
--------------------- Kernel Begin ------------------------
1 Time(s): PROTO=TCP SPT=388 DPT=53395 WINDOW=1448 RES=0x00 ACK URGP=0 1 Time(s): URGP=0 1 Time(s): WINDOW=1448 RES=0x00 ACK URGP=0 1 Time(s): 0 PREC=0x00 TTL=49 ID=37550 DF PROTO=TCP SPT=388 DPT=53395 WINDOW=1448 RES=0x00 ACK URGP=0 1 Time(s): 0 PREC=0x00 TTL=49 ID=56466 DF PROTO=TCP SPT=388 DPT=53395 WINDOW=1448 RES=0x00 ACK URGP=0 1 Time(s): 0 SRC=216.104.158.222 DST=165.91.140.32 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=16424 DF PROTO=TCP SPT=53395 DPT=388 WINDOW=2534 RES=0x00 ACK URGP=0 1 Time(s): 04.158.222 LEN=1500 TOS=0x00 PREC=0x00 TTL=49 ID=48422 DF PROTO=TCP SPT=388 DPT=53395 WINDOW=1448 RES=0x00 ACK URGP=0 1 Time(s): 0x00 TTL=49 ID=4348 DF PROTO=TCP SPT=388 DPT=53395 WINDOW=1448 RES=0x00 ACK URGP=0 1 Time(s): 1.140.32 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=19772 DF PROTO=TCP SPT=53395 DPT=388 WINDOW=2534 RES=0x00 ACK URGP=0 1 Time(s): 20:20:08:00 SRC=165.91.140.32 DST=216.104.158.222 LEN=1500 TOS=0x00 PREC=0x00 TTL=49 ID=32346 DF PROTO=TCP SPT=388 DPT=53395 WINDOW=1448 RES=0x00 ACK URGP=0 1 Time(s): 22 LEN=1500 TOS=0x00 PREC=0x00 TTL=49 ID=60844 DF PROTO=TCP SPT=388 DPT=53395 WINDOW=1448 RES=0x00 ACK URGP=0 1 Time(s): 2860 DF PROTO=TCP SPT=388 DPT=53395 WINDOW=1448 RES=0x00 ACK URGP=0 1 Time(s): 48 RES=0x00 ACK URGP=0 1 Time(s): 48:2e:69:48:00:01:97:00:20:20:08:00 SRC=165.91.140.32 DST=216.104.158.222 LEN=1500 TOS=0x00 PREC=0x00 TTL=49 ID=49338 DF PROTO=TCP SPT=388 DPT=53395 WINDOW=1448 RES=0x00 ACK URGP=0 1 Time(s): 500 TOS=0x00 PREC=0x00 TTL=49 ID=44842 DF PROTO=TCP SPT=388 DPT=53395 WINDOW=1448 RES=0x00 ACK URGP=0 1 Time(s): 63 RES=0x00 ACK URGP=0 1 Time(s): 890 DF PROTO=TCP SPT=388 DPT=53395 WINDOW=1448 RES=0x00 ACK URGP=0 1 Time(s): 91.140.32 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=47322 DF PROTO=TCP SPT=53395 DPT=388 WINDOW=2263 RES=0x00 ACK URGP=0 1 Time(s): 9:48:00:01:97:00:20:20:08:00 SRC=165.91.140.32 DST=216.104.158.222 LEN=1500 TOS=0x00 PREC=0x00 TTL=49 ID=43080 DF PROTO=TCP SPT=388 DPT=53395 WINDOW=1448 RES=0x00 ACK URGP=0 1 Time(s): :00:20:20:08:00 SRC=165.91.140.32 DST=216.104.158.222 LEN=1500 TOS=0x00 PREC=0x00 TTL=49 ID=12262 DF PROTO=TCP SPT=388 DPT=53395 WINDOW=1448 RES=0x00 ACK URGP=0 1 Time(s): :08:00 SRC=165.91.140.32 DST=216.104.158.222 LEN=1500 TOS=0x00 PREC=0x00 TTL=49 ID=12384 DF PROTO=TCP SPT=388 DPT=53395 WINDOW=1448 RES=0x00 ACK URGP=0 1 Time(s): :30:48:2e:69:48:00:01:97:00:20:20:08:00 SRC=165.91.140.32 DST=216.104.158.222 LEN=52 TOS=0x00 PREC=0x00 TTL=49 ID=40706 DF PROTO=TCP SPT=388 DPT=53395 WINDOW=1448 RES=0x00 ACK URGP=0 1 Time(s): < PROTO=TCP SPT=388 DPT=53395 WINDOW=1448 RES=0x00 ACK URGP=0 1 Time(s): < SPT=388 DPT=53395 WINDOW=1448 RES=0x00 ACK URGP=0 1 Time(s): <0 ACK URGP=0 1 Time(s): <00 TTL=49 ID=39518 DF PROTO=TCP SPT=388 DPT=53395 WINDOW=1448 RES=0x00 ACK URGP=0 1 Time(s): <165.91.140.32 DST=216.104.158.222 LEN=1500 TOS=0x00 PREC=0x00 TTL=49 ID=57258 DF PROTO=TCP SPT=388 DPT=53395 WINDOW=1448 RES=0x00 ACK URGP=0 1 Time(s): <388 DPT=53395 WINDOW=1448 RES=0x00 ACK URGP=0 1 Time(s): <4 ID=24199 DF PROTO=TCP SPT=53395 DPT=388 WINDOW=2534 RES=0x00 ACK URGP=0 1 Time(s): <5.91.140.32 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=9616 DF PROTO=TCP SPT=53395 DPT=388 WINDOW=2534 RES=0x00 ACK URGP=0 1 Time(s): <7 1 Time(s): <7 WINDOW=1448 RES=0x00 ACK URGP=0 1 Time(s): <7.104.158.222 DST=165.91.140.32 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49890 DF PROTO=TCP SPT=53395 DPT=388 WINDOW=2534 RES=0x00 ACK URGP=0 1 Time(s): <7.91.140.32 DST=216.104.158.222 LEN=52 TOS=0x00 PREC=0x00 TTL=49 ID=51756 DF PROTO=TCP SPT=388 DPT=53395 WINDOW=1448 RES=0x00 ACK URGP=0 1 Time(s): <7104.158.222 DST=165.91.140.32 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=38046 DF PROTO=TCP SPT=53395 DPT=388 WINDOW=2534 RES=0x00 ACK URGP=0 1 Time(s): <7140.32 DST=216.104.158.222 LEN=1500 TOS=0x00 PREC=0x00 TTL=49 ID=43334 DF PROTO=TCP SPT=388 DPT=53395 WINDOW=1448 RES=0x00 ACK URGP=0 1 Time(s): <722 DST=165.91.140.32 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=34918 DF PROTO=TCP SPT=53395 DPT=388 WINDOW=2534 RES=0x00 ACK URGP=0 1 Time(s): <7222 LEN=1500 TOS=0x00 PREC=0x00 TTL=49 ID=37676 DF PROTO=TCP SPT=388 DPT=53395 WINDOW=1448 RES=0x00 ACK URGP=0 1 Time(s): <72e:69:48:00:01:97:00:20:20:08:00 SRC=165.91.140.32 DST=216.104.158.222 LEN=1500 TOS=0x00 PREC=0x00 TTL=49 ID=58174 DF PROTO=TCP SPT=388 DPT=53395 WINDOW=1448 RES=0x00 ACK URGP=0 1 Time(s): <73126 DF PROTO=TCP SPT=388 DPT=53395 WINDOW=1448 RES=0x00 ACK URGP=0 1 Time(s): <753395 WINDOW=1448 RES=0x00 ACK URGP=0 1 Time(s): <78 RES=0x00 ACK URGP=0 1 Time(s): <7= MAC=00:30:48:2e:69:48:00:01:97:00:20:20:08:00 SRC=165.91.140.32 DST=216.104.158.222 LEN=1500 TOS=0x00 PREC=0x00 TTL=49 ID=27320 DF PROTO=TCP SPT=388 DPT=53395 WINDOW=1448 RES=0x00 ACK URGP=0 1 Time(s): <7=0 1 Time(s): <7=388 WINDOW=2534 RES=0x00 ACK URGP=0 1 Time(s): <7C=165.91.140.32 DST=216.104.158.222 LEN=1500 TOS=0x00 PREC=0x00 TTL=49 ID=26652 DF PROTO=TCP SPT=388 DPT=53395 WINDOW=1448 RES=0x00 ACK URGP=0 1 Time(s): <7CK URGP=0 1 Time(s): <7F PROTO=TCP SPT=388 DPT=53395 WINDOW=1448 RES=0x00 ACK URGP=0 1 Time(s): <7PROTO=TCP SPT=388 DPT=53395 WINDOW=1448 RES=0x00 ACK URGP=0 1 Time(s): <7PROTO=TCP SPT=53395 DPT=388 WINDOW=2534 RES=0x00 ACK URGP=0 1 Time(s): <7ST=216.104.158.222 LEN=1500 TOS=0x00 PREC=0x00 TTL=49 ID=25830 DF PROTO=TCP SPT=388 DPT=53395 WINDOW=1448 RES=0x00 ACK URGP=0 1 Time(s): <8 RES=0x00 ACK URGP=0 1 Time(s): <C=165.91.140.32 DST=216.104.158.222 LEN=1500 TOS=0x00 PREC=0x00 TTL=49 ID=3154 DF PROTO=TCP SPT=388 DPT=53395 WINDOW=1448 RES=0x00 ACK URGP=0 1 Time(s): <EC=0x00 TTL=49 ID=60608 DF PROTO=TCP SPT=388 DPT=53395 WINDOW=1448 RES=0x00 ACK PSH URGP=0 1 Time(s): <EN=1500 TOS=0x00 PREC=0x00 TTL=49 ID=162 DF PROTO=TCP SPT=388 DPT=53395 WINDOW=1448 RES=0x00 ACK URGP=0 1 Time(s): <T=eth0 SRC=216.104.158.222 DST=165.91.140.32 LEN=80 TOS=0x00 PREC=0x00 TTL=64 ID=36235 DF PROTO=TCP SPT=53395 DPT=388 WINDOW=2263 RES=0x00 ACK PSH URGP=0 1 Time(s): =10498 DF PROTO=TCP SPT=53395 DPT=388 WINDOW=2263 RES=0x00 ACK URGP=0 1 Time(s): =80 TOS=0x00 PREC=0x00 TTL=64 ID=27364 DF PROTO=TCP SPT=53395 DPT=388 WINDOW=2534 RES=0x00 ACK PSH URGP=0 1 Time(s): ACK URGP=0 1 Time(s): C=140.90.192.168 DST=216.104.158.222 LEN=1500 TOS=0x00 PREC=0x00 TTL=55 ID=54847 DF PROTO=TCP SPT=35318 DPT=50884 WINDOW=1448 RES=0x00 ACK URGP=0 1 Time(s): C=216.104.158.222 DST=165.91.140.32 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=54472 DF PROTO=TCP SPT=53395 DPT=388 WINDOW=2263 RES=0x00 ACK URGP=0 1 Time(s): CK URGP=0 1 Time(s): CP SPT=388 DPT=53395 WINDOW=1448 RES=0x00 ACK URGP=0 1 Time(s): CP SPT=53395 DPT=388 WINDOW=2263 RES=0x00 ACK URGP=0 1 Time(s): EN=1500 TOS=0x00 PREC=0x00 TTL=49 ID=64772 DF PROTO=TCP SPT=388 DPT=53395 WINDOW=1448 RES=0x00 ACK URGP=0 1 Time(s): OS=0x00 PREC=0x00 TTL=64 ID=22791 DF PROTO=TCP SPT=53395 DPT=388 WINDOW=2534 RES=0x00 ACK URGP=0 1 Time(s): ROTO=TCP SPT=35318 DPT=50884 WINDOW=1448 RES=0x00 ACK URGP=0 1 Time(s): T=39772 DPT=35379 WINDOW=1448 RES=0x00 ACK URGP=0
---------------------- Kernel End -------------------------
sam wrote:
I've been getting LOTS of messages like the below in the daily log, and from all indications, it appears to all be related to the cpu; the machine is just over a year old, and was the old vortex.wa4phy;net server from the downtown co-lo site. Aside from huge log files, and lots of other fluff, numerous problems of other nature have started cropping up. Anyone have any suggestions as to what to do besides make a boat anchor out of it? It's all greek to me, so I'm totally at the mercy of those folks who understand the bits and bites (lit) that have run amok in things. Any suggestions as a possible fault other than the cpu just becoming more toasty brown? :)
Those messages don't look particularly bad, is the system crashing or something?
It seems like some sort of packet logging, perhaps iptables log rules or something?
I get messages like this in my logs all the time, no harm done: IN=eth0 OUT= MAC=00:0c:29:68:2f:4a:00:0b:bf:73:84:1b:08:00 SRC=125.89.73.173 DST=209.90.228.140 LEN=40 TOS=0x00 PREC=0x00 TTL=104 ID=256 PROTO=TCP SPT=6000 DPT=3306 WINDOW=16384 RES=0x00 SYN URGP=0 IN=eth0 OUT= MAC=00:0c:29:68:2f:4a:00:0b:bf:73:84:1b:08:00 SRC=74.160.133.83 DST=209.90.228.140 LEN=48 TOS=0x00 PREC=0x00 TTL=112 ID=14491 DF PROTO=TCP SPT=4664 DPT=139 WINDOW=64512 RES=0x00 SYN URGP=0 IN=eth0 OUT= MAC=00:0c:29:68:2f:4a:00:0b:bf:73:84:1b:08:00 SRC=74.160.133.83 DST=209.90.228.140 LEN=48 TOS=0x00 PREC=0x00 TTL=112 ID=15210 DF PROTO=TCP SPT=4664 DPT=139 WINDOW=64512 RES=0x00 SYN URGP=0 IN=eth0 OUT= MAC=00:0c:29:68:2f:4a:00:0b:bf:73:84:1b:08:00 SRC=95.67.143.212 DST=209.90.228.140 LEN=48 TOS=0x00 PREC=0x00 TTL=109 ID=6209 DF PROTO=TCP SPT=4604 DPT=139 WINDOW=65535 RES=0x00 SYN URGP=0 IN=eth0 OUT= MAC=00:0c:29:68:2f:4a:00:0b:bf:73:84:1b:08:00 SRC=95.67.143.212 DST=209.90.228.140 LEN=48 TOS=0x00 PREC=0x00 TTL=109 ID=6884 DF PROTO=TCP SPT=4604 DPT=139 WINDOW=65535 RES=0x00 SYN URGP=0 IN=eth0 OUT= MAC=00:0c:29:68:2f:4a:00:0b:bf:73:84:1b:08:00 SRC=207.183.171.254 DST=209.90.228.140 LEN=52 TOS=0x00 PREC=0x00 TTL=51 ID=1415 DF PROTO=TCP SPT=14669 DPT=139 WINDOW=60352 RES=0x00 SYN URGP=0 IN=eth0 OUT= MAC=00:0c:29:68:2f:4a:00:0b:bf:73:84:1b:08:00 SRC=207.183.171.254 DST=209.90.228.140 LEN=52 TOS=0x00 PREC=0x00 TTL=51 ID=2157 DF PROTO=TCP SPT=14669 DPT=139 WINDOW=60352 RES=0x00 SYN URGP=0
nate
I dunno Nate,
It started all this extraenous logging stuff after kicking memory from a paltry 256m up to 2 Gigs. The system has on occasion crashed with little else in the log files that would indicate any kind of other hardware problem, so with all the rejected packets or partial entries (none which showed up on this particular snippett leads me to believe it's dropping sync somehow. I'll watch the further logs and stuff to see if I can find something more definitive.
Thanks..
Sam
sam wrote:
I dunno Nate,
It started all this extraenous logging stuff after kicking memoryfrom a paltry 256m up to 2 Gigs. The system has on occasion crashed with little else in the log files that would indicate any kind of other hardware problem, so with all the rejected packets or partial entries (none which showed up on this particular snippett leads me to believe it's dropping sync somehow. I'll watch the further logs and stuff to see if I can find something more definitive.
System crashes often do not log anything.. if possible connect a serial console to the system and configure it to put the console on the serial port, then connect that to another system or terminal server..
Or you can run something like memtest86, though it can take a while sometimes to trace memory errors(days). There is a burn-in test suite that a lot of vendors use created by VA Linux a long time ago called Cerberus (ctcs), you can find it on sourceforge, in my experience if the hardware is bad ctcs will crash the system in a matter of hours every time(assuming motherboard/cpu/ram), and it helps catch bad disk controllers as well as disks too, putting the system under incredible strain.
nate
On Fri, 2009-05-22 at 20:18 -0400, sam wrote:
I dunno Nate,
It started all this extraenous logging stuff after kicking memoryfrom a paltry 256m up to 2 Gigs. The system has on occasion crashed with little else in the log files that would indicate any kind of other hardware problem, so with all the rejected packets or partial entries (none which showed up on this particular snippett leads me to believe it's dropping sync somehow. I'll watch the further logs and stuff to see if I can find something more definitive.
Along with the memtest86 mentioned in another reply, consider this.
When I upgraded the memory on my CentOS 4.x box, an Acer Ak77-400 (Max/N) unit with the via kt-400 chipset, last year to 2GB, it wouldn't work. To make a long story short, the DIMMS had no specs. The "auto" was set in BIOS for voltage. My fix was to change that to manual and bump the voltage a couple of tenths. More memory draws more amperage, may need more voltage to drive sufficient amperage.
*MAKE SURE YOU DON'T VOID YOUR WARRANTY*
*If* this tweak fries your memory (one or two tenths should not), you might be SOL. You might want to check the specs on the memory, check with your vendor about the issue (including warranty), etc. before proceding.
Thanks..
Sam
<snip sig stuff>
HTH
on 5-22-2009 5:18 PM sam spake the following:
I dunno Nate,
It started all this extraenous logging stuff after kicking memoryfrom a paltry 256m up to 2 Gigs. The system has on occasion crashed with little else in the log files that would indicate any kind of other hardware problem, so with all the rejected packets or partial entries (none which showed up on this particular snippett leads me to believe it's dropping sync somehow. I'll watch the further logs and stuff to see if I can find something more definitive.
Thanks..
Sam
Is the memory compatible? Quality memory or swapmeet crap? Adequate power supply or also swapmeet junk.
Scott,
All the memory is first rate stuff, DDR2 and new. Not sure if it *could* be a memory problem, as I've not dropped back to the 256 config to see if they go away but the whole machine crashed last night, and am now just going to do a memtest86, then reinstall. I never did quite like the way I partitioned things in the get-go, plus I'll add a raid device to boot.
Thanks...
Sam
on 5-28-2009 4:49 AM sam spake the following:
Scott,
All the memory is first rate stuff, DDR2 and new. Not sure if it *could* be a memory problem, as I've not dropped back to the 256 config to see if they go away but the whole machine crashed last night, and am now just going to do a memtest86, then reinstall. I never did quite like the way I partitioned things in the get-go, plus I'll add a raid device to boot.
Thanks...
Sam
Is the memory on the list for the motherboard? Some newer cutting edge boards kind of "pushed" their specs a bit to tweak some extra speed and they have a list of "known good" ram parts that they recommend. This is usually most common with desktop boards that have been pressed into service as a server, which is less than ideal, since they really weren't designed for constant use.
Also, hopefully the system is balanced with the ram in proper slots, since most ddr2 boards like matching pairs.
I'd let the memtest run as long as you can, and maybe cover a few slots on the system if you can to get the heat up a bit, or at least leave all the covers on if it is on a bench.
On Fri, May 22, 2009 at 4:35 PM, sam sam@wa4phy.net wrote:
I've been getting LOTS of messages like the below in the daily log, and from all indications, it appears to all be related to the cpu; the machine is just over a year old, and was the old vortex.wa4phy;net server from the downtown co-lo site. Aside from huge log files, and lots of other fluff, numerous problems of other nature have started cropping up. Anyone have any suggestions as to what to do besides make a boat anchor out of it? It's all greek to me, so I'm totally at the mercy of those folks who understand the bits and bites (lit) that have run amok in things. Any suggestions as a possible fault other than the cpu just becoming more toasty brown? :)
To me, it looks like the messages all have to do with communications. What are the other problems that you say "have started cropping up"? Are those things in the logs or the box rebooting by itself or some other issue?
On Fri, 22 May 2009, Lanny Marcus wrote:
On Fri, May 22, 2009 at 4:35 PM, sam sam@wa4phy.net wrote:
I've been getting LOTS of messages like the below in the daily log, and from all indications, it appears to all be related to the cpu; the machine is just over a year old, and was the old vortex.wa4phy;net server from the downtown co-lo site. Aside from huge log files, and lots of other fluff, numerous problems of other nature have started cropping up. Anyone have any suggestions as to what to do besides make a boat anchor out of it? It's all greek to me, so I'm totally at the mercy of those folks who understand the bits and bites (lit) that have run amok in things. Any suggestions as a possible fault other than the cpu just becoming more toasty brown? :)
To me, it looks like the messages all have to do with communications. What are the other problems that you say "have started cropping up"? Are those things in the logs or the box rebooting by itself or some other issue?
Setting up diskdump (or preferably) netconsole from netdump may help you find the root-cause. We retrofitted all 400 systems with netconsole at one customer after we were stuck finding why one server died on us. If it is not reproducable you lost your (maybe only) opportunity to have it fixed.
So in the end the effort of setting up netconsole for a company's datacenter is something you better do from the start, because the moment you really need it, it's already too late :)
-
Dag Wieers -
Lanny Marcus -
nate -
sam -
Scott Silva -
William L. Maltby