Trying to debug a database server getting cpu softlocks causing SSHD to hang and not let anybody log in. Figured that a good first step would be to get some kernel dumps when the problem occurs. According to what I read at the following web site, I can get dumps for softlock problems too. Here's the site: http://blog.kreyolys.com/2011/03/17/no-panic-its-just-a-kernel-panic/
Using a throwaway 32 bit dev server, I followed the directions exactly with the below modifications.
A) I got the kernel-debuginfo and kernel-debuginfo-common by enabling /etc/yum.repos.d/CentOS-Debuginfo.repo and then installing with yum.
B) One other change is that I had to use an alternate partition instead of the one resolving "/var" because the server was set up with LVM. The partition that I used ("/dev/sdb3") is actually mounted at /alt/home.
So when I simulate a crash by piping a "c" to /proc/sysrq-trigger, I see the following in /var/log/messages: May 30 13:37:24 norman kernel: SysRq : HELP : loglevel(0-9) reBoot Crash terminate-all-tasks(E) memory-full-oom-kill(F) kill-all-tasks(I) thaw-filesystems(J) saK show-backtrace-all-active-cpus(L) show-memory-usage(M) nice-all-RT-tasks(N) powerOff show-registers(P) show-all-timers(Q) unRaw Sync show-task-states(T) Unmount show-blocked-tasks(W) dump-ftrace-buffer(Z)
But nothing else. NO files in /var/crash, or /alt/home/var/crash (directory exists) Nothing happens at all when I run # echo 1 > /proc/sys/kernel/softlockup_panic
I've tried tweaking a few settings in /etc/kdump.conf, restarting after each time (since it creates a new boot img) to no avail. What am I messing?
Here's a bunch of hopefully relevant info:
kernel line in grub.conf: kernel /vmlinuz-2.6.32-358.6.2.el6.i686 ro root=/dev/mapper/VolGroup-lv_root nomodeset rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD rd_LVM_LV=VolGroup/lv_swap SYSFONT=latarcyrheb-sun16 rhgb crashkernel=auto quiet rd_LVM_LV=VolGroup/lv_root KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM crashkernel=128M nmi_watchdog=1
# grep -v "#" /etc/kdump.conf ext4 /dev/sdb3 path /var/crash core_collector makedumpfile -c -d 31
# cat /proc/meminfo MemTotal: 379352 kB MemFree: 232748 kB Buffers: 22496 kB Cached: 79040 kB SwapCached: 0 kB Active: 60512 kB Inactive: 60152 kB Active(anon): 20028 kB Inactive(anon): 4640 kB Active(file): 40484 kB Inactive(file): 55512 kB Unevictable: 0 kB Mlocked: 0 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 379352 kB LowFree: 232748 kB SwapTotal: 1048568 kB SwapFree: 1048568 kB Dirty: 48 kB Writeback: 0 kB AnonPages: 19124 kB Mapped: 14320 kB Shmem: 5556 kB Slab: 16216 kB SReclaimable: 8444 kB SUnreclaim: 7772 kB KernelStack: 816 kB PageTables: 2188 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 1238244 kB Committed_AS: 164964 kB VmallocTotal: 506396 kB VmallocUsed: 5344 kB VmallocChunk: 491856 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 9692 kB DirectMap2M: 514048 kB
// IN sysctl.conf kernel.unknown_nmi_panic=1 kernel.panic_on_unrecovered_nmi=1 vm.panic_on_oom=1 kernel.sysrq = 1
# /cat/proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 15 model : 1 model name : Intel(R) Pentium(R) 4 CPU 1.50GHz stepping : 2 cpu MHz : 1495.192 cache size : 256 KB fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 2 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm up pebs bts bogomips : 2990.38 clflush size : 64 cache_alignment : 128 address sizes : 36 bits physical, 32 bits virtual power management:
# dmesg: Initializing cgroup subsys cpuset Initializing cgroup subsys cpu Linux version 2.6.32-358.6.2.el6.i686 (mockbuild@c6b8.bsys.dev.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-3) (GCC) ) #1 SMP Thu May 16 18:12:13 UTC 2013 KERNEL supported cpus: Intel GenuineIntel AMD AuthenticAMD NSC Geode by NSC Cyrix CyrixInstead Centaur CentaurHauls Transmeta GenuineTMx86 Transmeta TransmetaCPU UMC UMC UMC UMC BIOS-provided physical RAM map: BIOS-e820: 0000000000000000 - 00000000000a0000 (usable) BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved) BIOS-e820: 0000000000100000 - 000000001ff77000 (usable) BIOS-e820: 000000001ff77000 - 000000001ff79000 (ACPI NVS) BIOS-e820: 000000001ff79000 - 0000000020000000 (reserved) BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved) BIOS-e820: 00000000fee00000 - 00000000fee10000 (reserved) BIOS-e820: 00000000ffb00000 - 0000000100000000 (reserved) DMI 2.3 present. SMBIOS version 2.3 @ 0xF0430 DMI: Dell Computer Corporation OptiPlex GX240 /OptiPlex GX240 , BIOS A05 07/19/2002 e820 update range: 0000000000000000 - 0000000000001000 (usable) ==> (reserved) e820 remove range: 00000000000a0000 - 0000000000100000 (usable) last_pfn = 0x1ff77 max_arch_pfn = 0x400000 MTRR default type: uncachable MTRR fixed ranges enabled: 00000-9FFFF write-back A0000-BFFFF uncachable C0000-DFFFF write-protect E0000-EFFFF uncachable F0000-FFFFF write-protect MTRR variable ranges enabled: 0 base 000000000 mask FE0000000 write-back 1 disabled 2 disabled 3 disabled 4 disabled 5 disabled 6 disabled 7 disabled x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106 initial memory mapped : 0 - 01000000 init_memory_mapping: 0000000000000000-000000001ff77000 Using x86 segment limits to approximate NX protection 0000000000 - 0000200000 page 4k 0000200000 - 001fe00000 page 2M 001fe00000 - 001ff77000 page 4k kernel direct mapping tables up to 1ff77000 @ 7000-f000 RAMDISK: 1f211000 - 1ff66d6a ACPI: RSDP 000fd560 00014 (v00 DELL ) ACPI: RSDT 000fd574 00034 (v01 DELL GX240 00000008 ASL 00000061) ACPI: FACP 000fd5a8 00074 (v01 DELL GX240 00000008 ASL 00000061) ACPI: DSDT fffe3c22 02393 (v01 DELL dt_ex 00001000 MSFT 0100000D) ACPI: FACS 1ff77000 00040 ACPI: SSDT fffe60f2 000BA (v01 DELL st_ex 00001000 MSFT 0100000D) ACPI: APIC 000fd61c 0005C (v01 DELL GX240 00000008 ASL 00000061) ACPI: BOOT 000fd678 00028 (v01 DELL GX240 00000008 ASL 00000061) ACPI: Local APIC address 0xfee00000 0MB HIGHMEM available. 511MB LOWMEM available. mapped low ram: 0 - 1ff77000 low ram: 0 - 1ff77000 node 0 low ram: 00000000 - 1ff77000 node 0 bootmap 00002000 - 00005ff0 (9 early reservations) ==> bootmem [0000000000 - 001ff77000] #0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000] #1 [0000001000 - 0000002000] EX TRAMPOLINE ==> [0000001000 - 0000002000] #2 [0000006000 - 0000007000] TRAMPOLINE ==> [0000006000 - 0000007000] #3 [0000400000 - 0000c6f610] TEXT DATA BSS ==> [0000400000 - 0000c6f610] #4 [001f211000 - 001ff66d6a] RAMDISK ==> [001f211000 - 001ff66d6a] #5 [000009fc00 - 0000100000] BIOS reserved ==> [000009fc00 - 0000100000] #6 [0000c70000 - 0000c78174] BRK ==> [0000c70000 - 0000c78174] #7 [0000007000 - 000000b000] PGTABLE ==> [0000007000 - 000000b000] #8 [0000002000 - 0000006000] BOOTMAP ==> [0000002000 - 0000006000] found SMP MP-table at [c00fe710] fe710 Reserving 128MB of memory at 16MB for crashkernel (System RAM: 511MB) Zone PFN ranges: DMA 0x00000001 -> 0x00001000 Normal 0x00001000 -> 0x0001ff77 HighMem 0x0001ff77 -> 0x0001ff77 Movable zone start PFN for each node early_node_map[2] active PFN ranges 0: 0x00000001 -> 0x000000a0 0: 0x00000100 -> 0x0001ff77 On node 0 totalpages: 130838 DMA zone: 32 pages used for memmap DMA zone: 0 pages reserved DMA zone: 3967 pages, LIFO batch:0 Normal zone: 991 pages used for memmap Normal zone: 125848 pages, LIFO batch:31 Using APIC driver default ACPI: PM-Timer IO Port: 0x808 ACPI: Local APIC address 0xfee00000 ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] disabled) ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0]) IOAPIC[0]: apic_id 1, version 32, address 0xfec00000, GSI 0-23 ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) ACPI: IRQ0 used by override. ACPI: IRQ2 used by override. ACPI: IRQ9 used by override. Using ACPI (MADT) for SMP configuration information SMP: Allowing 2 CPUs, 1 hotplug CPUs nr_irqs_gsi: 24 PM: Registered nosave memory: 00000000000a0000 - 00000000000f0000 PM: Registered nosave memory: 00000000000f0000 - 0000000000100000 Allocating PCI resources starting at 20000000 (gap: 20000000:dec00000) Booting paravirtualized kernel on bare hardware NR_CPUS:32 nr_cpumask_bits:32 nr_cpu_ids:2 nr_node_ids:1 PERCPU: Embedded 14 pages/cpu @c9600000 s36088 r0 d21256 u1048576 pcpu-alloc: s36088 r0 d21256 u1048576 alloc=1*2097152 pcpu-alloc: [0] 0 1 Built 1 zonelists in Zone order, mobility grouping on. Total pages: 129815 Kernel command line: ro root=/dev/mapper/VolGroup-lv_root nomodeset rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD rd_LVM_LV=VolGroup/lv_swap SYSFONT=latarcyrheb-sun16 rhgb crashkernel=auto quiet rd_LVM_LV=VolGroup/lv_root KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM crashkernel=128M nmi_watchdog=1 PID hash table entries: 2048 (order: 1, 8192 bytes) Dentry cache hash table entries: 65536 (order: 6, 262144 bytes) Inode-cache hash table entries: 32768 (order: 5, 131072 bytes) Enabling fast FPU save and restore... done. Enabling unmasked SIMD FPU exception support... done. Initializing CPU#0 Initializing HighMem for node 0 (00000000:00000000) Memory: 364908k/523740k available (4419k kernel code, 158168k reserved, 2500k data, 512k init, 0k highmem) virtual kernel memory layout: fixmap : 0xffad5000 - 0xfffff000 (5288 kB) pkmap : 0xff600000 - 0xff800000 (2048 kB) vmalloc : 0xe0777000 - 0xff5fe000 ( 494 MB) lowmem : 0xc0000000 - 0xdff77000 ( 511 MB) .init : 0xc0ac3000 - 0xc0b43000 ( 512 kB) .data : 0xc0850ea8 - 0xc0ac2008 (2500 kB) .text : 0xc0400000 - 0xc0850ea8 (4419 kB) Checking if this processor honours the WP bit even in supervisor mode...Ok. Hierarchical RCU implementation. NR_IRQS:2304 nr_irqs:424 Console: colour VGA+ 80x25 console [tty0] enabled allocated 1048576 bytes of page_cgroup please try 'cgroup_disable=memory' option if you don't want memory cgroups Fast TSC calibration using PIT Detected 1495.192 MHz processor. Calibrating delay loop (skipped), value calculated using timer frequency.. 2990.38 BogoMIPS (lpj=1495192) pid_max: default: 32768 minimum: 301 Security Framework initialized SELinux: Initializing. SELinux: Starting in permissive mode Mount-cache hash table entries: 512 Initializing cgroup subsys ns Initializing cgroup subsys cpuacct Initializing cgroup subsys memory Initializing cgroup subsys devices Initializing cgroup subsys freezer Initializing cgroup subsys net_cls Initializing cgroup subsys blkio Initializing cgroup subsys perf_event Initializing cgroup subsys net_prio CPU0: Hyper-Threading is disabled mce: CPU supports 4 MCE banks CPU0: Thermal monitoring enabled (TM1) SMP alternatives: switching to UP code ACPI: Core revision 20090903 Enabling APIC mode: Flat. Using 1 I/O APICs ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 CPU0: Intel(R) Pentium(R) 4 CPU 1.50GHz stepping 02 Performance Events: Netburst events, Netburst P4/Xeon PMU driver. ... version: 0 ... bit width: 40 ... generic registers: 18 ... value mask: 000000ffffffffff ... max period: 0000007fffffffff ... fixed-purpose events: 0 ... event mask: 000000000003ffff NMI watchdog enabled, takes one hw-pmu counter. Brought up 1 CPUs Total of 1 processors activated (2990.38 BogoMIPS). sizeof(vma)=100 bytes sizeof(page)=32 bytes sizeof(inode)=352 bytes sizeof(dentry)=132 bytes sizeof(ext3inode)=508 bytes sizeof(buffer_head)=56 bytes sizeof(skbuff)=184 bytes sizeof(task_struct)=1348 bytes devtmpfs: initialized PM: Registering ACPI NVS region at 1ff77000 (8192 bytes) regulator: core version 0.5 NET: Registered protocol family 16 ACPI: bus type pci registered PCI: PCI BIOS revision 2.10 entry at 0xfbe2e, last bus=2 PCI: Using configuration type 1 for base access bio: create slab <bio-0> at 0 ACPI: EC: Look up EC in DSDT ACPI: Interpreter enabled ACPI: (supports S0 S1 S3 S4 S5) ACPI: Using IOAPIC for interrupt routing ACPI: No dock devices found. HEST: Table not found. PCI: Ignoring host bridge windows from ACPI; if necessary, use "pci=use_crs" and report a bug ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) pci_root PNP0A03:00: host bridge window [io 0x0000-0x0cf7] (ignored) pci_root PNP0A03:00: host bridge window [io 0x0d00-0xffff] (ignored) pci_root PNP0A03:00: host bridge window [mem 0x000a0000-0x000bffff] (ignored) pci_root PNP0A03:00: host bridge window [mem 0x000c8000-0x000dffff] (ignored) pci_root PNP0A03:00: host bridge window [mem 0xc0000000-0xffdfffff] (ignored) pci 0000:00:00.0: reg 10: [mem 0xf0000000-0xf3ffffff pref] pci 0000:00:1f.0: quirk: [io 0x0800-0x087f] claimed by ICH4 ACPI/GPIO/TCO pci 0000:00:1f.0: quirk: [io 0x0880-0x08bf] claimed by ICH4 GPIO pci 0000:00:1f.1: reg 20: [io 0xffa0-0xffaf] pci 0000:00:1f.2: reg 20: [io 0xff80-0xff9f] pci 0000:00:1f.3: reg 20: [io 0xdcd0-0xdcdf] pci 0000:00:1f.4: reg 20: [io 0xff60-0xff7f] pci 0000:00:1f.5: reg 10: [io 0xd800-0xd8ff] pci 0000:00:1f.5: reg 14: [io 0xdc40-0xdc7f] pci 0000:01:00.0: reg 10: [mem 0xfc000000-0xfcffffff] pci 0000:01:00.0: reg 14: [mem 0xe0000000-0xefffffff pref] pci 0000:01:00.0: reg 18: [mem 0xfb000000-0xfbffffff] pci 0000:01:00.0: reg 30: [mem 0x80000000-0x8001ffff pref] pci 0000:00:01.0: PCI bridge to [bus 01-01] pci 0000:00:01.0: bridge window [io 0xf000-0x0000] (disabled) pci 0000:00:01.0: bridge window [mem 0xfb000000-0xfdffffff] pci 0000:00:01.0: bridge window [mem 0xe0000000-0xefffffff pref] pci 0000:02:0c.0: reg 10: [io 0xec80-0xecff] pci 0000:02:0c.0: reg 14: [mem 0xfe1ffc00-0xfe1ffc7f] pci 0000:02:0c.0: reg 30: [mem 0xfe200000-0xfe21ffff pref] pci 0000:02:0c.0: supports D1 D2 pci 0000:02:0c.0: PME# supported from D0 D1 D2 D3hot D3cold pci 0000:02:0c.0: PME# disabled pci 0000:00:1e.0: PCI bridge to [bus 02-02] (subtractive decode) pci 0000:00:1e.0: bridge window [io 0xe000-0xefff] pci 0000:00:1e.0: bridge window [mem 0xfe100000-0xfe2fffff] pci 0000:00:1e.0: bridge window [mem 0xfff00000-0x000fffff pref] (disabled) pci 0000:00:1e.0: bridge window [io 0x0000-0xffff] (subtractive decode) pci 0000:00:1e.0: bridge window [mem 0x00000000-0xfffffffff] (subtractive decode) pci_bus 0000:00: on NUMA node 0 ACPI: PCI Interrupt Routing Table [_SB_.PCI0._PRT] ACPI: PCI Interrupt Routing Table [_SB_.PCI0.PCI1._PRT] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 9 10 *11 12 15) ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 9 *10 11 12 15) ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 9 10 *11 12 15) ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 9 10 *11 12 15) ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 9 10 11 12 15) *0, disabled. ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 9 10 11 12 15) *0, disabled. ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 9 10 11 12 15) *0, disabled. ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 7 *9 10 11 12 15) vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none vgaarb: loaded vgaarb: bridge control possible 0000:01:00.0 SCSI subsystem initialized libata version 3.00 loaded. usbcore: registered new interface driver usbfs usbcore: registered new interface driver hub usbcore: registered new device driver usb PCI: Using ACPI for IRQ routing PCI: old code would have set cacheline size to 128 bytes, but clflush_size = 64 PCI: pci_cache_line_size set to 64 bytes NetLabel: Initializing NetLabel: domain hash size = 128 NetLabel: protocols = UNLABELED CIPSOv4 NetLabel: unlabeled traffic allowed by default Switching to clocksource jiffies pnp: PnP ACPI init ACPI: bus type pnp registered pnp 00:00: [mem 0x00000000-0x0009ffff] pnp 00:00: [mem 0x00100000-0x00ffffff] pnp 00:00: [mem 0x01000000-0x1ff76fff] pnp 00:00: [mem 0x000f0000-0x000fffff] pnp 00:00: [mem 0xfec00000-0xfec0ffff] pnp 00:00: [mem 0xfee00000-0xfee0ffff] pnp 00:00: [mem 0xffb00000-0xffbfffff] pnp 00:00: [mem 0xffc00000-0xffffffff] pnp 00:00: Plug and Play ACPI device, IDs PNP0c01 (active) pnp 00:01: [io 0x0cf8-0x0cff] pnp 00:01: Plug and Play ACPI device, IDs PNP0a03 (active) pnp 00:02: [io 0x0080-0x009f] pnp 00:02: [io 0x0000-0x001f] pnp 00:02: [io 0x00c0-0x00df] pnp 00:02: [dma 4] pnp 00:02: Plug and Play ACPI device, IDs PNP0200 (active) pnp 00:03: [io 0x00f0-0x00ff] pnp 00:03: [irq 13] pnp 00:03: Plug and Play ACPI device, IDs PNP0c04 (active) pnp 00:04: [io 0x0061] pnp 00:04: Plug and Play ACPI device, IDs PNP0800 (active) pnp 00:05: [io 0x0070-0x007f] pnp 00:05: [irq 8] pnp 00:05: Plug and Play ACPI device, IDs PNP0b00 (active) pnp 00:06: [io 0x03f0-0x03f5] pnp 00:06: [io 0x03f7] pnp 00:06: [irq 6] pnp 00:06: [dma 2] pnp 00:06: Plug and Play ACPI device, IDs PNP0700 (active) pnp 00:07: [io 0x03f8-0x03ff] pnp 00:07: [irq 4] pnp 00:07: Plug and Play ACPI device, IDs PNP0501 (active) pnp 00:08: [io 0x02f8-0x02ff] pnp 00:08: [irq 3] pnp 00:08: Plug and Play ACPI device, IDs PNP0501 (active) pnp 00:09: [io 0x0378-0x037f] pnp 00:09: [io 0x0778-0x077f] pnp 00:09: [irq 7] pnp 00:09: [dma 0 disabled] pnp 00:09: Plug and Play ACPI device, IDs PNP0401 (active) pnp 00:0a: [io 0x0060] pnp 00:0a: [io 0x0064] pnp 00:0a: [io 0x0062-0x0063] pnp 00:0a: [io 0x0065-0x006f] pnp 00:0a: [io 0x00e0-0x00ef] pnp 00:0a: [io 0x0800-0x085f] pnp 00:0a: [io 0x0c00-0x0c7f] pnp 00:0a: [io 0x0860-0x08ff] pnp 00:0a: disabling [io 0x0800-0x085f] because it overlaps 0000:00:1f.0 BAR 13 [io 0x0800-0x087f] pnp 00:0a: disabling [io 0x0860-0x08ff] because it overlaps 0000:00:1f.0 BAR 13 [io 0x0800-0x087f] pnp 00:0a: Plug and Play ACPI device, IDs PNP0c01 (active) pnp: PnP ACPI: found 11 devices ACPI: ACPI bus type pnp unregistered system 00:00: [mem 0x00000000-0x0009ffff] could not be reserved system 00:00: [mem 0x00100000-0x00ffffff] could not be reserved system 00:00: [mem 0x01000000-0x1ff76fff] could not be reserved system 00:00: [mem 0x000f0000-0x000fffff] could not be reserved system 00:00: [mem 0xfec00000-0xfec0ffff] could not be reserved system 00:00: [mem 0xfee00000-0xfee0ffff] has been reserved system 00:00: [mem 0xffb00000-0xffbfffff] has been reserved system 00:00: [mem 0xffc00000-0xffffffff] has been reserved system 00:0a: [io 0x0c00-0x0c7f] has been reserved Switching to clocksource acpi_pm pci 0000:01:00.0: no compatible bridge window for [mem 0x80000000-0x8001ffff pref] pci 0000:02:0c.0: address space collision: [mem 0xfe200000-0xfe21ffff pref] conflicts with PCI Bus 0000:02 [mem 0xfe100000-0xfe2fffff] PCI: max bus depth: 1 pci_try_num: 2 pci 0000:00:1e.0: BAR 15: assigned [mem 0x20000000-0x200fffff pref] pci 0000:01:00.0: BAR 6: assigned [mem 0xfd000000-0xfd01ffff pref] pci 0000:00:01.0: PCI bridge to [bus 01-01] pci 0000:00:01.0: PCI bridge to [bus 01-01] pci 0000:00:01.0: bridge window [io disabled] pci 0000:00:01.0: bridge window [mem 0xfb000000-0xfdffffff] pci 0000:00:01.0: bridge window [mem 0xe0000000-0xefffffff pref] pci 0000:02:0c.0: BAR 6: assigned [mem 0x20000000-0x2001ffff pref] pci 0000:00:1e.0: PCI bridge to [bus 02-02] pci 0000:00:1e.0: PCI bridge to [bus 02-02] pci 0000:00:1e.0: bridge window [io 0xe000-0xefff] pci 0000:00:1e.0: bridge window [mem 0xfe100000-0xfe2fffff] pci 0000:00:1e.0: bridge window [mem 0x20000000-0x200fffff pref] pci 0000:00:1e.0: setting latency timer to 64 pci_bus 0000:00: resource 0 [io 0x0000-0xffff] pci_bus 0000:00: resource 1 [mem 0x00000000-0xfffffffff] pci_bus 0000:01: resource 1 [mem 0xfb000000-0xfdffffff] pci_bus 0000:01: resource 2 [mem 0xe0000000-0xefffffff pref] pci_bus 0000:02: resource 0 [io 0xe000-0xefff] pci_bus 0000:02: resource 1 [mem 0xfe100000-0xfe2fffff] pci_bus 0000:02: resource 2 [mem 0x20000000-0x200fffff pref] pci_bus 0000:02: resource 4 [io 0x0000-0xffff] pci_bus 0000:02: resource 5 [mem 0x00000000-0xfffffffff] NET: Registered protocol family 2 IP route cache hash table entries: 4096 (order: 2, 16384 bytes) TCP established hash table entries: 16384 (order: 5, 131072 bytes) TCP bind hash table entries: 16384 (order: 5, 131072 bytes) TCP: Hash tables configured (established 16384 bind 16384) TCP reno registered NET: Registered protocol family 1 alloc irq_desc for 19 on node -1 alloc kstat_irqs on node -1 pci 0000:00:1f.2: PCI INT D -> GSI 19 (level, low) -> IRQ 19 pci 0000:00:1f.2: PCI INT D disabled alloc irq_desc for 23 on node -1 alloc kstat_irqs on node -1 pci 0000:00:1f.4: PCI INT C -> GSI 23 (level, low) -> IRQ 23 pci 0000:00:1f.4: PCI INT C disabled pci 0000:01:00.0: Boot video device Trying to unpack rootfs image as initramfs... Freeing initrd memory: 13655k freed Simple Boot Flag at 0x7a set to 0x1 apm: BIOS version 1.2 Flags 0x03 (Driver version 1.16ac) apm: overridden by ACPI. audit: initializing netlink socket (disabled) type=2000 audit(1369944362.104:1): initialized HugeTLB registered 2 MB page size, pre-allocated 0 pages VFS: Disk quotas dquot_6.5.2 Dquot-cache hash table entries: 1024 (order 0, 4096 bytes) msgmni has been set to 739 SELinux: Registering netfilter hooks alg: No test for stdrng (krng) ksign: Installing public key data Loading keyring - Added public key 834D293187AE1760 - User ID: CentOS (Kernel Module GPG key) Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251) io scheduler noop registered io scheduler anticipatory registered io scheduler deadline registered io scheduler cfq registered (default) pci_hotplug: PCI Hot Plug PCI Core version: 0.5 pciehp: PCI Express Hot Plug Controller Driver version: 0.4 acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 ipmi message handler version 39.2 IPMI System Interface driver. ipmi_si: Adding default-specified kcs state machine ipmi_si: Trying default-specified kcs state machine at i/o address 0xca2, slave address 0x0, irq 0 ipmi_si: Interface detection failed ipmi_si: Adding default-specified smic state machine ipmi_si: Trying default-specified smic state machine at i/o address 0xca9, slave address 0x0, irq 0 ipmi_si: Interface detection failed ipmi_si: Adding default-specified bt state machine ipmi_si: Trying default-specified bt state machine at i/o address 0xe4, slave address 0x0, irq 0 ipmi_si: Interface detection failed ipmi_si: Unable to find any System Interface(s) input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input0 ACPI: Power Button [VBTN] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input1 ACPI: Power Button [PWRF] ACPI: acpi_idle registered with cpuidle ERST: Table is not found! GHES: HEST is not enabled! isapnp: Scanning for PnP cards... isapnp: No Plug & Play device found Non-volatile memory driver v1.3 Linux agpgart interface v0.103 agpgart-intel 0000:00:00.0: Intel i845 Chipset agpgart-intel 0000:00:00.0: AGP aperture is 64M @ 0xf0000000 crash memory driver: version 1.1 Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A 00:07: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A 00:08: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A brd: module loaded loop: module loaded input: Macintosh mouse button emulation as /devices/virtual/input/input2 Fixed MDIO Bus: probed ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver uhci_hcd: USB Universal Host Controller Interface driver uhci_hcd 0000:00:1f.2: PCI INT D -> GSI 19 (level, low) -> IRQ 19 uhci_hcd 0000:00:1f.2: setting latency timer to 64 uhci_hcd 0000:00:1f.2: UHCI Host Controller uhci_hcd 0000:00:1f.2: new USB bus registered, assigned bus number 1 uhci_hcd 0000:00:1f.2: irq 19, io base 0x0000ff80 usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 usb usb1: Product: UHCI Host Controller usb usb1: Manufacturer: Linux 2.6.32-358.6.2.el6.i686 uhci_hcd usb usb1: SerialNumber: 0000:00:1f.2 usb usb1: configuration #1 chosen from 1 choice hub 1-0:1.0: USB hub found hub 1-0:1.0: 2 ports detected uhci_hcd 0000:00:1f.4: PCI INT C -> GSI 23 (level, low) -> IRQ 23 uhci_hcd 0000:00:1f.4: setting latency timer to 64 uhci_hcd 0000:00:1f.4: UHCI Host Controller uhci_hcd 0000:00:1f.4: new USB bus registered, assigned bus number 2 uhci_hcd 0000:00:1f.4: irq 23, io base 0x0000ff60 usb usb2: New USB device found, idVendor=1d6b, idProduct=0001 usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1 usb usb2: Product: UHCI Host Controller usb usb2: Manufacturer: Linux 2.6.32-358.6.2.el6.i686 uhci_hcd usb usb2: SerialNumber: 0000:00:1f.4 usb usb2: configuration #1 chosen from 1 choice hub 2-0:1.0: USB hub found hub 2-0:1.0: 2 ports detected PNP: No PS/2 controller found. Probing ports directly. serio: i8042 KBD port at 0x60,0x64 irq 1 serio: i8042 AUX port at 0x60,0x64 irq 12 mice: PS/2 mouse device common for all mice rtc_cmos 00:05: RTC can wake from S4 rtc_cmos 00:05: rtc core: registered rtc_cmos as rtc0 rtc0: alarms up to one day, 242 bytes nvram cpuidle: using governor ladder cpuidle: using governor menu EFI Variables Facility v0.08 2004-May-17 usbcore: registered new interface driver hiddev usbcore: registered new interface driver usbhid usbhid: v2.6:USB HID core driver TCP cubic registered Initializing XFRM netlink socket NET: Registered protocol family 17 Using IPI No-Shortcut mode registered taskstats version 1 rtc_cmos 00:05: setting system clock to 2013-05-30 20:06:03 UTC (1369944363) Initalizing network drop monitor service Freeing unused kernel memory: 512k freed Write protecting the kernel text: 4420k Write protecting the kernel read-only data: 1924k dracut: dracut-004-303.el6 Refined TSC clocksource calibration: 1495.148 MHz. Switching to clocksource tsc device-mapper: uevent: version 1.0.3 device-mapper: ioctl: 4.23.6-ioctl (2012-07-25) initialised: dm-devel@redhat.com udev: starting version 147 ACPI: WMI: Mapper loaded [drm] Initialized drm 1.1.0 20060810 dracut: Starting plymouth daemon ata_piix 0000:00:1f.1: version 2.13 ata_piix 0000:00:1f.1: setting latency timer to 64 scsi0 : ata_piix scsi1 : ata_piix ata1: PATA max UDMA/100 cmd 0x1f0 ctl 0x3f6 bmdma 0xffa0 irq 14 ata2: PATA max UDMA/100 cmd 0x170 ctl 0x376 bmdma 0xffa8 irq 15 ata1.00: ATA-7: ST3500630A, 3.AAE, max UDMA/100 ata1.00: 976773168 sectors, multi 8: LBA48 ata2.00: ATA-7: ST3500630A, 3.AAF, max UDMA/100 ata2.00: 976773168 sectors, multi 8: LBA48 ata2.00: limited to UDMA/33 due to 40-wire cable ata1.00: configured for UDMA/100 scsi 0:0:0:0: Direct-Access ATA ST3500630A 3.AA PQ: 0 ANSI: 5 ata2.00: configured for UDMA/33 scsi 1:0:0:0: Direct-Access ATA ST3500630A 3.AA PQ: 0 ANSI: 5 sd 0:0:0:0: [sda] 976773168 512-byte logical blocks: (500 GB/465 GiB) sd 0:0:0:0: [sda] Write Protect is off sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00 sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA sda: sd 1:0:0:0: [sdb] 976773168 512-byte logical blocks: (500 GB/465 GiB) sd 1:0:0:0: [sdb] Write Protect is off sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00 sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA sdb: sda1 sda2 sd 0:0:0:0: [sda] Attached SCSI disk sdb1 sdb2 sdb3 sd 1:0:0:0: [sdb] Attached SCSI disk dracut: Scanning devices sda2 for LVM logical volumes VolGroup/lv_swap VolGroup/lv_root dracut: inactive '/dev/VolGroup/lv_root' [50.00 GiB] inherit dracut: inactive '/dev/VolGroup/lv_home' [414.27 GiB] inherit dracut: inactive '/dev/VolGroup/lv_swap' [1.00 GiB] inherit EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: dracut: Mounted root filesystem /dev/mapper/VolGroup-lv_root SELinux: Disabled at runtime. SELinux: Unregistering netfilter hooks type=1404 audit(1369944367.063:2): selinux=0 auid=4294967295 ses=4294967295 dracut: dracut: Switching root udev: starting version 147 intel_rng: FWH not detected iTCO_vendor_support: vendor-support=0 iTCO_wdt: Intel TCO WatchDog Timer Driver v1.07rh iTCO_wdt: Found a ICH2 TCO device (Version=1, TCOBASE=0x0860) iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0) sd 0:0:0:0: Attached scsi generic sg0 type 0 sd 1:0:0:0: Attached scsi generic sg1 type 0 alloc irq_desc for 17 on node -1 alloc kstat_irqs on node -1 i801_smbus 0000:00:1f.3: PCI INT B -> GSI 17 (level, low) -> IRQ 17 Intel ICH 0000:00:1f.5: PCI INT B -> GSI 17 (level, low) -> IRQ 17 Intel ICH 0000:00:1f.5: setting latency timer to 64 intel8x0_measure_ac97_clock: measured 50922 usecs (4 samples) intel8x0: measured clock 78 rejected intel8x0_measure_ac97_clock: measured 50946 usecs (2852 samples) intel8x0: clocking to 41157 alloc irq_desc for 18 on node -1 alloc kstat_irqs on node -1 3c59x 0000:02:0c.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18 3c59x: Donald Becker and others. 0000:02:0c.0: 3Com PCI 3c905C Tornado at e0ec0c00. dcdbas dcdbas: Dell Systems Management Base Driver (version 5.6.0-3.2) parport_pc 00:09: reported by Plug and Play ACPI parport0: PC-style at 0x378 (0x778), irq 7 [PCSPP,TRISTATE] ppdev: user-space parallel port driver EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: EXT4-fs (dm-2): mounted filesystem with ordered data mode. Opts: EXT4-fs (sdb2): warning: checktime reached, running e2fsck is recommended EXT4-fs (sdb2): mounted filesystem with ordered data mode. Opts: EXT4-fs (sdb1): warning: checktime reached, running e2fsck is recommended EXT4-fs (sdb1): mounted filesystem with ordered data mode. Opts: EXT4-fs (sdb3): warning: checktime reached, running e2fsck is recommended EXT4-fs (sdb3): mounted filesystem with ordered data mode. Opts: Adding 1048568k swap on /dev/mapper/VolGroup-lv_swap. Priority:-1 extents:1 across:1048568k NET: Registered protocol family 10 lo: Disabled Privacy Extensions ip6_tables: (C) 2000-2006 Netfilter Core Team nf_conntrack version 0.5.0 (5927 buckets, 23708 max) ip_tables: (C) 2000-2006 Netfilter Core Team eth0: setting full-duplex. eth0: no IPv6 routers present SysRq : HELP : loglevel(0-9) reBoot Crash terminate-all-tasks(E) memory-full-oom-kill(F) kill-all-tasks(I) thaw-filesystems(J) saK show-backtrace-all-active-cpus(L) show-memory-usage(M) nice-all-RT-tasks(N) powerOff show-registers(P) show-all-timers(Q) unRaw Sync show-task-states(T) Unmount show-blocked-tasks(W) dump-ftrace-buffer(Z) SysRq : HELP : loglevel(0-9) reBoot Crash terminate-all-tasks(E) memory-full-oom-kill(F) kill-all-tasks(I) thaw-filesystems(J) saK show-backtrace-all-active-cpus(L) show-memory-usage(M) nice-all-RT-tasks(N) powerOff show-registers(P) show-all-timers(Q) unRaw Sync show-task-states(T) Unmount show-blocked-tasks(W) dump-ftrace-buffer(Z)