I've been having a similar problem and wanted to provide what hopefully is additional useful information. Like the original poster, things seem OK when booting with the network active. However, network access is extremely slow and timeouts are likely. If I do a service network stop, everything slows down, and keyboard and mouse activity is likely to be dropped. I also get error output which is included below. Booting with irqpoll doesn't seem to have any effect.
Of note is that my ethernet driver (r1000) is source built, not a driver included with the build. However, this driver was fine under CentOS 4.4.
Let me know what other information may be useful. Thanks
Mark
----- irq 177: nobody cared (try booting with the "irqpoll" option) [<c044aacb>] __report_bad_irq+0x2b/0x69 [<c044acb8>] note_interrupt+0x1af/0x1e7 [<c0570092>] usb_hcd_irq+0x23/0x50 [<c044a2ff>] handle_IRQ_event+0x23/0x49 [<c044a3d8>] __do_IRQ+0xb3/0xe8 [<c04063f4>] do_IRQ+0x93/0xae [<c040492e>] common_interrupt+0x1a/0x20 [<c045a83d>] __handle_mm_fault+0x7d/0x87b [<c04b8ea8>] avc_has_perm+0x3a/0x44 [<c05fd4ef>] do_page_fault+0x20a/0x4b8 [<c05fd2e5>] do_page_fault+0x0/0x4b8 [<c0404a71>] error_code+0x39/0x40 [<c04df804>] __copy_to_user_ll+0xd3/0xda [<c0429957>] do_proc_dointvec+0x1ee/0x291 [<c0429a94>] proc_dointvec+0x15/0x19 [<c0428c9c>] do_proc_dointvec_conv+0x0/0x36 [<c0429134>] do_rw_proc+0xaa/0xee [<c042918e>] proc_readsys+0x0/0x13 [<c042919e>] proc_readsys+0x10/0x13 [<c046af3c>] vfs_read+0x9f/0x141 [<c046b38a>] sys_read+0x3c/0x63 [<c0403eff>] syscall_call+0x7/0xb ======================= handlers: [<c057006f>] (usb_hcd_irq+0x0/0x50) Disabling IRQ #177