Benjamin Smith wrote, On 11/17/2009 01:46 PM:
See comments below...
On Tuesday 17 November 2009 07:52:01 Todd Denniston wrote:
Benjamin Smith wrote, On 11/16/2009 10:56 PM:
I have a 1TB USB drive plugged into a USB2 port that I use to back up the production drives (which are SCSI). It's working fine, but while doing backups (hourly) the load average on the server shoots up from the normal 0.5 - 1.5 or so up to a high between 10 and 30. Strangely, even though the "load is high" the server is completely responsive, even the USB drives being accessed are!
Using top to diagnose, nothing seems to be particularly high! IoWait seems reasonable (10-30%) and CPUs are 0.5%, Idle is 70-90%. Even accessing the USB partition while the load is "high" is responsive!
you might add another field to top while you are watching, Last used cpu (SMP), i.e., start top press f press j press enter
this should let you see if your process is bouncing between processors.
As workarounds perhaps asking the kernel to schedule in a specific way might help, i.e.: #1 set the backup on a particular set of processors, # replace the pg_dump line above with taskset -c 3-4 pg_dump <options> mydatabase > \ /media/backups/mydatabase.$hour.pgsql;
There are 8 cores on the machine, none of which are reporting more than 5% load. That's what has me perplexed. When I run top, I see a max of about 30% user. Everything else is zero. When I run the backup script to a non-USB drive, the load average is completely normal (below 0.50, often below 0.10)
USB chewing up more CPU than normal disks has been my experience all along, this just seems a little extreme.
#2 set the usb-storage on a particular set of processors, # Note USBSTORPID= line prototyped on CentOS 5 machine not 4. USBSTORPID=`ps aux |grep usb-storage|head -1 |awk '{print $2}'` taskset -p -c 3-4 $USBSTORPID #you might even go back and reduce the processor list #to just 3 or 4 instead of both.
Could you explain to me what this should accomplish? I'm curious as to why you went this route...
Even though the process is not using much processor time, having it bounce around between processors can: * thrash the cache of each processor as it goes there * waste time context switching in the next processor * bounce other processes around and cascade the same effects as they go along
I know that there has been some scheduler work over time to have these switches be less likely, but I have also seen some good effects by locking certain processes into a processor instead of letting it float. Usually the best processes to do to are ones that use large amounts of memory, like X or Firefox which are large enough that they thoroughly toss anything else out of a processor's cache.