See comments below...
On Tuesday 17 November 2009 07:52:01 Todd Denniston wrote:
Benjamin Smith wrote, On 11/16/2009 10:56 PM:
I have a 1TB USB drive plugged into a USB2 port that I use to back up the production drives (which are SCSI). It's working fine, but while doing backups (hourly) the load average on the server shoots up from the normal 0.5 - 1.5 or so up to a high between 10 and 30. Strangely, even though the "load is high" the server is completely responsive, even the USB drives being accessed are!
Backup script is really simple, run via cron, pretty much just:
#! /bin/sh hour=`date +%k`; pg_dump <options> mydatabase > /media/backups/mydatabase.$hour.pgsql;
where /media/backups is the mount point for the USB drive.
Using top to diagnose, nothing seems to be particularly high! IoWait seems reasonable (10-30%) and CPUs are 0.5%, Idle is 70-90%. Even accessing the USB partition while the load is "high" is responsive!
I'm guessing that something changed in how load average is counted?
Server Stats: Late model 8-way Xeon, SuperMicro brand. CentOS 4.x / 64 (all updates applied, booted after last kernel update) Kernel 2.6.9-89.0.16.ELsmp 4 GB ECC RAM 300 GB SCSI HDD. Standard Apache/PHP, Postgres 8.4.
Any idea how to revert to the old load average tracking behavior short of using a stale and potentially insecure kernel?
Are you saying that when you were running a previous kernel the same operations with the same devices did not have the high load?
Correct!
Which specific kernels worked as desired (if someone is going to bisect the problem they need a start point)?
kernel-smp-devel-2.6.9-89.0.15.EL (I always keep my machines updated on at least a weekly scheduule)
Are there other processes on the machine that are waiting to use the db while the dump is occurring?
No. Database is actually on a different machine and backups are being done over the network.
How many postgres processes are waiting for the dump to finish (it has been a while since I ran postgres so I don't recall how it deals with query's during a dump)?
One - the one performing the backup. Postgres uses MVCC so pg_dump doesn't block any other connections from continuing/finishing.
As workarounds perhaps asking the kernel to schedule in a specific way might help, i.e.: #1 set the backup on a particular set of processors, # replace the pg_dump line above with taskset -c 3-4 pg_dump <options> mydatabase > \ /media/backups/mydatabase.$hour.pgsql;
There are 8 cores on the machine, none of which are reporting more than 5% load. That's what has me perplexed. When I run top, I see a max of about 30% user. Everything else is zero. When I run the backup script to a non-USB drive, the load average is completely normal (below 0.50, often below 0.10)
#2 set the usb-storage on a particular set of processors, # Note USBSTORPID= line prototyped on CentOS 5 machine not 4. USBSTORPID=`ps aux |grep usb-storage|head -1 |awk '{print $2}'` taskset -p -c 3-4 $USBSTORPID #you might even go back and reduce the processor list #to just 3 or 4 instead of both.
Could you explain to me what this should accomplish? I'm curious as to why you went this route...
#3 don't update atime # (should at worst be a minor thing, and you say that # the usb mounted file system is responsive, # but perhaps it would help some.) mount -oremount,noatime /media/backups/
Already mounted noatime... here's the mount line in the backup script: # mount -o rw,noatime -t ext3 /dev/sdc1 /home/backup/localdb/