[CentOS] Server locking up everyday around 3:30 AM

Fri Mar 11 19:05:08 UTC 2011
Steve Thompson <smt at vgersoft.com>

> PJ wrote:
>> Mar  8 03:33:18 web1 kernel: INFO: task wget:13608 blocked for more than
> 120 seconds.

Check the number of dirty pages:

 	grep Dirty /proc/meminfo

relative to the dirty_ratio setting:

 	cat /proc/sys/vm/dirty_ratio

to see if the system is going into synhronous flush mode around that time 
(especially if dirty_ratio is large and you have a lot of physical 
memory). This is what I usually see as the cause of the "blocked for more 
than" message. I've also found that it can be several minutes, and up to 
20 minutes, before the system recovers (but recover it always does).

-Steve