[CentOS] GFS performance issue

[Pasi Kärkkäinen]

On Mon, Jul 19, 2010 at 11:41:07AM -0400, Fred Wittekind wrote:
>  Two web servers, both virtualized with CentOS Xen servers as host
> (residing on two different physical servers). 
> GFS used to store home directories containing web document roots.
> 
> Shared block device used by GFS is an ISCSI target with the ISCSI
> initiator residing on the Dom-0, and presented to Dom-U webservers as
> drives.
> Also, providing a second shared block device for quorum disk.
> 
> If I hit the web site on just one of the nodes, it behaves as expected. 
> If I try to load the websites from both nodes at the same time (two web
> browser instances), the load average on both nodes spikes, and page
> loads very slowly.  The site I am trying to host is very high traffic,
> and if the servers can be nearly brought to there knees with two web
> browser instances running on a single workstation, that's not going to work.
> 
> I am not seeing any error messages in the logs regarding the cluster.
> 
> Any help or suggestions on how to troubleshoot this issue would be
> greatly appreciated.
> 

You might want to read linux-cluster mailinglist archives for 
GFS performance tuning tips.

Make sure you modify (increase) at least the max lock rate.

-- Pasi

> 
> [root at www3 www]# find ./ | wc -l
> 64815
> 
> [root at www3 www]# gfs2_tool df /home/www
> /home/www:
>   SB lock proto = "lock_dlm"
>   SB lock table = "Web:homewww"
>   SB ondisk format = 1801
>   SB multihost format = 1900
>   Block size = 4096
>   Journals = 2
>   Resource Groups = 316
>   Mounted lock proto = "lock_dlm"
>   Mounted lock table = "Web:homewww"
>   Mounted host data = "jid=1:id=393217:first=0"
>   Journal number = 1
>   Lock module flags = 0
>   Local flocks = FALSE
>   Local caching = FALSE
> 
>   Type           Total Blocks   Used Blocks    Free Blocks    use%
>   ------------------------------------------------------------------------
>   data           20707148       4347237        16359911       21%
>   inodes         16426386       66475          16359911       0%
> 
> [root at www3 www]# gfs2_tool gettune /home/www
> new_files_directio = 0
> new_files_jdata = 0
> quota_scale = 1.0000   (1, 1)
> logd_secs = 1
> recoverd_secs = 60
> statfs_quantum = 30
> stall_secs = 600
> quota_cache_secs = 300
> quota_simul_sync = 64
> statfs_slow = 0
> complain_secs = 10
> max_readahead = 262144
> quota_quantum = 60
> quota_warn_period = 10
> jindex_refresh_secs = 60
> log_flush_secs = 60
> incore_log_blocks = 1024
> 
> [root at www3 www]# cat /etc/cluster/cluster.conf |egrep '(dlm)|(gfs)'
>         <dlm plock_ownership="1" plock_rate_limit="0"/>
>         <gfs_controld plock_rate_limit="0"/>
> 
> 
> Fred Wittekind
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos