System : CentOS 4.2 2.6.9-22.0.1.ELsmp System fully up-to-date. 3GB RAM 3ware 9000S card, with Raid5 array.
I think that's about all relevant info ...
Had file on disk (not array), attempted to mv file to array. Went fine till 2.4GB was copied, then it slowed down to a meg every few minutes. Free memory was ~50MB (Typically is 1.5-2GB), and cache was 2.5GB.
Stopped the move, however cache not being cleared out -- and this issue continues to happen - unsure if it was always a problem/known issue, it does however only occur when copying a single file > 2.4GB (I have copied 5+GB of 1+GB files in the past without issue).
Also, saw following in dmesg while this was going on: Out of Memory: Killed process 12467 (mysqld).
I'll be restarting server later on tonight, which will 'fix' immediate problem, however just curious on thoughts in regards to this.
Regards,
Sean O Sullivan
On Wed, 30 Nov 2005 at 9:00pm, Sean O Sullivan wrote
System : CentOS 4.2 2.6.9-22.0.1.ELsmp System fully up-to-date. 3GB RAM 3ware 9000S card, with Raid5 array.
I think that's about all relevant info ...
Had file on disk (not array), attempted to mv file to array. Went fine till 2.4GB was copied, then it slowed down to a meg every few minutes. Free memory was ~50MB (Typically is 1.5-2GB), and cache was 2.5GB.
I'm guessing this was your problem (workarounds suggested in the bug):
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=156437
'mount -o noreservation' the FS and try it again.
Joshua Baker-LePain wrote:
I'm guessing this was your problem (workarounds suggested in the bug):
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=156437
'mount -o noreservation' the FS and try it again.
Haven't tried this yet - still reading the bugzilla report - sounds fairly similar to what was happening though - thanks.
Regards,
Sean
} } I'm guessing this was your problem (workarounds suggested in the bug): } } https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=156437 } } 'mount -o noreservation' the FS and try it again. } } -- } Joshua Baker-LePain
greetings,
i take it that the suggested kernel bigfix(es) has not moved into the current production kernels or kernel fixes yet?
how would i find out... i did read the entire bugzilla mentioned above yet i am not an expert in knowing the implications etc
thanks
- rh
-- Robert Hanson - Abba Communications Computer & Internet Services (509) 624-7159 - www.abbacomm.net
Robert wrote:
} } I'm guessing this was your problem (workarounds suggested in the bug): } } https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=156437 } } 'mount -o noreservation' the FS and try it again. } } -- } Joshua Baker-LePain
greetings,
i take it that the suggested kernel bigfix(es) has not moved into the current production kernels or kernel fixes yet?
It's mensioned on Bugzilla thread that it will hopefully make it in U3, in other words Update 3, or in CentOS terms, 4.3
Regards,
Sean
Sean O Sullivan seanos@seanos.net wrote:
System : CentOS 4.2 2.6.9-22.0.1.ELsmp System fully up-to-date. 3GB RAM 3ware 9000S card, with Raid5 array. I think that's about all relevant info ...
You have the latest 9.2.1.1 3Ware firmware release, correct? And the latest driver for that release, correct?
Had file on disk (not array), attempted to mv file to array. Went fine till 2.4GB was copied, then it slowed down to a meg every few minutes. Free memory was ~50MB (Typically is 1.5-2GB), and cache was 2.5GB. Stopped the move, however cache not being cleared out -- and this issue continues to happen - unsure if it was always a problem/known issue, it does however only occur
when
copying a single file > 2.4GB (I have copied 5+GB of 1+GB files in the past without issue).
Hit the 3Ware's knowlegebse on tweaking the kernel buffer settings for your configuration. They are extremely helpful/useful in this regard. You might have to play with a few settings before it works the most ideal.
Also, saw following in dmesg while this was going on: Out of Memory: Killed process 12467 (mysqld). I'll be restarting server later on tonight, which will 'fix' immediate problem, however just curious on thoughts in regards to this.
Again, everytime I run into this, I hit 3Ware's knowlegebase and find some settings that I play with and they not only solve the problem, but increase performance. And if it persists, you'll be ready to talk to their tech support.
A good whitepaper to read on 3Ware 9000 series cards on kernel 2.6 (applicable to your CentOS 4.2 system) is here: http://www.3ware.com/kb/attachments/Linux2.6WP_0701.pdf
Bryan J. Smith wrote:
3ware 9000S card, with Raid5 array. I think that's about all relevant info ...
You have the latest 9.2.1.1 3Ware firmware release, correct? And the latest driver for that release, correct?
Using latest firmware, yes. I'm using the driver included in kernel, not 3ware's - I'm unsure if this is 'right' or not - be fairly curious on your experience in this matter (since you seem to have a fair bit of 3ware experience/knowledge).
Again, everytime I run into this, I hit 3Ware's knowlegebase and find some settings that I play with and they not only solve the problem, but increase performance. And if it persists, you'll be ready to talk to their tech support.
A good whitepaper to read on 3Ware 9000 series cards on kernel 2.6 (applicable to your CentOS 4.2 system) is here: http://www.3ware.com/kb/attachments/Linux2.6WP_0701.pdf
Thanks will look into that. Unsure if if related to this problem - bugzilla link given by Joshua looks like a fairly likely culprit.
Using latest firmware, yes. I'm using the driver included
in
kernel, not 3ware's - I'm unsure if this is 'right' or not
What does "modinfo 3w-9xxx" give you?
The driver for release 9.2.1.1 is 2.26.03.019[fw]. If you're not running that version, grab the driver source and rebuild, add the module to your tree and create a new initrd. If you need some help in doing that, just let me know.
- be fairly curious on your experience in this matter
(since
you seem to have a fair bit of 3ware experience/knowledge).
I haven't deployed but one (1) system with a 3Ware Escalade 9500S, so my exposure to the 9500S line has been limited. I have deployed countless Escalade 6000-8000 series products though.
Thanks will look into that. Unsure if if related to this problem - bugzilla link given by Joshua looks like a fairly likely culprit.
Yeah, it looks like it is probably unrelated to the 3Ware card. But still, take the time to read the whitepaper on performance with the 9000 series -- and the couple of tweaks they recommend.
The Linux kernel VM/buffer defaults _never_ assume there is an on-board storage intelligence that can command and queue a lot closer to the hardware than it can. ;->
-
Bryan J. Smith -
Joshua Baker-LePain -
Robert -
Sean O Sullivan