I have an issue with a busy CentOS server exporting iSCSI and NFS/SMB shares. Some of the files are very large, and when they get deleted IO climbs to an unacceptable rate. Is there a way to purge a file with little to no IO overhead on ext3?
Thanks! jlc
On Sun, Mar 1, 2009 at 3:39 PM, Joseph L. Casale JCasale@activenetwerx.com wrote:
I have an issue with a busy CentOS server exporting iSCSI and NFS/SMB shares. Some of the files are very large, and when they get deleted IO climbs to an unacceptable rate. Is there a way to purge a file with little to no IO overhead on ext3?
Is it the ext3 or the exports. M
1) What is the local storage on (controller, disks, raid, etc) 2) Does the IO go up if you do the delete locally, remotely or both 3) What is your definition of unacceptable rate?
I have seen rm on NFS tie up things on various servers.. not sure if its something the protocol does or something with the filesystem.
Is it the ext3 or the exports. M
It's on the ext3 fs when rm'ed locally via ssh (I export some ext3 fs's over NFS and SMB).
- What is the local storage on (controller, disks, raid, etc)
HP MSA20 (scsi => sata discs)
- Does the IO go up if you do the delete locally, remotely or both
- What is your definition of unacceptable rate?
Well, enough that the whole machine becomes slightly unresponsive. I tried ionice'ing it at -c3 but that had no effect. I will play with different schedulers and see if I can find one that matches my needs.
Thanks! jlc
Joseph L. Casale schrieb:
I have an issue with a busy CentOS server exporting iSCSI and NFS/SMB shares. Some of the files are very large, and when they get deleted IO climbs to an unacceptable rate. Is there a way to purge a file with little to no IO overhead on ext3?
Thanks! jlc
Have you tried to delete locally, instead of over NFS?
Maybe by deleting over SMB from a Windows machine, the file is not deleted but rather moved to a "Trash" folder on a different disk (which would explain the I/O)? (Same could happen with a Unix desktop, like KDE)
Have you tried the "unlink" command instead of "rm" ?
Kay
On Mar 2, 2009, at 7:06 AM, "Joseph L. Casale" <JCasale@activenetwerx.com
wrote:
Have you tried to delete locally, instead of over NFS?
As I said, the deletes are done locally via ssh.
Have you tried the "unlink" command instead of "rm" ?
Nope, but I will now!
If it's not too late you could try XFS as I believe it is efficient in handling large files.
Or you could try a different journal option to your ext3 fs.
Problem is the fragmentation that occurs with these files created over NFS and how ext3 deallocates extents.
-Ross
On Mar 2, 2009, at 11:05 AM, Ross Walker rswwalker@gmail.com wrote:
On Mar 2, 2009, at 7:06 AM, "Joseph L. Casale" <JCasale@activenetwerx.com
wrote:
Have you tried to delete locally, instead of over NFS?
As I said, the deletes are done locally via ssh.
Have you tried the "unlink" command instead of "rm" ?
Nope, but I will now!
If it's not too late you could try XFS as I believe it is efficient in handling large files.
Or you could try a different journal option to your ext3 fs.
Try the data=writeback option which emulates the way XFS handles log operations. With UPS and/or battery backed caching controller the risk should be minimum.
That will help, but ext3 is not good at handling large files at all, so if you can switch to XFS that will make a bigger improvement. Make sure XFS journal is set to max size and noatime and nodiratime are set to make regular/small file operations perform decently.
-Ross
Try the data=writeback option which emulates the way XFS handles log operations. With UPS and/or battery backed caching controller the risk should be minimum.
That will help, but ext3 is not good at handling large files at all, so if you can switch to XFS that will make a bigger improvement. Make sure XFS journal is set to max size and noatime and nodiratime are set to make regular/small file operations perform decently.
I just walked in that room and noticed some issues with that server so I may be rebuilding to new hardware! I will explore xfs on the rebuild.
Thanks everyone! jlc
On Mar 2, 2009, at 2:34 AM, Kay Diederichs wrote:
Joseph L. Casale schrieb:
I have an issue with a busy CentOS server exporting iSCSI and NFS/ SMB shares. Some of the files are very large, and when they get deleted IO climbs to an unacceptable rate. Is there a way to purge a file with little to no IO overhead on ext3?
Thanks! jlc
Have you tried to delete locally, instead of over NFS?
Maybe by deleting over SMB from a Windows machine, the file is not deleted but rather moved to a "Trash" folder on a different disk (which would explain the I/O)? (Same could happen with a Unix desktop, like KDE)
Have you tried the "unlink" command instead of "rm" ?
Kay
I've seen length delete times too when deleting a 30+ GB file on an ext3 filesystem, locally. I don't think that Windows (XP at least) will move remote files to a "trash" folder, only local files. Though there is no telling what it might do if you have some 3rd party application installed that may provide an "undelete" functionality.
I haven't researched it enough to check I/O stats.
On Mon, 2009-03-02 at 17:36 -0600, Kevin Krieser wrote:
On Mar 2, 2009, at 2:34 AM, Kay Diederichs wrote:
Joseph L. Casale schrieb:
I have an issue with a busy CentOS server exporting iSCSI and NFS/ SMB shares. Some of the files are very large, and when they get deleted IO climbs to an unacceptable rate. Is there a way to purge a file with little to no IO overhead on ext3?
Thanks! jlc
Have you tried to delete locally, instead of over NFS?
Maybe by deleting over SMB from a Windows machine, the file is not deleted but rather moved to a "Trash" folder on a different disk (which would explain the I/O)? (Same could happen with a Unix desktop, like KDE)
Have you tried the "unlink" command instead of "rm" ?
Kay
I've seen length delete times too when deleting a 30+ GB file on an ext3 filesystem, locally. I don't think that Windows (XP at least) will move remote files to a "trash" folder, only local files. Though there is no telling what it might do if you have some 3rd party application installed that may provide an "undelete" functionality.
I haven't researched it enough to check I/O stats.
This is mostly a problem with windows clients especially under Samba Server. Read the following data from Intel where they tested this under Samba. It's to do with the way windows accesses the file share. This is not something that has just rose up from the dead. http://software.intel.com/en-us/articles/windows-client-cifs-behavior-can-sl... I have indeed noticed the difference that Intel talks about with using XFS and NTFS It has nothing to do with rm -f or unlink! It has everything to do with the amount of file extents that are created for the file (fragmentation). You guys can replicate this for your self in a test machine.
JohnStanley
-
JohnS -
Joseph L. Casale -
Kay Diederichs -
Kevin Krieser -
Ross Walker -
Stephen John Smoogen