Hi All.
I currently have a problem on my backup server with very large number of small files in a large number of directories. I would like to delete them as fast as possible. Currently I use:
rsync -a --delete /empty_directory/ dir_to_clean/
I've read that rsync will be faster than rm or find. Can you recommend something? I use an ext4 filesystem.
Best regards, Rafal.
On Fri, 23 Aug 2013 07:34:50 +0200 Rafał Radecki wrote:
Can you recommend something? I use an ext4 filesystem.
Without actually trying to to see, I suspect that a C program that calls remove() would probably be faster than just about anything else.
On Thu, 2013-08-22 at 23:40 -0600, Frank Cox wrote:
On Fri, 23 Aug 2013 07:34:50 +0200 Rafał Radecki wrote:
Can you recommend something? I use an ext4 filesystem.
Without actually trying to to see, I suspect that a C program that calls remove() would probably be faster than just about anything else.
I agree with what you are saying, and I think you may even want to use unlink() if they are all files, no dirs. remove() actually will call unlink() so you may be able to save few function calls and logical statements.
On 23.08.2013 07:52, Xinyun Zhou wrote:
On Thu, 2013-08-22 at 23:40 -0600, Frank Cox wrote:
On Fri, 23 Aug 2013 07:34:50 +0200 Rafał Radecki wrote:
Can you recommend something? I use an ext4 filesystem.
Without actually trying to to see, I suspect that a C program that calls remove() would probably be faster than just about anything else.
I agree with what you are saying, and I think you may even want to use unlink() if they are all files, no dirs. remove() actually will call unlink() so you may be able to save few function calls and logical statements.
I doubt saving functions calls is going to gain you anything in this case as 99.9% of the time the rm takes is on disk I/O. If you want to reduce the rm time you have to find a way to reduce the disk I/O it requires.
Regards, Dennis
On Fri, Aug 23, 2013 at 12:40:51PM +0200, Dennis Jacobfeuerborn wrote:
I doubt saving functions calls is going to gain you anything in this case as 99.9% of the time the rm takes is on disk I/O. If you want to reduce the rm time you have to find a way to reduce the disk I/O it requires.
Correct.
If it's a whole directory (tree) that needs removing then I find
mv dir dir.o ; mkdir dir ; chown ##:## dir; chmod ### dir ; rm -r dir.o &
type stuff works just fine; the rm can chunk along in the background while there's now a nice clean empty directory for the application.
In article 20130823104855.GA15299@mercury.spuddy.org, Stephen Harris lists@spuddy.org wrote:
On Fri, Aug 23, 2013 at 12:40:51PM +0200, Dennis Jacobfeuerborn wrote:
I doubt saving functions calls is going to gain you anything in this case as 99.9% of the time the rm takes is on disk I/O. If you want to reduce the rm time you have to find a way to reduce the disk I/O it requires.
Correct.
If it's a whole directory (tree) that needs removing then I find
mv dir dir.o ; mkdir dir ; chown ##:## dir; chmod ### dir ; rm -r dir.o &
type stuff works just fine; the rm can chunk along in the background while there's now a nice clean empty directory for the application.
You can even dispense with the ## variable parts. And I would get the owner and modes correct before making the new directory appear:
mkdir dir.n && chown --reference=dir dir.n && chmod --reference=dir dir.n && \ mv dir dir.o && mv dir.n dir && echo rm -fr dir.o ; echo dir.o deleted | batch
That will do the removal in a batch job without hanging on to the tty, and will email you a quick note (the output from batch) when it's done.
Cheers Tony
----- Original Message ----- | Hi All. | | I currently have a problem on my backup server with very large number | of | small files in a large number of directories. I would like to delete | them | as fast as possible. Currently I use: | | rsync -a --delete /empty_directory/ dir_to_clean/ | | I've read that rsync will be faster than rm or find. | Can you recommend something? I use an ext4 filesystem. | | Best regards, | Rafal.
If it's on a different file system than mkfs is the fastest. ;)