Maybe this is a silly question, but i have a few million files i need to delete but i can't just reformat the volume.
Right now the fastest thing i can think of is
nice -20 rm -Rf /folder-i-want-to-delete
is there a better or faster way to do this?
Thanks, Jamie
On Sat, Sep 29, 2007 at 04:43:43AM -0700, Jamie Lists wrote:
Maybe this is a silly question, but i have a few million files i need to delete but i can't just reformat the volume.
Right now the fastest thing i can think of is
nice -20 rm -Rf /folder-i-want-to-delete
is there a better or faster way to do this?
No, but if you want to re-use the directory name, rename it before removing it: mv folder-i-want-to-delete _removed_folder rm -fr _removed_folder & keep working...
On 9/29/07, Luciano Rocha strange@nsk.no-ip.org wrote:
On Sat, Sep 29, 2007 at 04:43:43AM -0700, Jamie Lists wrote:
Maybe this is a silly question, but i have a few million files i need to delete but i can't just reformat the volume.
Right now the fastest thing i can think of is
nice -20 rm -Rf /folder-i-want-to-delete
is there a better or faster way to do this?
No, but if you want to re-use the directory name, rename it before removing it: mv folder-i-want-to-delete _removed_folder rm -fr _removed_folder & keep working...
--
Thanks for the tip.. We just have millions and millions of files and it's taking FOREVER haha!
I thought maybe there might be some other command i'm not aware of that would wipe this stuff out faster ya know.
James
mv folder-to-be-deleted /dev/null ?
Geoff
Sent from my BlackBerry wireless handheld.
-----Original Message----- From: "Jamie Lists" jamielist@gmail.com
Date: Sat, 29 Sep 2007 14:19:12 To:"CentOS mailing list" centos@centos.org Subject: Re: [CentOS] Silly question - Anything faster than rm?
On 9/29/07, Luciano Rocha strange@nsk.no-ip.org wrote:
On Sat, Sep 29, 2007 at 04:43:43AM -0700, Jamie Lists wrote:
Maybe this is a silly question, but i have a few million files i need to delete but i can't just reformat the volume.
Right now the fastest thing i can think of is
nice -20 rm -Rf /folder-i-want-to-delete
is there a better or faster way to do this?
No, but if you want to re-use the directory name, rename it before removing it: mv folder-i-want-to-delete _removed_folder rm -fr _removed_folder & keep working...
--
Thanks for the tip.. We just have millions and millions of files and it's taking FOREVER haha!
I thought maybe there might be some other command i'm not aware of that would wipe this stuff out faster ya know.
James _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On Sat, Sep 29, 2007, Jamie Lists wrote:
On 9/29/07, Luciano Rocha strange@nsk.no-ip.org wrote:
On Sat, Sep 29, 2007 at 04:43:43AM -0700, Jamie Lists wrote:
Maybe this is a silly question, but i have a few million files i need to delete but i can't just reformat the volume.
Right now the fastest thing i can think of is
nice -20 rm -Rf /folder-i-want-to-delete
is there a better or faster way to do this?
No, but if you want to re-use the directory name, rename it before removing it: mv folder-i-want-to-delete _removed_folder rm -fr _removed_folder & keep working...
--
Thanks for the tip.. We just have millions and millions of files and it's taking FOREVER haha!
I thought maybe there might be some other command i'm not aware of that would wipe this stuff out faster ya know.
If there's not much other data on the file system it might be faster to copy that off, make a new file system, then copy the data back.
Bill -- INTERNET: bill@celestial.com Bill Campbell; Celestial Software LLC URL: http://www.celestial.com/ PO Box 820; 6641 E. Mercer Way FAX: (206) 232-9186 Mercer Island, WA 98040-0820; (206) 236-1676
Government's view of the economy could be summed up in a few short phrases: If it moves, tax it. If it keeps moving, regulate it. And if it stops moving, subsidize it -- Ronald Reagan
On Sat, 2007-09-29 at 14:19 -0700, Jamie Lists wrote:
On 9/29/07, Luciano Rocha strange@nsk.no-ip.org wrote:
On Sat, Sep 29, 2007 at 04:43:43AM -0700, Jamie Lists wrote:
Maybe this is a silly question, but i have a few million files i need to delete but i can't just reformat the volume.
<snip>
Thanks for the tip.. We just have millions and millions of files and it's taking FOREVER haha!
I thought maybe there might be some other command i'm not aware of that would wipe this stuff out faster ya know.
Unfortunately, it's not the command that is slow. A very large number of "hosekeeping" functions related to file system must be performed if you do this in place. Update free lists, used lists, etc. All involve reading and/or rewriting disk stuff. And remember that the cache of this stuff must be flushed periodically.
If it's ext3, mount as ext2 (someone else suggested it I think). You can also use tune2fs's dir_index to see if that helps (although I doubt it).
I forget how to do it now, but you can also increase the flush intervals to reduce the number of rewrites to the disk. I'm sure one of the folsk more knowledgeable can provide the hint for this.
If you have space, the suggestion by another that you copy off and back may be faster. If you have a list of all files/directories and a list of ones to delete, you can use comm to get the list of those to be kept (after sorting both lists) and feed this to cpio to archive the files to keep. The unmount and remake you file system and restore from cpio.
Cpio can also be used to copy directly to another directory/file system. This might be slower than the archiving process though, depending on several things.
Some folks prefer tar over cpio, but the same concept can apply.
James
<snip sig stuff>
HTH -- Bill
Some hint :
- use a partition dedicated for that, that you can format every time you need to reset it. - try other filesystem. Some filesystem are faster for delete operations. - try to change the journaling during the delete operations. If you use ext3, I'm thinking about something like:
# mount -o remount,noload /data # rm -rf /data/* # mount -o remount /data
or
# mount -o remount,data=writeback /data # rm -rf /data/* # mount -o remount,data=ordered /data
On 9/29/07, Jamie Lists jamielist@gmail.com wrote:
Maybe this is a silly question, but i have a few million files i need to delete but i can't just reformat the volume.
Right now the fastest thing i can think of is
nice -20 rm -Rf /folder-i-want-to-delete
is there a better or faster way to do this?
Thanks, Jamie _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Alain Spineux wrote:
Some hint :
- use a partition dedicated for that, that you can format every time
you need to reset it.
- try other filesystem. Some filesystem are faster for delete operations.
- try to change the journaling during the delete operations. If you
use ext3, I'm thinking about something like:
If you are going to change the way you store the files, just break them up into subdirectories that each only hold a few thousand files. That makes the lookups and other operations much faster.
Go with fat16 or fat32 instead of ext3fs.
Geoff
Sent from my BlackBerry wireless handheld.
-----Original Message----- From: Les Mikesell lesmikesell@gmail.com
Date: Sun, 30 Sep 2007 11:58:44 To:CentOS mailing list centos@centos.org Subject: Re: [CentOS] Silly question - Anything faster than rm?
Alain Spineux wrote:
Some hint :
- use a partition dedicated for that, that you can format every time
you need to reset it.
- try other filesystem. Some filesystem are faster for delete operations.
- try to change the journaling during the delete operations. If you
use ext3, I'm thinking about something like:
If you are going to change the way you store the files, just break them up into subdirectories that each only hold a few thousand files. That makes the lookups and other operations much faster.
-- Les Mikesell lesmikesell@gmail.com
_______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On Sun, Sep 30, 2007 at 07:14:54PM +0000, gjgowey@tmo.blackberry.net wrote:
Go with fat16 or fat32 instead of ext3fs.
For performance???
On Sun, Sep 30, 2007, Matthew Miller wrote:
On Sun, Sep 30, 2007 at 07:14:54PM +0000, gjgowey@tmo.blackberry.net wrote:
Go with fat16 or fat32 instead of ext3fs.
For performance???
For humor, a very bad joke.
Bill -- INTERNET: bill@celestial.com Bill Campbell; Celestial Software LLC URL: http://www.celestial.com/ PO Box 820; 6641 E. Mercer Way FAX: (206) 232-9186 Mercer Island, WA 98040-0820; (206) 236-1676
I don't make jokes, I just watch the Government and report the facts... Will Rogers
For deleting performance. When it comes to deleting files, fat seems fairly quick.
Geoff
Sent from my BlackBerry wireless handheld.
-----Original Message----- From: Matthew Miller mattdm@mattdm.org
Date: Sun, 30 Sep 2007 16:14:48 To:CentOS mailing list centos@centos.org Subject: Re: [CentOS] Silly question - Anything faster than rm?
On Sun, Sep 30, 2007 at 07:14:54PM +0000, gjgowey@tmo.blackberry.net wrote:
Go with fat16 or fat32 instead of ext3fs.
For performance???
-- Matthew Miller mattdm@mattdm.org http://mattdm.org/ Boston University Linux ------> http://linux.bu.edu/ _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Or another way of putting it: nothing makes files disappear faster than fat.
Geoff
Sent from my BlackBerry wireless handheld.
-----Original Message----- From: gjgowey@tmo.blackberry.net
Date: Sun, 30 Sep 2007 20:37:34 To:"CentOS mailing list" centos@centos.org Subject: Re: [CentOS] Silly question - Anything faster than rm?
For deleting performance. When it comes to deleting files, fat seems fairly quick.
Geoff
Sent from my BlackBerry wireless handheld.
-----Original Message----- From: Matthew Miller mattdm@mattdm.org
Date: Sun, 30 Sep 2007 16:14:48 To:CentOS mailing list centos@centos.org Subject: Re: [CentOS] Silly question - Anything faster than rm?
On Sun, Sep 30, 2007 at 07:14:54PM +0000, gjgowey@tmo.blackberry.net wrote:
Go with fat16 or fat32 instead of ext3fs.
For performance???
-- Matthew Miller mattdm@mattdm.org http://mattdm.org/ Boston University Linux ------> http://linux.bu.edu/ _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On Sun, Sep 30, 2007, gjgowey@tmo.blackberry.net wrote:
For deleting performance. When it comes to deleting files, fat seems fairly quick.
It even can magically delete ones you wanted.
Bill -- INTERNET: bill@celestial.com Bill Campbell; Celestial Software LLC URL: http://www.celestial.com/ PO Box 820; 6641 E. Mercer Way FAX: (206) 232-9186 Mercer Island, WA 98040-0820; (206) 236-1676
UNIX was not designed to stop you from doing stupid things, because that would also stop you from doing clever things. -- Doug Gwyn
No journaling, no worries. :>
Geoff
Sent from my BlackBerry wireless handheld.
-----Original Message----- From: Bill Campbell centos@celestial.com
Date: Sun, 30 Sep 2007 13:44:00 To:centos@centos.org Subject: Re: [CentOS] Silly question - Anything faster than rm?
On Sun, Sep 30, 2007, gjgowey@tmo.blackberry.net wrote:
For deleting performance. When it comes to deleting files, fat seems fairly quick.
It even can magically delete ones you wanted.
Bill -- INTERNET: bill@celestial.com Bill Campbell; Celestial Software LLC URL: http://www.celestial.com/ PO Box 820; 6641 E. Mercer Way FAX: (206) 232-9186 Mercer Island, WA 98040-0820; (206) 236-1676
UNIX was not designed to stop you from doing stupid things, because that would also stop you from doing clever things. -- Doug Gwyn _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
-
Alain Spineux -
Bill Campbell -
gjgowey＠tmo.blackberry.net -
Jamie Lists -
Les Mikesell -
Luciano Rocha -
Matthew Miller -
William L. Maltby