Fastest way of removing very large number of files?

Question

I currently have a problem on my backup server with very large number of small files in a large number of directories. I would like to delete them as fast as possible. Currently I use:

rsync -a --delete /empty_directory/ dir_to_clean/

I've read that rsync will be faster than rm or find. Can someone recommend something? I use an ext4 filesystem.

Answer 1

A much faster way of deleting files – the “find” command has a “-delete” flag built right in!!

[root@localhost Desktop]# find . -type f -delete

Using this method, I was deleting files at a rate of about 2000 files/second.

You can also show the filenames as you’re deleting them:

[root@localhost Desktop]# find . -type d -print -delete

Or even show how many files will be deleted, then time how long it takes to delete them:

[root@localhost Desktop]# ls -1 | wc -l && time find . -type f -delete
real    0m3.660s
user    0m0.036s
sys     0m0.552s

Answer 2

I suspect that a C program that calls remove() would probably be faster than just about anything else.

Comments

I agree with what you are saying, and I think you may even want to use unlink() if they are all files, no dirs. remove() actually will call unlink() so you may be able to save few function calls and logical statements.

Answer 3

If it's on a different file system than mkfs is the fastest. ;)