Using 'ls' and 'xargs' to manage large numbers of files

I have previously noted elsewhere that using xargs with ls comes in handy when trying to remove more than 64K files. This is something that I ran into back in 1998 - a system operator at the large corporation I worked for back then was trying to clean up some old log files from a directory. These log files went back months, and there were about 75,000 files he wanted to delete.

Because it was impossible to simply do a 'rm *', or use a '*' in any way to create a single rm command, and because there were so many files to be deleted, I suggested using 'ls' with 'xargs'. Now it would have been possible to 'rm' multiple times against shorter lists, but that would not have been very productive - it would have worked out to running 'rm' more than twenty times with the different filelist parameters. It was easier to use 'xargs'.

xargs is a command that will take list input and execute the same command over and over for each item on the list.

To demonstrate a bit of the problem, you can easily create 90,000 empty files with this script:

#!/bin/bash
for i in `seq 1 30000`;
do
        touch $i.tmp
        touch $i.txt
        touch $i.lst
done

Save this as 'tmp.sh' in an empty directory. Set the script to be executable with "chmod +x tmp.sh" and run it:
>$ ./tmp.sh

It'll take maybe five minutes to finish creating 90,000 empty files. Now, if you attempt to delete all 90,000 files at once with "rm *", you'll get an error similar to "bash: /bin/rm: Argument list too long"

At this point you could try to do "rm *.tmp; rm *.txt; rm *.lst" and be done with it, but you'll get the same error. You could try "rm 26*" and that would work, but how many times do you want to execute rm?

So what do you do? Now that you executed that script I gave you above, you've got 90,000 empty files to get rid of. Silly reader - why did you listen to me?

'xargs' to the rescue!

Feed the list of all those empty files to the rm command one at a time, and you're done:
>$ ls -tr |xargs -t -I{} rm -f {}

BTW - the command as constructed above will also delete the shell script used to create the files in the first place.

Also, I am not responsible for any mess anyone makes of their directories by following my instructions above. I've tested the above steps on my own machines, but I'm not prescient enough to anticipate all possible misreadings, so caveat emptor.

And remember, Trix are for kids.