Simple way to Copy and Move a Large Number of Files using Terminal

Want to copy or move 5,00,000 or more files at once, but the terminal complains that the “argument list is too long.”

If you have created many files using the method described in this article (“create multiple files and directories at once in a Linux terminal“), it may be hard to move or copy them all at once to a new location.

Fortunately, there are a few methods for quickly and easily moving or copying a large number of files in Linux. One such method is to use the command-line tools cp, tar, and rsync, which allow for the copying and moving of large numbers of files quickly and easily, but they have a certain threshold after which a command will throw an error.

So, let’s take a look at how to efficiently copy and move a large number of files from the command line.

Create a problem to move large number of files using touch command

For the sake of clarity, let me use my system to create 5,00,000 files with the touch command, which will take a few minutes, and then we’ll test our commands on the newly created files.

If you want, you can also create a large number of files by executing the following command:

$ for i in {1..500000};do touch filename$i.txt; done

Once it is ready, let me run the mv command to move all the files to the new location. As expected, the terminal prints “mv * temp zsh: argument list too long: mv.”

Solve a problem to move large number of files using find command

The reason is simple: the kernel has some limitations that vary from operating system to operating system that restrict the number of arguments you can pass on a single command line.

And when you execute commands like mv, cp, rm, or tar, its argument list becomes too long as the number of files increases beyond the kernel’s limit, and this will eventually lead to the argument list too long error.

To solve this problem, we can use the find command with the -exec option to execute our command on the large number of files

But if you want to know the kernel limit, you can run the command getconf ARG_MAX in your terminal, which will give you the maximum number of arguments that can be passed on a single command line to avoid the arguments error.

All right, that’s enough chit-chat! Let me now demonstrate a method for quickly copying or moving a large number of files from a terminal.

Command Section

By utilizing the find command with -exec option, it allows for a much more efficient copying of large numbers of files, making it easier for users to access their data and store them in the location specified

When you want to copy a large number of files, you have to use the find command with -exec, which will copy the file from the result of a find command and move it to the location that is specified.

Here is the syntax that you need to use on your terminal; just replace data with the actual one.

$ find /FILE-PATH/ -name "PASS-FILE-PATTERN" -exec cp {} "/PASS-TARGET-DIRECTORY/" \;

Let’s say I want to copy all the files in /home/shen/test-arena to the new directory /home/shen/test-dir, then I’ll create the following command to copy all the files.

$ find /home/shen/test-arena -name "filename*" -exec cp {} -v /home/shen/test-dir \;

Same I can do for moving all the large file from test-arena to test-dir by just replacing the cp command to mv and rest all the things will be the same

$ find /home/shen/test-arena -name "filename*" -exec mv {} -v /home/shen/test-dir \;

And you can even restrict the find command to looking for files only in the current directory using the -maxdepth parameters, like shown in the below command:

$ find . -maxdepth 1 -name "file_*" -exec cp {} -v target/ \;
or
$ find . -maxdepth 1 -name "file_*" -exec mv {} -v target/ \;

Wrap up

I hope this article has helped you understand how to move and copy a lot of files in a Unix-based system and how to find out what the kernel limit is.

If you know of any other methods of moving and copying large numbers of files, please share them in the comments section.

With that in mind, I’ll wrap up this guide. If you have any further suggestions, please share them below.

Leave a Reply