Using Xargs to Download Files
One of my favorite ways to batch download files on the CLI is using xargs, cat and cURL. I know, I know… there are more efficient ways of doing this - usually involving parallel, but I’ve found this to be the most reliable way when moving between Linux distros.
First, you need a file(files.txt) with a list of files to download. I like to name my files when they are downloaded, so my list also has a name for each file.
"filename.mp4" https://example.com/path/to/filename.mp4?foo=bar "filename2.mp4" https://example.com/path/to/filename2.mp4?baz=qux
Then to download these files we just run the following:
cat files.txt | xargs -n 2 curl -L -o
That’s it, the files will download one at a time with the names in the file.
If you’d like to do this in parallel, simply add the
cat files.txt | xargs -n 2 -P 2 curl -L -o