Dave, I have searched the web over and can’t seem to find an appropriate answer to this question. I have a need for a script that will create millions of 1-5Kbyte files in a linux filesystem for testing purposes. I have to do this on a regular basis and it would be really cool to have it scripted. I am not sure what the quickest way to do this would be?
A bit weird as questions go, but I understand exactly what you’re talking about and the good news is that you can accomplish this task quickly and easily without having to have a “for i=1…1000000 do; touch x; done”.
The trick is to use a command called split which, as the man page explains, “The split utility reads the given file (or standard input if no file is specified) and breaks it up into files of 1000 lines each.”
How would you use this, then? Create a big file and use split with its useful “-b size” flag to create the millions of files you need. Let’s step through the process…
Let’s assume that you want to have, literally, a million 2.5K files. Now let’s assume that you want to run the command one hundred times to produce this, which means that the source file needs to be 1/100th of 2.5 million kilobytes in size, or 250 megabytes. (yeah, I know, a K is 1024 bytes, not 1000 bytes)
Go look at your log files: you might well have a file sitting in /var/log or a similar location that’s already that size. I have a /var/log/messages file on my system that clearly needs to be trimmed – it’s 381MB in size. Perfect!
You can find your own really big files by using find. Here’s how I scrounged my server to find something really big:
$ sudo find . -type f -size +10000 -print
Typically, find works in terms of 512-byte blocks on the size specifier, so this search isn’t looking for files that are greater than 100MB, but greater than 50MB. No worries, though, it will invariably find a dozen or more files and one of ’em will be perfect for our needs. If you can’t find one that’s quite the right size, create the size you need:
$ ls -lh ./private/var/vm/swapfile0
-rw------T 1 root wheel 64M 10 Feb 08:21 ./private/var/vm/swapfile0
$ cat "./private/var/vm/swapfile0" "./private/var/vm/swapfile0" > BigFile
$ ls -l /tmp/BigFile
-rw-r--r-- 1 taylor wheel 134217728 10 Feb 09:57 /tmp/BigFile
Now we have a good size file, one that’ll cover your needs. Time to move into a subdirectory and split out oodles of subfiles:
$ ls | head
xaa xhe xoi xvm ycq yju yqy yyc zfg zmk
xab xhf xoj xvn ycr yjv yqz yyd zfh zml
xac xhg xok xvo ycs yjw yra yye zfi zmm
xad xhh xol xvp yct yjx yrb yyf zfj zmn
xae xhi xom xvq ycu yjy yrc yyg zfk zmo
xaf xhj xon xvr ycv yjz yrd yyh zfl zmp
xag xhk xoo xvs ycw yka yre yyi zfm zmq
xah xhl xop xvt ycx ykb yrf yyj zfn zmr
xai xhm xoq xvu ycy ykc yrg yyk zfo zms
xaj xhn xor xvv ycz ykd yrh yyl zfp zmt
Voila! I think you can run with it from here. Good luck.
Original: https://www.askdavetaylor.com/how_can_i_quickly_create_millions_of_tiny_files_in_linux/