In near past I was presented with the interesting problem in the Linux OS world. The standard solutions of this problem in standard cases involve two or three commands to end troubles instantly. However, to my surprise, Linux was not prepared for any solutions to this particular case. This post shares my solution.
The problem description sounds too easy: need to delete files in one directory. But not just some amount of files – insanely huge amount of files, more than 10 millions of them. Maybe the backstory of “why so much” is interesting, but lets keep to the point. Linux tools, intended to execute file managing commands, can hold and process only definite list of files. However, when file list is too large, all standard command line tools fail to operate because of their internal limits. I mean, filesystem (ext3, ext4, ufs) can hold large amount of files – several millions of inodes usually – and you can access and move them if you do it by calling full path to each individual file. However, if you have files with random names (maybe even in different path depths), you have no tool to manage them.
(more…)