[GoLUG] Gigantic deletion

[Steve Litt]

Hi all,

My 6.9TB /scratch partition got up to 83% full, which triggers alarm
bells even on spinning rust. In my investigation I found that a backup
set , of /home/slitt, on /scratch, contained thousands, tens of
thousands, or hundreds of thousands of files, in one deep directory
under .wine. Because this was a year old backup, I performed:

rm -Rf .wine

This mass deletion has been running for about 14 hours. 

Let me ask you a question: What kind of animal lets their software
write more than 1000, let alone more than 100,000, files in the same
directory? When files exceed 1000, lookup becomes slower. By the time
you have 100,000 files, I'd estimate you can find maybe 100 files per
second, and to do a deletion your Ext4 partition must search each file.

When I make a program destined to make a lot of files, I find a way to
put them in a tree, with leaf directories each holding less (and
hopefully many less) than 1000 files, so files can be found by just
navigating down the tree. But nooooooo!

/scratch has gone from 83% full to 76% full in those 14 hours. There's
plenty of other stuff on /scratch so I'll be lucky to get to 50% or
60%, given that most of these hundreds of thousands of files are in the
20KB range, while this /scratch contains many multi-megabyte and a few
multi-gigabyte files. But at least with all these little basuras
deleted, hopefully I can do a find command and find the big files.

One nice thing is that the more of these files that get deleted, the
faster each deletion will go.

When this is done, /scratch will have *a lot more* free inodes.

SteveT

Steve Litt 
http://444domains.com