Deleting Large Number of Files from SD is Slow

[Gord_W]

Hi,

I have a rp3 with a large number of files (>100,000) each of which is about 80K bytes that need to be deleted from the SD card. They are image files that I'm capturing over a day. So that is several GB of data in total. I'm finding that the two methods that I've tried take a very long time, on the order of 10 minutes to remove them all.

The card is formatted ext4. It is 32GB Samsung EVO+

During the delete process the card seems to be written to in 80K chunks over and over again. So it doesn't appear that only the file allocation table (or whatever it is for ext4) is being updated. It seems that the data on the card is being overwritten.

Top shows that 1cpu is at or near 100% wait state, the others are 95%+ idle.

I've tried:
rm pathtofile/* This doesn't work as written as * causes an "argument list too long" error so I need to do it block by block.

find pathtodir -type f -delete This is slower than rm for 10,000 files and fails - goes nowhere after hours - for larger numbers of files.

I know that SD cards operate differently than hard disks, and writes take a longer time, but I thought that only the file allocation table would be written to so although there are a large number of files there wouldn't be that much writing to the card to delete them.

Any pointers would be appreciated that can move this from minutes to seconds time frame.

Gord_W

[Ernst]

Have you considered to delete the complete directory ?

Code: Select all

mv <directory> <directory.tmp>
mkdir <directory>
rm -rf <directory.tmp>

[mfa298]

Hi,

I have a rp3 with a large number of files (>100,000) each of which is about 80K bytes that need to be deleted from the SD card. They are image files that I'm capturing over a day. So that is several GB of data in total. I'm finding that the two methods that I've tried take a very long time, on the order of 10 minutes to remove them all.

Are these all in the same directory ?

If they are that would likely still be slow on a harddrive/ssd, ext4 isn't very good at handling lots of files in one directory.

Going forward it would probably be sensible to create a directory structure to store the files in so there aren't too many files per folder.

[k-pi]

You don't say, but if the card is separate from your operating system, just re format it.

[r3d4]

I've tried:
rm pathtofile/* This doesn't work as written as * causes an "argument list too long" error so I need to do it block by block.

Try xargs ?

arbitrarily long lists of parameters cannot be passed to a command,[1] so xargs breaks the list of arguments into sublists small enough to be acceptable.

[1] gnu.org/software/coreutils/faq/coreutils-faq.html#Argument-list-too-long

[Gavinmc42]

I usually use MC (Midnight commander)
This will delete directory and all files in it.
I also use rm *.* in the directory.
I use for deleting for thumbs and videos for security cams.