doubleudee1
Posts: 164
Joined: Fri Nov 30, 2018 2:39 pm
Location: Wales

how to remove 320,000 files?

Fri Sep 25, 2020 3:12 pm

I have a directory which recieves files from my security camera.
I have a file which has been running for 14 months or so which sorts the files weekly and then moves them from that folder into a new one created for the dates they are made for.
I have not bothered with it for about 6 months knowing that it had been running smoothly, but I checked today and found that the files have not been moved to the other folders since June, now there are over 320,000 files in this directory (on a hard disk where they had been moved to by the initial script as should be).
I cannot get the move script to run as it keeps giving the error './CreatejpgDirsF_WithDays.sh: line 6: /bin/mv: Argument list too long' ?

Can anyone help please

edit:-
If its any help here's the troublesome line in the script (I think)

Code: Select all

mv /home/pi/mnt/CamFiles/FrontOfHouse/*.jpg to /home/pi/mnt/CamFiles/FrontOfHouse/jpgFiles/
Last edited by doubleudee1 on Fri Sep 25, 2020 3:22 pm, edited 1 time in total.
Make the most of your family and friends , your children grow up too quickly and you don't notice yourself ageing, friends dissapearing, moving on. You make plans for your future, but they can all be smashed in an instant, live life and enjoy. :)

User avatar
kerry_s
Posts: 1557
Joined: Thu Jan 30, 2020 7:14 pm

Re: how to remove 320,000 files?

Fri Sep 25, 2020 3:21 pm

it would help if you posted line 6 of your script so we can see what the error is referring to.

PhilE
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 3455
Joined: Mon Sep 29, 2014 1:07 pm
Location: Cambridge

Re: how to remove 320,000 files?

Fri Sep 25, 2020 3:22 pm

Something along the lines of this?:

Code: Select all

find <the-directory> -type f -exec mv {} <destination-folder> \;

doubleudee1
Posts: 164
Joined: Fri Nov 30, 2018 2:39 pm
Location: Wales

Re: how to remove 320,000 files?

Fri Sep 25, 2020 3:24 pm

Here is the script:-

Code: Select all

#!/bin/bash
#source $HOME/.bash_profile
## this file is being ran from the directory on the HDD '/CamFiles/FrontOfHouse/' (which is connected to the pi)
########FIRST MOVE ALL FILES TO /home/pi/mnt/CamFiles/FrontOfHouse/jpgFiles/################

mv /home/pi/mnt/CamFiles/FrontOfHouse/*.jpg to /home/pi/mnt/CamFiles/FrontOfHouse/jpgFiles/

# create an array with all the files/dir inside ~/CamFiles/FrontOfHouse/

arr=(/home/pi/mnt/CamFiles/FrontOfHouse/jpgFiles/*.jpg)  #  *.jpg = "ARC20190315080521.jpg" (date starts at pos 47)

# iterate through array using a counter
for ((i=0; i<${#arr[@]}; i++)); do

##check if array holds elements
####if(arr.length > 1)
################### SET UP VARIABLES FOR CODE ###########################
   j=${arr[$i]:47:6} ## 201903
echo "j=  " ${j}
   myday=${arr[$i]:53:2} #01
echo "today is "${myday}
   myday2=${arr[$i]:47:8} #20190301
echo "today is "${myday2}
   k=${arr[$i]:0:44} ## /home/pi/mnt/CamFiles/FrontOfHouse/jpgFiles/
echo "k= " ${k}
   h=${arr[$i]:0:65} ## /home/pi/mnt/CamFiles/FrontOfHouse/jpgFiles/ARC20190315080521.jpg
echo "h= " ${l}
   m=${arr[$i]:0:65} ##/home/pi/mnt/CamFiles/FrontOfHouse/jpgFiles/ARC20190315080521.jpg
echo "m- " ${m}
   n=${k}${j}              ## /home/pi/mnt/CamFiles/FrontOfHouse/jpgFiles/201903
echo "n= "${n}
  o=${arr[$i]:47:6}    ## 201903
echo "o=  " ${o}
oo=${k}${myday2} ### /home/pi/mnt/CamFiles/FrontOfHouse/jpgFiles/20190301
echo "oo=  " ${oo}
   l="${k}${j}"            ### /home/pi/mnt/CamFiles/FrontOfHouse/jpgFiles/201903
echo "l= " ${l}
###################  CREATE ANY NEW DIRECTORIES THAT NEED CREATING ##############
 mkdir -p "${oo}" ### this creates new directories in location of this exec file
echo " DIR " ${oo} " Built"
################# MOVE ALL FILES TO RELEVANT DIRECTORIES NEXT ##################
######HAVE TO MOVE THEM DAILY HERE, SO THERE WILL BE A LOT OF DATA TO CHECK ##############
### mv pathtofileA pathtofileB ###
aa=${myday2}
if [ ${myday2} = ${aa} ]; then
       echo "moving " "${m}" " to " "${oo}"
     mv "${m}" "${oo}"
fi

##end of check for array elements
###fi
done


################# ABOVE OUTPUTS HAVE CREATED DIRS AND PUT FILES IN THEM ################
Make the most of your family and friends , your children grow up too quickly and you don't notice yourself ageing, friends dissapearing, moving on. You make plans for your future, but they can all be smashed in an instant, live life and enjoy. :)

PhilE
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 3455
Joined: Mon Sep 29, 2014 1:07 pm
Location: Cambridge

Re: how to remove 320,000 files?

Fri Sep 25, 2020 3:29 pm

You shouldn't have a "to" in your "mv" parameters.

doubleudee1
Posts: 164
Joined: Fri Nov 30, 2018 2:39 pm
Location: Wales

Re: how to remove 320,000 files?

Fri Sep 25, 2020 3:36 pm

Sory, that was my test script, I found out that shouldn't be there and removed it when I installed it.
Here is the script as it is now, taken from the pi itself.

Code: Select all

#!/bin/bash
#source $HOME/.bash_profile
## this file is being ran from the directory on the HDD '/CamFiles/FrontOfHouse/' (which is connected$

mkdir -p /home/pi/mnt/CamFiles/FrontOfHouse/jpgFiles
mv /home/pi/mnt/CamFiles/FrontOfHouse/*.jpg  /home/pi/mnt/CamFiles/FrontOfHouse/jpgFiles/
#mv /home/pi/mnt/CamFiles/FrontOfHouse/201910_jpg/*.jpg  /home/pi/mnt/CamFiles/FrontOfHouse/jpgFiles/

# create an array with all the files/dir inside ~/CamFiles/FrontOfHouse/

arr=(/home/pi/mnt/CamFiles/FrontOfHouse/jpgFiles/*.jpg)  #  *.jpg = "ARC20190315080521.jpg" (date sta$

# iterate through array using a counter
for ((i=0; i<${#arr[@]}; i++)); do
################### SET UP VARIABLES FOR CODE ###########################
   j=${arr[$i]:47:6} ## 201903
echo "j=  " ${j}

   myday=${arr[$i]:53:2} #01
echo "Day to move is    "${myday}

   myday2=${arr[$i]:47:8} #20190301
echo "Dir for file is    "${myday2}

   k=${arr[$i]:0:44} ## /home/pi/mnt/CamFiles/FrontOfHouse/jpgFiles/
echo "k= " ${k}
   h=${arr[$i]:0:65} ## /home/pi/mnt/CamFiles/FrontOfHouse/jpgFiles/ARC20190315080521.jpg
echo "h= " ${l}
   m=${arr[$i]:0:65} ##/home/pi/mnt/CamFiles/FrontOfHouse/jpgFiles/ARC20190315080521.jpg
echo "m- " ${m}
   n=${k}${j}              ## /home/pi/mnt/CamFiles/FrontOfHouse/jpgFiles/201903
echo "n= "${n}
  o=${arr[$i]:47:6}    ## 201903
echo "o=  " ${o}

oo=${k}${myday2} ### /home/pi/mnt/CamFiles/FrontOfHouse/jpgFiles/20190301
echo "oo=  " ${oo}

   l="${k}${j}"            ### /home/pi/mnt/CamFiles/FrontOfHouse/jpgFiles/201903
echo "l= " ${l}

###################  CREATE ANY NEW DIRECTORIES THAT NEED CREATING ##############
 mkdir -p "${oo}" ### this creates new directories for location of each file
echo " DIR " ${oo} " Built"
################# MOVE ALL FILES TO RELEVANT DIRECTORIES NEXT ##################
######HAVE TO MOVE THEM DAILY HERE, SO THERE WILL BE A LOT OF DATA TO CHECK ##############
### mv pathtofileA pathtofileB ###
aa=${myday2}

if [ ${myday2} = ${aa} ]; then
       echo "moving " "${m}" " to " "${oo}"
        mv "${m}" "${oo}"
fi

done
Make the most of your family and friends , your children grow up too quickly and you don't notice yourself ageing, friends dissapearing, moving on. You make plans for your future, but they can all be smashed in an instant, live life and enjoy. :)

doubleudee1
Posts: 164
Joined: Fri Nov 30, 2018 2:39 pm
Location: Wales

Re: how to remove 320,000 files?

Fri Sep 25, 2020 3:42 pm

I am thinking maybe there are too many files in the list for the pi's memory size?

Could I manually move them to temporarily created directories in blocks of maybe 20,000 or so?
Last edited by doubleudee1 on Fri Sep 25, 2020 3:45 pm, edited 1 time in total.
Make the most of your family and friends , your children grow up too quickly and you don't notice yourself ageing, friends dissapearing, moving on. You make plans for your future, but they can all be smashed in an instant, live life and enjoy. :)

User avatar
jahboater
Posts: 6278
Joined: Wed Feb 04, 2015 6:38 pm
Location: Wonderful West Dorset

Re: how to remove 320,000 files?

Fri Sep 25, 2020 3:45 pm

doubleudee1 wrote:
Fri Sep 25, 2020 3:42 pm
I am thinking maybe there are too many files in the list for the pi's memory size?
No, its the parameter list for the program after its been expanded by the shell is too long.
Pi4 8GB running PIOS64 Lite

PhilE
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 3455
Joined: Mon Sep 29, 2014 1:07 pm
Location: Cambridge

Re: how to remove 320,000 files?

Fri Sep 25, 2020 3:51 pm

The Pi has more than enough memory for that. In a real programming language you'd use opendir and readdir to iterate through the files. but if my simple unconditional mv isn't sophisticated enough you could refactor your script to be called from find:

Code: Select all

find <the-directory> -type f -exec mv_script {} \;
where mv_script is called repeatedly with a single parameter - the name of the file (including the path) to move (or not).

doubleudee1
Posts: 164
Joined: Fri Nov 30, 2018 2:39 pm
Location: Wales

Re: how to remove 320,000 files?

Fri Sep 25, 2020 3:56 pm

Thanks, I'll give it a shot and get back here to let you know if it worked. It may not be tonight though.
Thks again.
Make the most of your family and friends , your children grow up too quickly and you don't notice yourself ageing, friends dissapearing, moving on. You make plans for your future, but they can all be smashed in an instant, live life and enjoy. :)

GlowInTheDark
Posts: 1067
Joined: Sat Nov 09, 2019 12:14 pm

Re: how to remove 320,000 files?

Fri Sep 25, 2020 4:10 pm

You could use mmv. (apt install mmv)

It has a mode where it can read lines from stdin, so avoids any line length limits.

I wouldn't be mentioning this if I thought this was all a one-off - other people who show you ways to do it with built-in tools would be closer to the mark - but mmv is a very useful tool all around and I consider it an essential install on any system I work on.
GitD's list of things that are not ready for prime time:
1) IPv6
2) 64 bit OSes
3) USB 3
4) Bluetooth

Loves Linux; loves to dance.

PhilE
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 3455
Joined: Mon Sep 29, 2014 1:07 pm
Location: Cambridge

Re: how to remove 320,000 files?

Fri Sep 25, 2020 4:15 pm

You can do something similar with ls in a regular shell script:

Code: Select all

ls -1f <the-directory> | while read -r file; do
	# insert logic using $file here, e.g.
	mv $file <destination-folder>
done

GlowInTheDark
Posts: 1067
Joined: Sat Nov 09, 2019 12:14 pm

Re: how to remove 320,000 files?

Fri Sep 25, 2020 4:33 pm

Do you really want to be forking and execing 320,000 times?

Actually, xargs is designed to address just this sort of problem. It will break it down into manageable chunks, so you're fork/exec'ing as little as possible.

Still, I strongly recommend mmv.
GitD's list of things that are not ready for prime time:
1) IPv6
2) 64 bit OSes
3) USB 3
4) Bluetooth

Loves Linux; loves to dance.

PhilE
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 3455
Joined: Mon Sep 29, 2014 1:07 pm
Location: Cambridge

Re: how to remove 320,000 files?

Fri Sep 25, 2020 4:40 pm

That's the Unix way.

onebitcpu
Posts: 41
Joined: Sun Feb 09, 2020 6:19 pm

Re: how to remove 320,000 files?

Fri Sep 25, 2020 5:55 pm

You can also modify the find command to find older files only:
find . -name "*jpg" -mtime +30 -exec mv {} /path/to/destination/ \;
This will find all .jpg files older than 30 days, and move them.
Real life needs save points

GlowInTheDark
Posts: 1067
Joined: Sat Nov 09, 2019 12:14 pm

Re: how to remove 320,000 files?

Fri Sep 25, 2020 5:57 pm

onebitcpu wrote:
Fri Sep 25, 2020 5:55 pm
You can also modify the find command to find older files only:
find . -name "*jpg" -mtime +30 -exec mv {} /path/to/destination/ \;
This will find all .jpg files older than 30 days, and move them.
That still does way too many fork/exec's.

If you are going to do it using only conventional tools, you just have to use xargs.
GitD's list of things that are not ready for prime time:
1) IPv6
2) 64 bit OSes
3) USB 3
4) Bluetooth

Loves Linux; loves to dance.

User avatar
jahboater
Posts: 6278
Joined: Wed Feb 04, 2015 6:38 pm
Location: Wonderful West Dorset

Re: how to remove 320,000 files?

Fri Sep 25, 2020 7:03 pm

GlowInTheDark wrote:
Fri Sep 25, 2020 5:57 pm
That still does way too many fork/exec's.

If you are going to do it using only conventional tools, you just have to use xargs.
Yes, probably.

For interest, fork/exec are very fast and comparatively light weight (compared to CreateProcess() say in Windows).
Before threads were invented, UNIX used fork() for lightweight multitasking all the time.
Also mv is both tiny and memory resident!
Pi4 8GB running PIOS64 Lite

cleverca22
Posts: 1877
Joined: Sat Aug 18, 2012 2:33 pm

Re: how to remove 320,000 files?

Fri Sep 25, 2020 7:59 pm

this sounds like the exact job xargs was made for
GlowInTheDark wrote:
Fri Sep 25, 2020 4:33 pm
Do you really want to be forking and execing 320,000 times?

Actually, xargs is designed to address just this sort of problem. It will break it down into manageable chunks, so you're fork/exec'ing as little as possible.

Still, I strongly recommend mmv.
thats exactly the problem xargs solves over the plain find shown above

Code: Select all

find /source -type f -name '*.jpg' -print0 | xargs -0 mv -t /destination
the -print0 and -0 say to pass filenames between the 2 with \0 seperators, so newlines and spaces in your filenames (if you where truely evil) still get handled properly
and `mv -t /destination` lets you put the destination first instead of last
xargs will then read names from stdin, and when it reaches the max arg length, it will fork out a process to deal with a chunk of them all at once

PhilE
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 3455
Joined: Mon Sep 29, 2014 1:07 pm
Location: Cambridge

Re: how to remove 320,000 files?

Fri Sep 25, 2020 8:10 pm

Which is a neat trick if you're going to use it a lot, but possibly unnecessary for a one-off invocation like this. I'd expect the I/O activity involved with the move to dwarf the forking overhead, but I could be wrong.

HvdW
Posts: 163
Joined: Tue Jun 17, 2014 12:41 pm

Re: how to remove 320,000 files?

Fri Sep 25, 2020 10:18 pm

Is it that important to keep your files?
Just do a once rm -r /your/directory and create a script that handles everything properly in the future.

I don't mean this cynical, I've lost a lot of files and later on I hardly missed the missing files.
Who knows knows
Who doesn't doesn't

GlowInTheDark
Posts: 1067
Joined: Sat Nov 09, 2019 12:14 pm

Re: how to remove 320,000 files?

Fri Sep 25, 2020 10:38 pm

First of all, let me point out that the Subject: of this thread is incorrect. OP is not trying to remove 320K files, he is trying to move them. And, no, "remove" does not mean "move again".

That said, I just looked more closely at OP's actual line:

mv /home/pi/mnt/CamFiles/FrontOfHouse/*.jpg /home/pi/mnt/CamFiles/FrontOfHouse/jpgFiles/

Let's make a simplifying assumption that *.jpg is all/every file in the directory. See below for further commentary on this assumption.

So, let's just do:

cd ~pi/mnt/CamFiles
mkdir tmp
# This next line puts FrontOfHouse to tmp/FrontOfHouse
mv FrontOfHouse tmp
# Next, make a new FrontOfHouse directory
mkdir FrontOfHouse
# Move the original FrontOfHouse to its new home
mv tmp/FrontOfHouse FrontOfHouse/jpgFiles
rmdir tmp

I think we're done. Notice that all of these moves are just moving directories around, so they will be very quick. BTW, if there are non-JPG files that end up in the new jpgFiles dir (i.e., exceptions to the assumption made above), they are probably few, and can be dealt with individually, after the main moving is done.

Note: I did something like this recently. I had a directory structure that was a little bit "off" and needed to be readjusted. Steps similar to the above were performed and everything was then as it should be.
GitD's list of things that are not ready for prime time:
1) IPv6
2) 64 bit OSes
3) USB 3
4) Bluetooth

Loves Linux; loves to dance.

User avatar
jojopi
Posts: 3353
Joined: Tue Oct 11, 2011 8:38 pm

Re: how to remove 320,000 files?

Fri Sep 25, 2020 10:58 pm

doubleudee1 wrote:
Fri Sep 25, 2020 3:12 pm
line 6: /bin/mv: Argument list too long
In modern Linux, the argument list (and environment) can be one quarter of the current stack limit. Distros typically have no hard stack limit, but a soft limit of around 8MiB to catch infinite recursion.

You can effectively avoid the E2BIG error by first running:

Code: Select all

ulimit -s unlimited
Most other POSIX systems do still have a fixed argument limit, and need workarounds.

cleverca22
Posts: 1877
Joined: Sat Aug 18, 2012 2:33 pm

Re: how to remove 320,000 files?

Fri Sep 25, 2020 11:51 pm

jojopi wrote:
Fri Sep 25, 2020 10:58 pm
doubleudee1 wrote:
Fri Sep 25, 2020 3:12 pm
line 6: /bin/mv: Argument list too long
In modern Linux, the argument list (and environment) can be one quarter of the current stack limit. Distros typically have no hard stack limit, but a soft limit of around 8MiB to catch infinite recursion.

You can effectively avoid the E2BIG error by first running:

Code: Select all

ulimit -s unlimited
Most other POSIX systems do still have a fixed argument limit, and need workarounds.

Code: Select all

[root@amd-nixos:~]# xargs --show-limits
Your environment variables take up 5682 bytes
POSIX upper limit on argument length (this system): 2089422
POSIX smallest allowable upper limit on argument length (all systems): 4096
Maximum length of command we could actually use: 2083740
Size of command buffer we are actually using: 131072
Maximum parallelism (--max-procs must be no greater): 2147483647
xargs can also fully explain what the current limits are as well

Code: Select all

[root@amd-nixos:~]# ulimit -s unlimited

[root@amd-nixos:~]# xargs --show-limits
Your environment variables take up 5682 bytes
POSIX upper limit on argument length (this system): 4611686018427380173
POSIX smallest allowable upper limit on argument length (all systems): 4096
Maximum length of command we could actually use: 4611686018427374491
Size of command buffer we are actually using: 131072
Maximum parallelism (--max-procs must be no greater): 2147483647
but i wasnt aware you could just raise the limit so easily, thats nice to know

doubleudee1
Posts: 164
Joined: Fri Nov 30, 2018 2:39 pm
Location: Wales

Re: how to remove 320,000 files?

Tue Sep 29, 2020 10:20 am

Thanks all for your code and ideas, I've picked a bit more knowledge up (again), I decided to use the code by GlowInTheDark for now as it seemed to be the quickest and eaiest fix for me

Code: Select all

cd ~pi/mnt/CamFiles
mkdir tmp
# This next line puts FrontOfHouse to tmp/FrontOfHouse
mv FrontOfHouse tmp
# Next, make a new FrontOfHouse directory
mkdir FrontOfHouse
# Move the original FrontOfHouse to its new home
mv tmp/FrontOfHouse FrontOfHouse/jpgFiles
rmdir tmp
I can now fix my broken code (which I probably did by messing with it a few months ago when I tried moving them not only into months/years, but days/months/years.
Which now gives me breathing space to try and get them sorted properly, so I'll revert to the old cod (which works OK) just moving them from the directory's jpgFiles and mp4Files, which are two different scripts that run from crontab daily.

Thanks again
WD
Make the most of your family and friends , your children grow up too quickly and you don't notice yourself ageing, friends dissapearing, moving on. You make plans for your future, but they can all be smashed in an instant, live life and enjoy. :)

GlowInTheDark
Posts: 1067
Joined: Sat Nov 09, 2019 12:14 pm

Re: how to remove 320,000 files?

Wed Sep 30, 2020 3:38 pm

I know that at this point this is a "problem solved", but a few additional thoughts have occurred to me.

First, I am glad that you got it solved using my "moving directories around instead of moving files around" method. That is good.

Second, it just occurred to me now that the REAL value and strength of mmv is that mmv does the wildcarding internally. It does not rely on the shell, like most Unix utilities do. This frees you up from any worrying about shell line length limitations. When working with mmv, you pass it in quoted strings containing wildcards and mmv handles it internally. So, you would do:

$ mmv -v '*.jpg' someDirectory

So, no worries about exceeding any line length or "exec" arg list length limits.

I had mentioned earlier the idea of using the "read stdin" mode of mmv, but then I realized that you don't need to go that route.
GitD's list of things that are not ready for prime time:
1) IPv6
2) 64 bit OSes
3) USB 3
4) Bluetooth

Loves Linux; loves to dance.

Return to “Troubleshooting”