doubleudee1
Posts: 158
Joined: Fri Nov 30, 2018 2:39 pm
Location: Wales

How will I Know [Are the directories created on my HDD and all files being put onto it]

Fri Oct 04, 2019 10:51 am

Well, this is not Whitney, but..........

I have setup my new HDD as below

Code: Select all

[email protected]:~ $ lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda           8:0    0  3.7T  0 disk
└─sda1        8:1    0  3.7T  0 part /pi/mnt/CamFiles
mmcblk0     179:0    0 29.9G  0 disk
├─mmcblk0p1 179:1    0  1.3G  0 part
├─mmcblk0p2 179:2    0  512B  0 part
├─mmcblk0p5 179:5    0   32M  0 part
├─mmcblk0p6 179:6    0  256M  0 part /boot
└─mmcblk0p7 179:7    0 28.3G  0 part /
[email protected]:~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        28G   26G  1.3G  96% /
devtmpfs        459M     0  459M   0% /dev
tmpfs           464M     0  464M   0% /dev/shm
tmpfs           464M  7.4M  456M   2% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           464M     0  464M   0% /sys/fs/cgroup
/dev/mmcblk0p6  253M   40M  213M  16% /boot
/dev/sda1       3.7T  181M  3.7T   1% /pi/mnt/CamFiles
tmpfs            93M     0   93M   0% /run/user/1000
my fstab is also below

Now I have executable .sh files (run via crontab) which monitor the directory '/pi/mnt/CamFiles' and they will create directories inside that to suit the file dates and then move the files from the base directory '/pi/mnt/CamFiles' into the new directories.

My question is:-
Are the directories created on my HDD and all files being put onto it, or are the files being moved onto my pi;s sd card and filling that up? as I seem to be having less and less space on my sd card of my pi?

I've linked from my windows 10 pc to the pi (see pics below),
am I actuallly looking at my HDD and the created structures on it? because the space on my pi's sd card seems to be decreasing
Image
Image

Thanks
Attachments
mnt Structure..JPG
mnt Structure..JPG (44.76 KiB) Viewed 781 times
CamFiles Proprties.JPG
CamFiles Proprties.JPG (33.15 KiB) Viewed 781 times
fstab.JPG
fstab.JPG (29.73 KiB) Viewed 781 times
Make the most of your family and friends , your children grow up too quickly and you don't notice yourself ageing, friends dissapearing, moving on. You make plans for your future, but they can all be smashed in an instant, live life and enjoy. :)

doubleudee1
Posts: 158
Joined: Fri Nov 30, 2018 2:39 pm
Location: Wales

Re: How will I Know [Are the directories created on my HDD and all files being put onto it,]

Fri Oct 04, 2019 11:23 am

I thought I'd created a NAS for extra space to use specifically for these files to go on from my CCTV cameras, they all FTP to 'pi/mnt/CamFiles' and I am definately having files onto my pi (or is it my NAS).
Have I missed some stage out?
Make the most of your family and friends , your children grow up too quickly and you don't notice yourself ageing, friends dissapearing, moving on. You make plans for your future, but they can all be smashed in an instant, live life and enjoy. :)

doubleudee1
Posts: 158
Joined: Fri Nov 30, 2018 2:39 pm
Location: Wales

Re: How will I Know [Are the directories created on my HDD and all files being put onto it,]

Fri Oct 04, 2019 12:34 pm

I'm going to let this run it's course, if the pi fills up, then I know I've missed something.
Last 2 times I did it, when I came home from holidays my pi was locked up, I couldn't edit any of the config files, I'd get the message 'unable to edit fstab, or crontab .... no space available' this was happening for whatever file i was trying to edit. So I had to power it down, then switch the powr back on and interupt NOOBS on loading by pressing the 'UP' key, and reload the lot and start from scratch. It's happened 3 times now, so it wasn't 3rd time lucky after all.
Make the most of your family and friends , your children grow up too quickly and you don't notice yourself ageing, friends dissapearing, moving on. You make plans for your future, but they can all be smashed in an instant, live life and enjoy. :)

User avatar
procount
Posts: 1854
Joined: Thu Jun 27, 2013 12:32 pm
Location: UK

Re: How will I Know [Are the directories created on my HDD and all files being put onto it,]

Fri Oct 04, 2019 12:52 pm

If your problem is the SD card filling up and not allowing you to start Raspbian, then you can use NOOBS to do some maintenance work without having to start from scratch.
In NOOBS, press shift to interrupt booting to Raspbian and go to the recovery menu.
Press ctrl-alt-f2 and login to the recovery shell with username=root password=raspberry.
From here, you should be able to mount your raspbian rootfs (probably /dev/mmcblk0p7) and your HDD (probably /dev/sda1), then move any excess photos from your SD card to the HDD.
Delete any photos you have copied to give you enough space to boot Raspbian.
Don't forget to sync to ensure they are all safe, and umount your drives.
Press ctrl-alt-F1 to go back to NOOBS and then boot into Raspbian.
PINN - NOOBS with the extras... https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=142574

doubleudee1
Posts: 158
Joined: Fri Nov 30, 2018 2:39 pm
Location: Wales

Re: How will I Know [Are the directories created on my HDD and all files being put onto it,]

Fri Oct 04, 2019 12:59 pm

procount wrote:
Fri Oct 04, 2019 12:52 pm
If your problem is the SD card filling up and not allowing you to start Raspbian, then you can use NOOBS to do some maintenance work without having to start from scratch.
In NOOBS, press shift to interrupt booting to Raspbian and go to the recovery menu.
Press ctrl-alt-f2 and login to the recovery shell with username=root password=raspberry.
From here, you should be able to mount your raspbian rootfs (probably /dev/mmcblk0p7) and your HDD (probably /dev/sda1), then move any excess photos from your SD card to the HDD.
Delete any photos you have copied to give you enough space to boot Raspbian.
Don't forget to sync to ensure they are all safe, and umount your drives.
Press ctrl-alt-F1 to go back to NOOBS and then boot into Raspbian.
Thanks, wasn't aware I could do that, but I'll be going around in circles doing it again and again if I can't get my incoming files onto my HDD and not fill the pi's sd card up.
Maybe I'm getting to old for this game, I have to write notes on everything as I'm on such high dosages of medication which make me forget so I have to look at my notes and decide which one I wrote was the correct one.
Thanks for your help, I just copied/pasted that onto one of my notes.
Make the most of your family and friends , your children grow up too quickly and you don't notice yourself ageing, friends dissapearing, moving on. You make plans for your future, but they can all be smashed in an instant, live life and enjoy. :)

User avatar
procount
Posts: 1854
Joined: Thu Jun 27, 2013 12:32 pm
Location: UK

Re: How will I Know [Are the directories created on my HDD and all files being put onto it,]

Fri Oct 04, 2019 1:07 pm

Yes, I realise it is only a band aid and doesn't fix the root cause of your problem.

But may be that is related to someone else's post of using a 3TB drive and Dougie's comment that GPT is required but not supported in any OS.
Maybe try repartitioning/formatting your external drive and creating a first partition on it <2TB in size and see if that solves your problem?
What filesystem are you using on it?
PINN - NOOBS with the extras... https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=142574

doubleudee1
Posts: 158
Joined: Fri Nov 30, 2018 2:39 pm
Location: Wales

Re: How will I Know [Are the directories created on my HDD and all files being put onto it,]

Fri Oct 04, 2019 2:38 pm

procount wrote:
Fri Oct 04, 2019 1:07 pm
Yes, I realise it is only a band aid and doesn't fix the root cause of your problem.

But may be that is related to someone else's post of using a 3TB drive and Dougie's comment that GPT is required but not supported in any OS.
Maybe try repartitioning/formatting your external drive and creating a first partition on it <2TB in size and see if that solves your problem?
What filesystem are you using on it?
I used the pi to format it as ntfs as it has ntfs-3g loaded on it, so are you saying perhaps that my old 500GB disk worked (but filled up too fast) but the pi will not handle a 4TB disk and must be partitioned into 2TB slices?

OK found mr Lawson's comments on that thread, looks like a reformat for me and maybe see if I can load each partition seperately onto the pi, I can then direct the heaviest load onto one partition, keeping the other two onto the second.
Make the most of your family and friends , your children grow up too quickly and you don't notice yourself ageing, friends dissapearing, moving on. You make plans for your future, but they can all be smashed in an instant, live life and enjoy. :)

doubleudee1
Posts: 158
Joined: Fri Nov 30, 2018 2:39 pm
Location: Wales

Re: How will I Know [Are the directories created on my HDD and all files being put onto it,]

Fri Oct 04, 2019 3:16 pm

Did another bit of searching and found this:-
https://medium.com/@aallan/adding-an-ex ... 321efce86a
Which makes things rather confusing?
Make the most of your family and friends , your children grow up too quickly and you don't notice yourself ageing, friends dissapearing, moving on. You make plans for your future, but they can all be smashed in an instant, live life and enjoy. :)

User avatar
procount
Posts: 1854
Joined: Thu Jun 27, 2013 12:32 pm
Location: UK

Re: How will I Know [Are the directories created on my HDD and all files being put onto it,]

Fri Oct 04, 2019 4:02 pm

Sorry if I misled. That previous post was purely about booting. Once booted, Linux does support GPT & big disks.
PINN - NOOBS with the extras... https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=142574

doubleudee1
Posts: 158
Joined: Fri Nov 30, 2018 2:39 pm
Location: Wales

Re: How will I Know [Are the directories created on my HDD and all files being put onto it,]

Sat Oct 05, 2019 10:33 am

Sorry for the late reply as I can only sit at my keyboard for short amounts of time as leaning forward gives me great pain and then I have to lay down for a while and take more painkillers (on top of all the other crap I have to take) but no probs, I'm glad for all input, every little bit of information I get helps one way or another. It's all in my notes.
Make the most of your family and friends , your children grow up too quickly and you don't notice yourself ageing, friends dissapearing, moving on. You make plans for your future, but they can all be smashed in an instant, live life and enjoy. :)

fbe
Posts: 540
Joined: Thu Aug 17, 2017 9:08 pm

Re: How will I Know [Are the directories created on my HDD and all files being put onto it,]

Sat Oct 05, 2019 10:04 pm

You need to investigate, what fills your root partition. First unmount your HD-drive to see whether there are files below /pi/mnt/CamFiles "hidden" by mounting your HD drive to that folder.

Next start investigating the disk usage with

Code: Select all

sudo du / -d 1 --exclude='/proc'
to figure out what actually fills your SD card. Most usage should be reported for "/" and "/usr".

doubleudee1
Posts: 158
Joined: Fri Nov 30, 2018 2:39 pm
Location: Wales

Re: How will I Know [Are the directories created on my HDD and all files being put onto it,]

Sun Oct 06, 2019 9:36 am

doubleudee1 wrote:
Fri Oct 04, 2019 11:23 am
I thought I'd created a NAS for extra space to use specifically for these files to go on from my CCTV cameras, they all FTP to 'pi/mnt/CamFiles' and I am definately having files onto my pi (or is it my NAS).
Have I missed some stage out?
####################################################
OK, my pi is full!
I just tried to create a new directory in 'pi/mnt/CamFiles' :-

Code: Select all

[email protected]:~ $ ls
'~CamFiles'   Downloads   log2.txt   MagPi   Pictures           Templates
 Desktop      ExecFiles   log3.txt   mnt     Public             Videos
 Documents    log1.txt    log4.txt   Music   symlinks_all.txt
[email protected]:~ $ cd mnt
[email protected]:~/mnt $ ls
CamFiles
[email protected]:~/mnt $ cd CamFiles
[email protected]:~/mnt/CamFiles $ ls
Dots  FrontOfHouse  RearOfHouse
[email protected]:~/mnt/CamFiles $ mkdir test
mkdir: cannot create directory ‘test’: No space left on device
[email protected]:~/mnt/CamFiles $ 
and I got the message :-

mkdir: cannot create directory ‘test’: No space left on device

Now I'm really baffled, my pi loads the HDD as a 'part' 1.6TB (as above, and all my cameras ftp into the directories within that.
I've connected from Windows 10 to the 'CamFiles' folder and can watch files being created within it, they were creating OK, names, size and every other attribute.
Now I can see lots of 0kb (Zero size) files being produced?

Can anyone help me with this please? It must be my fault, I must have done something wrong or made a mistake somewhere, there is lots of info in this thread, but if you want more, please ask.

Thanks
WD
Make the most of your family and friends , your children grow up too quickly and you don't notice yourself ageing, friends dissapearing, moving on. You make plans for your future, but they can all be smashed in an instant, live life and enjoy. :)

doubleudee1
Posts: 158
Joined: Fri Nov 30, 2018 2:39 pm
Location: Wales

Re: How will I Know [Are the directories created on my HDD and all files being put onto it,]

Sun Oct 06, 2019 9:41 am

Here's what it now looks like:-

Image
Attachments
Empty files.JPG
Empty files.JPG (181.26 KiB) Viewed 552 times
Make the most of your family and friends , your children grow up too quickly and you don't notice yourself ageing, friends dissapearing, moving on. You make plans for your future, but they can all be smashed in an instant, live life and enjoy. :)

doubleudee1
Posts: 158
Joined: Fri Nov 30, 2018 2:39 pm
Location: Wales

Re: How will I Know [Are the directories created on my HDD and all files being put onto it,]

Sun Oct 06, 2019 9:55 am

I can't even update anything:-

Code: Select all

[email protected]:~ $ sudo apt-get update
Get:1 http://archive.raspberrypi.org/debian buster InRelease [25.1 kB]
Err:1 http://archive.raspberrypi.org/debian buster InRelease
  Error writing to output file - write (28: No space left on device) [IP: 2a00:1098:0:80:1000:13:0:5 80]
Get:2 http://raspbian.raspberrypi.org/raspbian buster InRelease [15.0 kB]
Err:2 http://raspbian.raspberrypi.org/raspbian buster InRelease
  Error writing to output file - write (28: No space left on device) [IP: 2a00:1098:0:80:1000:75:0:3 80]
Reading package lists... Done
W: Failed to fetch http://raspbian.raspberrypi.org/raspbian/dists/buster/InRelease  Error writing to output file - write (28: No space left on device) [IP: 2a00:1098:0:80:1000:75:0:3 80]
W: Failed to fetch http://archive.raspberrypi.org/debian/dists/buster/InRelease  Error writing to output file - write (28: No space left on device) [IP: 2a00:1098:0:80:1000:13:0:5 80]
W: Some index files failed to download. They have been ignored, or old ones used instead.
[email protected]:~ $
Make the most of your family and friends , your children grow up too quickly and you don't notice yourself ageing, friends dissapearing, moving on. You make plans for your future, but they can all be smashed in an instant, live life and enjoy. :)

User avatar
rpdom
Posts: 15610
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: How will I Know [Are the directories created on my HDD and all files being put onto it,]

Sun Oct 06, 2019 12:20 pm

On a command line (terminal) or ssh session on your Pi, try typing

Code: Select all

df -h
to see what and where the disk usage is. You may have filled up your root (/) partition, but still have space for your files in another partition on your HDD.

doubleudee1
Posts: 158
Joined: Fri Nov 30, 2018 2:39 pm
Location: Wales

Re: How will I Know [Are the directories created on my HDD and all files being put onto it,]

Sun Oct 06, 2019 1:02 pm

rpdom wrote:
Sun Oct 06, 2019 12:20 pm
On a command line (terminal) or ssh session on your Pi, try typing

Code: Select all

df -h
to see what and where the disk usage is. You may have filled up your root (/) partition, but still have space for your files in another partition on your HDD.
#########################################################
Thanks for that, results below with other commands I've just run, I can also see disk useage using gparted.
I've run a multitude of commands (see below) checking various things and I now think the problem may lay in the build of the scripts I have which move the files and build new paths/directories where needed.
This is an example of one of them:- (not really exotic coding I'm afraid)

Code: Select all


#!/bin/bash
#source $HOME/.bash_profile

# create an array with all the files/dir inside ~home/pi/CamFiles/FrontOfHouse/
####!/bin/bash

arr=(/home/pi/CamFiles/FrontOfHouse/*.jpg)  #the camera for this FTP's the files into this Directory

# iterate through array using a counter
for ((i=0; i<${#arr[@]}; i++)); do
################### SET UP VARIABLES FOR CODE ########################### was CCTV_In
   j=${arr[$i]:33:6}"_jpg"                   ## 201903_jpg
echo "j=  " ${j}
   k=${arr[$i]:0:29}"/jpgFiles/"              ## /home/pi/CamFiles/FrontOfHouse/jpgFiles/
echo "k= " ${k}
   h=${arr[$i]:0:51}                     ## /home/pi/CamFiles/FrontOfHouse/ARC20190315080521.jpg
echo "h= " ${l}
   m=${arr[$i]:0:51}                   ##/home/pi/CamFiles/FrontOfHouse/ARC20190315080521.jpg
echo "m- " ${m}
   n=${k}${j}                  ## /home/pi/CamFiles/FrontOfHouse/201903
echo "n= "${n}
  o=${arr[$i]:33:6} ## 201903
echo "o=  " ${o}
oo="${k}${o}""_jpg"
echo "oo=  " ${oo}
   l="${k}${j}"
echo "l= " ${l}

###################  CREATE ANY NEW DIRECTORIES THAT NEED CREATING ##############
 mkdir -p "${oo}" ### this creates new directories in location of this exec file
echo " DIR " "${oo}" " Built"
################# MOVE ALL FILES TO RELEVANT DIRECTORIES NEXT ##################
### mv pathtofileA pathtofileB ###
aa="201901_jpg"
bb="201902_jpg"
cc="201903_jpg"
dd="201904_jpg"
ee="201905_jpg"
ff="201906_jpg"
gg="201907_jpg"
hh="201908_jpg"
ii="201909_jpg"
jj="201910_jpg"
kk="201911_jpg"
ll="201912_jpg"




if [ ${j} = ${aa} ]; then
####echo "moving " "${m}" " to " "${l}"
     mv "${m}" "${l}"
fi
 if [ ${j} = ${bb} ]; then
####echo "moving " "${m}" " to " "${l}"
     mv "${m}" "${l}"
fi
 if [ ${j} = ${cc} ]; then
fi
 if [ ${j} = ${cc} ]; then
####echo "moving " "${m}" " to " "${l}"
    mv "${m}" "${l}"
fi
 if [ ${j} = ${dd} ]; then
####echo "moving " "${m}" " to " "${l}"
    mv "${m}" "${l}"
 fi
if [ ${j} = ${ee} ]; then
####echo "moving " "${m}" " to " "${l}"
     mv "${m}" "${l}"
fi
 if [ ${j} = ${ff} ]; then
####echo "moving " "${m}" " to " "${l}"
     mv "${m}" "${l}"
fi
 if [ ${j} = ${gg} ]; then
####echo "moving " "${m}" " to " "${l}"
    mv "${m}" "${l}"
fi
 if [ ${j} = ${hh} ]; then
####echo "moving " "${m}" " to " "${l}"
    mv "${m}" "${l}"
fi
 if [ ${j} = ${ii} ]; then
####echo "moving " "${m}" " to " "${l}"
     mv "${m}" "${l}"
fi
 if [ ${j} = ${jj} ]; then
####echo "moving " "${m}" " to " "${l}"
    mv "${m}" "${l}"
 fi

done
Should I be deleting the files or doing something else after moving them?

and here are the commands I spoke of:-

Code: Select all

[email protected]:~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        28G   27G     0 100% /
devtmpfs        459M     0  459M   0% /dev
tmpfs           464M     0  464M   0% /dev/shm
tmpfs           464M   50M  414M  11% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           464M     0  464M   0% /sys/fs/cgroup
/dev/mmcblk0p6  253M   40M  213M  16% /boot
/dev/sda1       3.7T  181M  3.7T   1% /pi/mnt/CamFiles
tmpfs            93M     0   93M   0% /run/user/1000

===================================================
[email protected]:~/mnt/CamFiles $ ls -al
total 128
drwxrwxrwx+ 5 pi pi  4096 Oct  3 10:32 .
drwxrwxrwx  3 pi pi  4096 Oct  3 16:23 ..
drwxrwxrwx  5 pi pi  4096 Oct  5 13:50 Dots
drwxrwxrwx  4 pi pi 69632 Oct  6 11:09 FrontOfHouse
drwxrwxrwx  4 pi pi 36864 Oct  6 11:09 RearOfHouse
[email protected]:~/mnt/CamFiles $ cd /home/pi
[email protected]:~ $ ls
'~CamFiles'   Downloads   log2.txt   MagPi   Pictures           Templates
 Desktop      ExecFiles   log3.txt   mnt     Public             Videos
 Documents    log1.txt    log4.txt   Music   symlinks_all.txt
[email protected]:~ $ ln -s /mnt/CamFiles ~/CamFiles2
[email protected]:~ $ ls
'~CamFiles'   Documents   log1.txt   log4.txt   Music      symlinks_all.txt
 CamFiles2    Downloads   log2.txt   MagPi      Pictures   Templates
 Desktop      ExecFiles   log3.txt   mnt        Public     Videos
[email protected]:~ $ ls CamFiles2
[email protected]:~ $ ls ~CamFiles
Dots  FrontOfHouse  RearOfHouse
[email protected]:~ $ unlink CamFiles2
[email protected]:~ $ ls
'~CamFiles'   Downloads   log2.txt   MagPi   Pictures           Templates
 Desktop      ExecFiles   log3.txt   mnt     Public             Videos
 Documents    log1.txt    log4.txt   Music   symlinks_all.txt
[email protected]:~ $ ls -l /dev/disk/by-uuid/
total 0
lrwxrwxrwx 1 root root 15 Oct  5 13:26 21b67246-b867-4ade-8da9-503424962857 -> ../../mmcblk0p7
lrwxrwxrwx 1 root root 10 Oct  5 13:26 3A9FB160477B9A0A -> ../../sda1
lrwxrwxrwx 1 root root 15 Oct  5 13:26 617B-98D3 -> ../../mmcblk0p6
lrwxrwxrwx 1 root root 15 Oct  5 13:26 7ae383d9-d8c7-4cc1-844a-930d11d80aab -> ../../mmcblk0p5
lrwxrwxrwx 1 root root 15 Oct  5 13:26 90E6-4AE3 -> ../../mmcblk0p1
Make the most of your family and friends , your children grow up too quickly and you don't notice yourself ageing, friends dissapearing, moving on. You make plans for your future, but they can all be smashed in an instant, live life and enjoy. :)

User avatar
rpdom
Posts: 15610
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: How will I Know [Are the directories created on my HDD and all files being put onto it,]

Sun Oct 06, 2019 1:45 pm

doubleudee1 wrote:
Sun Oct 06, 2019 1:02 pm

Code: Select all

[email protected]:~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        28G   27G     0 100% /
devtmpfs        459M     0  459M   0% /dev
tmpfs           464M     0  464M   0% /dev/shm
tmpfs           464M   50M  414M  11% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           464M     0  464M   0% /sys/fs/cgroup
/dev/mmcblk0p6  253M   40M  213M  16% /boot
/dev/sda1       3.7T  181M  3.7T   1% /pi/mnt/CamFiles
tmpfs            93M     0   93M   0% /run/user/1000
Ok, from that we can see that your root (/) file system is 100% full and your CamFiles area is only 1% full. So it isn't the CamFiles that are causing the problem - unless they are being stored in the wrong place...
Having said that, I can see references to /home/pi/CamFiles in your script, but not /pi/mnt/CamFiles.

Also, you list ~/mnt/CamFiles which would be /home/pi/mnt/CamFiles.

Can you clarify where your disk is mounted and what you refer to it as, including any symlinks you may have created, because this is getting confusing.

doubleudee1
Posts: 158
Joined: Fri Nov 30, 2018 2:39 pm
Location: Wales

Re: How will I Know [Are the directories created on my HDD and all files being put onto it,]

Sun Oct 06, 2019 3:07 pm

Looking at gparted I can see the following info:-
Partition = dev/sda1
Name = CamFiles
File System = ntfs
MountPoint = /pi/mnt/CamFiles
Size = 3.64 TiB
Used = 180.88 MiB
Flags = msfdata

I think I have made a mistake in my Exec Files somewhere, here is one taken Directly from my DIR 'ExecFiles' which are the ones used.

Code: Select all

#!/bin/bash
#source $HOME/.bash_profile
## this file is being ran from the directory on the HDD '/CamFiles/FrontOfHouse/'

# create an array with all the files/dir inside ~/CamFiles/FrontOfHouse/
####!/bin/bash
#OLD arr=(/home/pi/CamFiles/FrontOfHouse/*.jpg)
arr=(/home/pi/mnt/CamFiles/FrontOfHouse/*.jpg)

# iterate through array using a counter
for ((i=0; i<${#arr[@]}; i++)); do
################### SET UP VARIABLES FOR CODE ########################### was CCTV_In
   j=${arr[$i]:38:6}"_jpg" ## 201903_jpg
echo "j=  " ${j}
   k=${arr[$i]:0:34}"/jpgFiles/" ## /home/pi/mnt/CamFiles/FrontOfHouse/jpgFiles/
echo "k= " ${k}
   h=${arr[$i]:0:56} ## /home/pi/mnt/CamFiles/FrontOfHouse/ARC20190315080521.jpg
echo "h= " ${l}
   m=${arr[$i]:0:56} ##/home/pi/mnt/CamFiles/FrontOfHouse/ARC20190315080521.jpg
echo "m- " ${m}
   n=${k}${j} ## /home/pi/mnt/CamFiles/FrontOfHouse/201903
echo "n= "${n}
  o=${arr[$i]:38:6} ## 201903
echo "o=  " ${o}
oo="${k}${o}""_jpg"
echo "oo=  " ${oo}
   l="${k}${j}"
echo "l= " ${l}


###################  CREATE ANY NEW DIRECTORIES THAT NEED CREATING ##############
 mkdir -p "${oo}" ### this creates new directories in location of this exec file
echo " DIR " "${oo}" " Built"
################# MOVE ALL FILES TO RELEVANT DIRECTORIES NEXT ##################
### mv pathtofileA pathtofileB ###
aa="201901_jpg"
bb="201902_jpg"
cc="201903_jpg"
dd="201904_jpg"
ee="201905_jpg"
ff="201906_jpg"
gg="201907_jpg"
hh="201908_jpg"
ii="201909_jpg"
jj="201910_jpg"
kk="201911_jpg"
ll="201912_jpg"

echo  "Starting to move files.........."


if [ ${j} = ${aa} ]; then
 echo "moving " "${m}" " to " "${l}"
     mv "${m}" "${l}"
fi
 if [ ${j} = ${bb} ]; then
echo "moving " "${m}" " to " "${l}"
     mv "${m}" "${l}"
fi
# if [ ${j} = ${cc} ]; then
#fi
 if [ ${j} = ${cc} ]; then
echo "moving " "${m}" " to " "${l}"
     mv "${m}" "${l}"
fi
 if [ ${j} = ${dd} ]; then
echo "moving " "${m}" " to " "${l}"
    mv "${m}" "${l}"
 fi
if [ ${j} = ${ee} ]; then
echo "moving " "${m}" " to " "${l}"
     mv "${m}" "${l}"
fi
 if [ ${j} = ${ff} ]; then
echo "moving " "${m}" " to " "${l}"
     mv "${m}" "${l}"
fi
 if [ ${j} = ${gg} ]; then
echo "moving " "${m}" " to " "${l}"
    mv "${m}" "${l}"
fi
 if [ ${j} = ${hh} ]; then
echo "moving " "${m}" " to " "${l}"
    mv "${m}" "${l}"
fi
 if [ ${j} = ${ii} ]; then
echo "moving " "${m}" " to " "${l}"
     mv "${m}" "${l}"
fi
 if [ ${j} = ${jj} ]; then
echo "moving " "${m}" " to " "${l}"
    mv "${m}" "${l}"
 fi
echo  "All files moved."
done

###edited as comments were wrong
Make the most of your family and friends , your children grow up too quickly and you don't notice yourself ageing, friends dissapearing, moving on. You make plans for your future, but they can all be smashed in an instant, live life and enjoy. :)

doubleudee1
Posts: 158
Joined: Fri Nov 30, 2018 2:39 pm
Location: Wales

Re: How will I Know [Are the directories created on my HDD and all files being put onto it,]

Mon Oct 07, 2019 10:52 am

Apologies to all, I think I've got it all wrong!

I thought that by connecting my external HDD (as per I have done with mountpoint /pi/mnt/CamFiles), I could then FTP files straight from my CCTV Cameras into that directory, do some changes to them, and they would be automatically moved onto my HDD. I created a symlink to be able to connect to (ln -s mnt/Camfiles ~CamFiles) from my login position, but it seems I somehow cannot move the files onto my HDD.

The cameras will FTP the files into that directory and I have done lots of searching on 'automatically moving files from a raspberry pi onto a USD HDD' and all I've found is information on creating mountpoints (which I've done in fstab) and dragging/dropping any files into that directory using either a mac or windows connection (mine is windows 10) through windows explorer.
All it seems I am doing is creating new directories based upon the file dates coming in, and then moving the files into them.
My pi is now full and if I run this command "sudo du -xh / | grep -P "G\t"" (found through my seraches) to find out by what, I get:-

Code: Select all

sudo du -xh / | grep -P "G\t"
4.2G    /home/pi/mnt/CamFiles/RearOfHouse/mp4Files/201910_mp4
4.2G    /home/pi/mnt/CamFiles/RearOfHouse/mp4Files
4.5G    /home/pi/mnt/CamFiles/RearOfHouse
2.2G    /home/pi/mnt/CamFiles/Dots/2019/10/01
3.3G    /home/pi/mnt/CamFiles/Dots/2019/10
3.3G    /home/pi/mnt/CamFiles/Dots/2019
3.3G    /home/pi/mnt/CamFiles/Dots
13G     /home/pi/mnt/CamFiles/FrontOfHouse/mp4Files/201910_mp4
13G     /home/pi/mnt/CamFiles/FrontOfHouse/mp4Files
14G     /home/pi/mnt/CamFiles/FrontOfHouse
21G     /home/pi/mnt/CamFiles
21G     /home/pi/mnt
21G     /home/pi
21G     /home
2.1G    /usr/lib
1.1G    /usr/share
3.3G    /usr
25G     /
So it seems to me the files are just building up inside 'pi/mnt/CamFiles' which my script (below) works on the files.

Code: Select all

#!/bin/bash
#source $HOME/.bash_profile
## this file is being ran from the directory on the HDD '/CamFiles/FrontOfHouse/'

# create an array with all the files/dir inside ~/CamFiles/FrontOfHouse/
####!/bin/bash
arr=(/home/pi/mnt/CamFiles/FrontOfHouse/*.jpg)

# iterate through array using a counter
for ((i=0; i<${#arr[@]}; i++)); do
################### SET UP VARIABLES FOR CODE ########################### 
   j=${arr[$i]:38:6}"_jpg" ## 201903_jpg
echo "j=  " ${j}
   k=${arr[$i]:0:34}"/jpgFiles/" ## /home/pi/mnt/CamFiles/FrontOfHouse/jpgFiles/
echo "k= " ${k}
   h=${arr[$i]:0:56} ## /home/pi/mnt/CamFiles/FrontOfHouse/ARC20190315080521.jpg
echo "h= " ${l}
   m=${arr[$i]:0:56} ##/home/pi/mnt/CamFiles/FrontOfHouse/ARC20190315080521.jpg
echo "m- " ${m}
   n=${k}${j} ## /home/pi/mnt/CamFiles/FrontOfHouse/201903
echo "n= "${n}
  o=${arr[$i]:38:6} ## 201903
echo "o=  " ${o}
oo="${k}${o}""_jpg"
echo "oo=  " ${oo}
   l="${k}${j}"
echo "l= " ${l}


###################  CREATE ANY NEW DIRECTORIES THAT NEED CREATING ##############
 mkdir -p "${oo}" ### this creates new directories in location of this exec file
echo " DIR " "${oo}" " Built"
################# MOVE ALL FILES TO RELEVANT DIRECTORIES NEXT ##################
### mv pathtofileA pathtofileB ###
aa="201901_jpg"
bb="201902_jpg"
cc="201903_jpg"
dd="201904_jpg"
ee="201905_jpg"
ff="201906_jpg"
gg="201907_jpg"
hh="201908_jpg"
ii="201909_jpg"
jj="201910_jpg"
kk="201911_jpg"
ll="201912_jpg"

echo  "Starting to move files.........."


if [ ${j} = ${aa} ]; then
 echo "moving " "${m}" " to " "${l}"
     mv "${m}" "${l}"
fi
 if [ ${j} = ${bb} ]; then
echo "moving " "${m}" " to " "${l}"
     mv "${m}" "${l}"
fi
# if [ ${j} = ${cc} ]; then
#fi
 if [ ${j} = ${cc} ]; then
echo "moving " "${m}" " to " "${l}"
     mv "${m}" "${l}"
fi
 if [ ${j} = ${dd} ]; then
echo "moving " "${m}" " to " "${l}"
    mv "${m}" "${l}"
 fi
if [ ${j} = ${ee} ]; then
echo "moving " "${m}" " to " "${l}"
     mv "${m}" "${l}"
fi
 if [ ${j} = ${ff} ]; then
echo "moving " "${m}" " to " "${l}"
     mv "${m}" "${l}"
fi
 if [ ${j} = ${gg} ]; then
echo "moving " "${m}" " to " "${l}"
    mv "${m}" "${l}"
fi
 if [ ${j} = ${hh} ]; then
echo "moving " "${m}" " to " "${l}"
    mv "${m}" "${l}"
fi
 if [ ${j} = ${ii} ]; then
echo "moving " "${m}" " to " "${l}"
     mv "${m}" "${l}"
fi
 if [ ${j} = ${jj} ]; then
echo "moving " "${m}" " to " "${l}"
    mv "${m}" "${l}"
 fi
echo  "All files moved."
done

I thought I could avoid buying a ready built CCTV monitoring system as this was by far the cheaper option, but it is looking more likely I will have to now.
I am going to give it a little more time searching, but am not holding my breath, as it looks to me the pi cannot automatically move files to my HDD.

Thanks
WD
(I'm going to add 'rm -rf RearOfHose' and all the other subdirectories to the end of my scripts to see what that does)
Make the most of your family and friends , your children grow up too quickly and you don't notice yourself ageing, friends dissapearing, moving on. You make plans for your future, but they can all be smashed in an instant, live life and enjoy. :)

User avatar
rpdom
Posts: 15610
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: How will I Know [Are the directories created on my HDD and all files being put onto it,]

Mon Oct 07, 2019 4:11 pm

I think you do not fully understand file paths and the Linux naming conventions yet.

You seem to be storing the files under /home/pi/mnt/CamFiles, but your storage is mounted in /pi/mnt/CamFiles. Those are two entirely different places. That means you are storing the files on the SD card.
doubleudee1 wrote:
Mon Oct 07, 2019 10:52 am
I created a symlink to be able to connect to (ln -s mnt/Camfiles ~CamFiles) from my login position, but it seems I somehow cannot move the files onto my HDD.
If you created that symlink when logged in as "pi", it will have linked /home/pi/mnt/Camfiles to /home/pi/CamFiles which isn't much use.

It would be easier if you just delete everything under /home/pi/mnt/CamFiles (unless you need to keep them), then change the mount point of the device to /home/pi/mnt/CamFiles. No need for symlinks.

doubleudee1
Posts: 158
Joined: Fri Nov 30, 2018 2:39 pm
Location: Wales

Re: How will I Know [Are the directories created on my HDD and all files being put onto it,]

Tue Oct 08, 2019 5:51 am

rpdom wrote:
Mon Oct 07, 2019 4:11 pm
I think you do not fully understand file paths and the Linux naming conventions yet.

You seem to be storing the files under /home/pi/mnt/CamFiles, but your storage is mounted in /pi/mnt/CamFiles. Those are two entirely different places. That means you are storing the files on the SD card.
doubleudee1 wrote:
Mon Oct 07, 2019 10:52 am
I created a symlink to be able to connect to (ln -s mnt/Camfiles ~CamFiles) from my login position, but it seems I somehow cannot move the files onto my HDD.
If you created that symlink when logged in as "pi", it will have linked /home/pi/mnt/Camfiles to /home/pi/CamFiles which isn't much use.

It would be easier if you just delete everything under /home/pi/mnt/CamFiles (unless you need to keep them), then change the mount point of the device to /home/pi/mnt/CamFiles. No need for symlinks.
Thank you for your help with my problems,
you are correct, I am new to Linux and the pi.
This is my first real project working with it and has been ongoing for months now but when my initial drive (500GB) filled up, I thought that by just unmounting my old one, and formatting a new larger one (4TB) I could just then plug that in and it would all continue as normal.

That was my first BIG mistake

it all fell to bits (I don't think t*ts up is politically correct here) and I had to reload NOOBS (3 times to date) and start again from scratch, but I could not seem to get it going no matter what i tried.
I am gratefull to all who have answered my queries on this, but as you say,
I think you do not fully understand file paths and the Linux naming conventions yet.
is true.
That is why I come to forums and ask for help and guidance from people like yourself.

I have done what you suggested and deleted everything under /home/pi/mnt/CamFiles
and have changed my mountpoint to /home/pi/mnt/CamFiles.

I will have to wait a while to see if it has cured my problem, and will check to see if the files are being loaded onto my HDD.
Many thanks
WD
I will give feedback to you on the state of play.
Make the most of your family and friends , your children grow up too quickly and you don't notice yourself ageing, friends dissapearing, moving on. You make plans for your future, but they can all be smashed in an instant, live life and enjoy. :)

doubleudee1
Posts: 158
Joined: Fri Nov 30, 2018 2:39 pm
Location: Wales

Re: How will I Know [Are the directories created on my HDD and all files being put onto it,]

Tue Oct 08, 2019 2:57 pm

@rpdom

Thanks!
Problem solved, my next task is to move the data into daily directories instead of monthly as there is a lot of files coming from the cameras because they are fired up on detction of movement, so in the garden it picks up birds as well as small animals (which are all welcome), but intruders are not.

We have had a spate of breakins around our area recently with the intruders using a short length of scaffolding pipe to smash the glass in UPVC doors or windows, so nothing is safe anymore, they seem to ignore the house alarms as they are in and out as quickly as possible, sometimes stealing your car keys which they then use for ram-raids elsewhere.
I will probably be adding more hidden cameras inside the house so we will have more chance of seeing their faces as they tend to wear hoodies, and may slip up on hiding their faces once inside, so my system may grow and I may have to add another pi as well.

Thanks again for guiding me to the solution.

WD :D :D
Make the most of your family and friends , your children grow up too quickly and you don't notice yourself ageing, friends dissapearing, moving on. You make plans for your future, but they can all be smashed in an instant, live life and enjoy. :)

Return to “Beginners”