Raspberry Pi Archiver

[clix]

Hi there,
I have a Raspberry Pi 2B and as a long term use for it, I thought I could use it to archive many files and folders (~200GB) on a couple of external hard disks. This would take a couple of days on my main machine, but I would like to move this work (which tends to eat the CPU) to the Pi, which is more cost efficient and safer to run 24/7. I installed p7zip-full and began compressing but it always crashes about 30mins-1hr in. Is this due to low/no memory as the Pi has only 1GB, or something else?

If so, is the way around this to create a swap file on the hard disk I am working from and use that, or are there some settings that will optimise 7zip for low memory? Compression ratio should be as high as possible as time is not an object, this can run for weeks and I don't mind. I've looked on the internet and using a Pi as an archiver seems to be unheard of so are there any major downsides/obstacles that I should be aware of (other than slow speed and low memory)? I'm interested in hearing your opinions.

Thanks

[elatllat]

check your logs, or run top to varify ram is the issue.


I have 7z-ed ~4GB on the pi1a and had no issues so you cold do it in chunks but that would likely reduce your compression ratio.

Likely RAM is the issue, using a swap file will work but might slow it to the end of the universe. The reason 7z is so much better than gz or zip is that it seeks io, and uses plenty of ram. You can tell it to compress a stream but then your down to gz performance...just use a recent i3/5/7 with 8+ GB ram and it will be done in 10 minutes.


man 7z


For your options if you want to try anyway.

You could try tar-ing first if it's having trouble with the number of files (vs the size)

[stevech]

My RPi 2, RPi Zero, are on my wired/WiFi network.
I setup SMB file sharing on the RPi's.
Gave them the username/password for their own logins to my small Network Attached Storage (NAS), on my LAN.
Each RPi can then read/write files on the NAS for their login. And vice-versa, a PC on the LAN can access the RPI's storage on the NAS to put/get files. Security for this is simple permissions setup in the NAS admin screens.

So I don't store files on the SD card of the RPi's.
I copy the SD cards for backup, to the NAS. Usually only when I change the RPI's config.

[elatllat]

Do this:

Code: Select all

git clone https://github.com/google/brotli.git && cd brotli &&\
python setup.py build && python setup.py install &&\
cd tools && make && cd ../../ &&\
./brotli/tools/bro -h

you should get this:

Code: Select all

Usage: bro [--force] [--quality n] [--decompress] [--input filename] [--output filename] [--repeat iters] [--verbose] [--window n]

then you can try this:

Code: Select all

./brotli/tools/bro -f -q 11 -i my_big.file -o my_big.file.bro

[cpc464]

Hi OP what you are trying to do sounds a little odd (I can't think why you would want to create huge zip files on external drives), but anyway what error message does p7zip give when it "crashes" ?