danieltanner
Posts: 3
Joined: Sat Sep 12, 2020 2:52 pm

Media Storage Advice

Sat Sep 12, 2020 3:10 pm

I would like to set up a storage solution to preserve family pictures, videos and files and wanted some advice.

Considerations mandatory:
+ Ability to copy files from mobiles phones and computers and store them centrally somewhere
+ Ability to protect data from loss (i.e., redundancy such as RAID but off site or cloud not needed)
+ Costs need to be kept relatively low say max £200 (without HDDs) but lower costs desirable
Considerations preferences:
+ Performance requirements are minimal, i.e., data transfer rates do not need to be fast
+ Storage connectivity – ideal is access via multiple routes (networked wireless/wired and USB - but plain old USB only would be an option)
+ Power and noise – it is preferred to keep noise to a minimum and to consume power only when needed (these will be infrequently used)

I have a number of Raspberry Pis and have operated one for the last five years as a DLNA server (Serviio) for serving my digital movie collection to my TV. I have a single external HDD attached to this Pi and it provides a network location that is good for my needs (but does not have either the capacity or redundancy that I need). I also have a good Windows gaming PC that may be useful here.

There are a wide range of options out there I am considering the following three:
1) Buy a system designed for this (e.g., a NAS like a Synology)
2) Buy an enclosure for HDDs that has a power supply input and a USB connection only (like https://www.amazon.co.uk/ORICO-4-Bay-Dr ... 061&sr=8-6) and connect this to either: a) my PC or b) a Raspberry Pi and with an appropriate sftp server.

Note: I don't have any major access requirements for the data and I think it would be sufficient to browse and access the data via the PC (either to a storage solution attached to the PC or via network to a headless Pi or NAS).

I am inclined to go for option 2b (as it may prove more satisfying to implement and reduce cost). There is loads of information about this on the internet but I was interested in your opinions. So I am asking here as I expect that many of you have been down this route and it involves a Pi. Any thoughts or advice? Any recommendations on hardware options or HDD (I am assuming standard 3.5” HDDs to provide 4-6Tb of storage)? I only need some pointers to avoid a bad experience and should be ok to figure out the implementation level considerations.

epoch1970
Posts: 5598
Joined: Thu May 05, 2016 9:33 am
Location: Paris, France

Re: Media Storage Advice

Sun Sep 13, 2020 11:34 am

Raid provides (sometimes) data redundancy via hardware redundancy. It is a high maintenance, high cost, high energy (and noise) design that can offer consistent availability and performance.

For an application that borders on archiving, you need none of that. A bunch of independent drives of various makes plus multiple copies will be simpler and safer.

Huge drives take an age to fill up and copy. If such a drive starts failing you’ll have to hope it hangs on for hours or even days in case you need to copy data to a new drive.
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

LTolledo
Posts: 3927
Joined: Sat Mar 17, 2018 7:29 am
Location: Anime Heartland

Re: Media Storage Advice

Sun Sep 13, 2020 12:19 pm

you may use my NAS build as reference.
viewtopic.php?t=244224#p1541272
am using it as tertiary backup, containing playable media, photos, SBC backup images, program scripts, etc

other NAS that I built and currently using as primary media storage is also based on RPi4B
has a 4TB 2.5" HDD as drive
data shared via samba

another NAS is based on RPi2Bv1.2, used as secondary media storage / backup of primary media NAS above.
also has a 4TB 2.5" HDD as drive.
data shared via samba
kept in sync with the primary NAS using "timed mirroring" via cron and rsync

for RAID system, am using a RPi4B-2GB connected to a dual 3.5" HDD drive enclosure with built in HW RAID function.

also have several have non-RPi SBC NAS units

images may be posted upon request.
"Don't come to me with 'issues' for I don't know how to deal with those
Come to me with 'problems' and I'll help you find solutions"

Some people be like:
"Help me! Am drowning! But dont you dare touch me nor come near me!"

danieltanner
Posts: 3
Joined: Sat Sep 12, 2020 2:52 pm

Re: Media Storage Advice

Sun Sep 20, 2020 10:01 am

Both, thanks for the feedback.
My current plan it to try the following:
Get 2 large capacity 3.5” HDDs (Seagate Ironwolf NAS 8Tb - https://www.ebuyer.com/885104-seagate-i ... t8000vn004)
Get 2 SATA to USB 3.0 Adaptors (both externally powered - https://www.amazon.co.uk/Inateck-Conver ... 778&sr=8-6)
Connect PI, and both drives to mains (via appropriate DC PSUs), connect both HDD USBs to PI USB 3.0. Make some rudimentary frame to mount stuff neatly. I have a Pi 3 so will play with this and will consider a Pi 4 upgrade if I feel faster data rates are needed.

Not sure if this will work but the risks are low and costs manageable. There is no cooling but this will be infrequently used for short periods of time. Considering either forming a RAID 1 array (mdadm) or just setting up a periodic sync between drives (e.g., scripting rsync and scheduling - like LTolledo's approach). I have no idea how much control I will have to poweroff/standby the drives (I believe it is inbuilt in the drives) but I am keen to keep noise and power consumption very low as, again, they will neither be frequently nor heavily used.
I will let you know how this progresses.

epoch1970
Posts: 5598
Joined: Thu May 05, 2016 9:33 am
Location: Paris, France

Re: Media Storage Advice

Sun Sep 20, 2020 10:37 am

danieltanner wrote:
Sun Sep 20, 2020 10:01 am
I am keen to keep noise and power consumption very low
And so forget RAID. Won’t do anything you want and will do a lot of thing you don’t want.
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

User avatar
dickon
Posts: 1729
Joined: Sun Dec 09, 2012 3:54 pm
Location: Home, just outside Reading

Re: Media Storage Advice

Sun Sep 20, 2020 6:17 pm

Media storage NAS units don't need to have the discs spinning all the time -- only when recording or playing back -- so a good look at hdparm's manpage is well worth a read. Also, stick the thing somewhere where the noise doesn't matter, and use a second PI to do the actual playback in silence behind the TV.

Heater
Posts: 16586
Joined: Tue Jul 17, 2012 3:02 pm

Re: Media Storage Advice

Sun Sep 20, 2020 7:21 pm

danieltanner wrote:
Sat Sep 12, 2020 3:10 pm
I would like to set up a storage solution to preserve family pictures, videos and files and wanted some advice.
From this I gather the files you want to preserve are valuable and treasured artifacts that you would like to be available to your children, and their children and generations to come.

If so your stated requirements for this storage system are not actually going to achieve what you want.
danieltanner wrote:
Sat Sep 12, 2020 3:10 pm
+ Ability to copy files from mobiles phones and computers and store them centrally somewhere
Straight away there you have a problem. A central system is always subject to failure and total loss of data. Be that through flood, fire, thunder strikes, warfare, whatever happens.
danieltanner wrote:
Sat Sep 12, 2020 3:10 pm
+ Ability to protect data from loss (i.e., redundancy such as RAID but off site or cloud not needed)
RAID is not a backup solution. RAID can possibly make a more reliable storage solution. But when you house burns down, your data is gone. When the RAID controller fails your data can be gone. See fire, flood, thunder strikes, warfare above..

To achieve what you want you exactly need geographically distributed, offsite storage. In the "cloud" or whatever.

To this end the first thing to do is copy all those pictures, videos etc to a bunch of hard drives or perhaps SSDs, USB sticks, whatever, and post them out to all members of the family who might be interested.
Memory in C++ is a leaky abstraction .

dustnbone
Posts: 357
Joined: Tue Nov 05, 2019 2:49 am

Re: Media Storage Advice

Sun Sep 20, 2020 7:51 pm

Agreed. If it's unique irrplaceable data that you value a great deal, offsite backups are a must, ideally more than one. It's nice having a local accessible copy to serve out to your devices, but a RAID only protects you from hardware failure, not human failure or theft/fire/etc. The amount of data I've seen lost to human error far outstrips that lost to disk failure.

If a file accidentally gets deleted from the RAID, or hit by ransomware or something similar, the RAID will dutifully delete it from the array. It doesn't provide any magical ability to recover from this, that's what actual (offline) backups are for.

Return to “General discussion”