JackElliott
Posts: 35
Joined: Wed May 03, 2017 11:57 pm

Setting up a Pi 3 for the long haul?

Fri May 26, 2017 11:51 pm

I'd like to set up a Raspberry Pi at our community radio station to be used to livestream the station's music and shows. We're presently using a funky old donated laptop running Linux, but who knows how long that thing will keep running?

The same software that does our streaming from the laptop, butt ("broadcast using this tool"), works just fine on the RPi. So the project looks feasible.

I'd like to do everything possible to prolong the life of the RPi, and one issue I keep reading about is the relatively limited number of write cycles SD cards have. I could move the /root partition to a flash drive, or I could move it to a self-powered external USB hard drive. I'm not reading anything that strongly suggests that one is more reliable than the other.

If this was your project, if it was something that you hoped would have an uptime measured in years, how would you go about it?

User avatar
Jednorozec
Posts: 809
Joined: Sun Nov 24, 2013 2:17 pm
Location: Deposit, NY

Re: Setting up a Pi 3 for the long haul?

Sat May 27, 2017 2:06 am

Use a USB hard drive.
The most important leg of a three legged stool is the one that's missing.
It's called thinking. Why don't you try it sometime?

ian_g
Posts: 36
Joined: Fri May 26, 2017 1:04 am
Location: East Midlands, UK

Re: Setting up a Pi 3 for the long haul?

Sat May 27, 2017 4:12 am

@Jednorozec LOL god you're good

ian_g
Posts: 36
Joined: Fri May 26, 2017 1:04 am
Location: East Midlands, UK

Re: Setting up a Pi 3 for the long haul?

Sat May 27, 2017 4:17 am

Whilst the rp is a fun bit of hardware you forget the basics. It's os runs on an sd card designed to take a few thousand pictures. There is no long run.

W. H. Heydt
Posts: 10988
Joined: Fri Mar 09, 2012 7:36 pm
Location: Vallejo, CA (US)

Re: Setting up a Pi 3 for the long haul?

Sat May 27, 2017 5:23 am

The Pi3 can be set up to boot directly from a USB device and then it won't require an SD card at all. The instructions to do it are here: https://www.raspberrypi.org/documentati ... des/msd.md

There are two ways I've used that. One is a PiDrive with a Pi3B. Everything is powered from a single PSU. The other is a CM3L in a WD Labs SATA Adapter with a 60GB SSD, but any low profile 2.5" SATA HDD should work. Again, only a single PSU is needed. I like the SATA Adapters enough that I just got two more to use a primary and backup system for running a convention registration system. I don't anticipate needing to change them until and unless there is a future CM that has USB3 and a SATA Adapter to match.

sparkyhall
Posts: 145
Joined: Mon Aug 27, 2012 9:14 am

Re: Setting up a Pi 3 for the long haul?

Sat May 27, 2017 7:39 am

I have often wondered how much truth there is in stories of limited life for SD cards. If you remember the PC SSD drives had the same stories and generally that is now considered nonsense. I currently have the following systems running 24/7 and so far I have not had a single SD card failure:

1) Two Pi based cctv, running since 2013, capturing 4-5 thousand stills per day (150MB written per day), Samsung 8GB Class 6 cards.
2) Pi based weather station, running for just over a year, recording data to Mysql every 5 minutes and running a local Apache web server, Samsung 16GB class 6 card.
3) VPN, time server & UPS monitor, running for about 1.5 years, Samsung 16GB class 6 card.
4) Three Pi's running at work logging results to Mariadb, now been running for 6 months, Sandisk Ultra 16GB cards.

They are all regularly backed up and clearly your use case is somewhat different to these. If it was me I would try it with an SD card and keep a spare card ready just in case.

achrn
Posts: 376
Joined: Wed Feb 13, 2013 1:22 pm

Re: Setting up a Pi 3 for the long haul?

Sat May 27, 2017 7:48 am

JackElliott wrote: If this was your project, if it was something that you hoped would have an uptime measured in years, how would you go about it?
I have three pis with uptimes measured in years (in aggregate) - the one that runs my heating, the one that monitors my fishtank, and the one that that backs up my PCs. They all run off SDcards (though the backup one also has a few TBs of disk for the backup archive). I've had one probable SD failure in four years, though it didn't lose any data. The heating one and the fishtank one write data regularly (the fishtank one writes a timelapse photo once a minute all the time it's not pitch dark, has been doing so for about 2.5 years, and is on its original SD card).

[I say in aggregate, because the heating one has been running more-or-less continuously since late 2012 but gets a reboot about once a year when the boiler gets a service, for example. The longest continuous uptime is probably just under two years, on the fishtank one.]

I'd use a SDcard (a decent brand from a reputable supplier), and add a USB hard drive if I needed more than a few GBs of data. I'd make sure I have backups, but everyone should have backups or everything anyway. I don't use a pi-specific disk - it's just a USB hard disk.

User avatar
rpdom
Posts: 15363
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Setting up a Pi 3 for the long haul?

Sat May 27, 2017 8:24 am

I'm using a Pi2B (upgrading to a 3B soon) with a 2TB Seagate Expansion USB HD and the official PSU for my main server. Been running fine for ages now. The /boot partition is still on an SD card, but that will go once I've set up the 3B.

My other Pis all run of SD cards. The longest uptime was about 500 days until I had a power cut. That is on a Pi 1A in my garage with an ancient SD card running of an old Kindle PSU doing temperature measurements and sending them back to the main server Pi over wifi.

ian_g
Posts: 36
Joined: Fri May 26, 2017 1:04 am
Location: East Midlands, UK

Re: Setting up a Pi 3 for the long haul?

Sat May 27, 2017 12:28 pm

I love being part of an unbiased forum supported by non anecdotal evidence :lol: . USB drive boot still in beta. Just because it hasn't happened to you doesn't mean it won't happen.

First rule of setting up for the long haul is expect failure. Buy two(maybe three or four) of everything

achrn
Posts: 376
Joined: Wed Feb 13, 2013 1:22 pm

Re: Setting up a Pi 3 for the long haul?

Sat May 27, 2017 12:44 pm

ian_g wrote:I love being part of an unbiased forum supported by non anecdotal evidence :lol: . USB drive boot still in beta. Just because it hasn't happened to you doesn't mean it won't happen.
Just because you can make up stuff that could happen, doesn't mean it you need to take measures to avoid it.

I assume from your dismissing of any differing opinion as mere anecdote that you have statistically valid comparisons of the service life of spinning magnetic disks versus SD cards in typical Raspbian use cases. Please share.

JackElliott
Posts: 35
Joined: Wed May 03, 2017 11:57 pm

Re: Setting up a Pi 3 for the long haul?

Sat May 27, 2017 2:47 pm

Please, kind people, don't let this thread get all kind of flamey! I'm hoping for engineering advice so I can build up a dependable little solution.

Here's my thinking so far:

All hardware can (and, eventually will) fail. Electrolytic capacitors have limited lifespan. So keep a spare (or two) of the RPi on the shelf. Power supplies fail, keep spares of those. SD cards have limited writes, but from what I have read, an SD card that is used for boot only isn't going to be worked very hard so it should be okay, right? Keep a clone on the shelf next to the other spares anyway.

For the OS, flash memory vs hard drives.

In flash, chips that use Single-Level Cell technology can handle ten times the number of erase/write cycles than Multi-Level Cell tech (https://media.kingston.com/pdfs/MKF_283 ... ide_EN.pdf, and https://www.mydigitaldiscount.com/every ... flash.html so that's the tech one would want for longer life.

Mechanical things with spinning and flying bits wear out, but today's hard drives run for a long long time. One survey showed Annualized Failure Rates anywhere between 3% to 25% depending on brand (https://arstechnica.com/information-tec ... are-equal/), but the test conditions used were for consumer-level disks run 24/7 in a server application. If the RPi was set up to write tmp data to RAM then the drive could last for decades I reckon.

Even the smallest HDDs that I can find are 80+GB which makes them kind of silly huge for the application, so from a cost standpoint, a 16GB flash drive with wear leveling should last, like, forever.

Keep backups for the OS drive (flash or HDD) on the shelf, too.

Are there any other known reliability issues with the relatively new RPi 3 that a fellow should know about? Our little radio station is going to want something that I can say will run for 10 years before an oil change.

W. H. Heydt
Posts: 10988
Joined: Fri Mar 09, 2012 7:36 pm
Location: Vallejo, CA (US)

Re: Setting up a Pi 3 for the long haul?

Sat May 27, 2017 4:11 pm

ian_g wrote:USB drive boot still in beta. Just because it hasn't happened to you doesn't mean it won't happen.
That is no longer true. Since early April this year the Raspbian downloads have included the boot-from-USB code. You still have to set the OTP bit so that a BCM2837 *will* boot over USB given the opportunity, but that process is well documented. And--FYI--it would work on a Pi2B2 and I've done it on a CM3L (and I'm going to do it to two more CM3Ls soon).

JackElliott
Posts: 35
Joined: Wed May 03, 2017 11:57 pm

Re: Setting up a Pi 3 for the long haul?

Sat May 27, 2017 4:50 pm

W. H. Heydt wrote: That is no longer true. Since early April this year the Raspbian downloads have included the boot-from-USB code. You still have to set the OTP bit so that a BCM2837 *will* boot over USB given the opportunity, but that process is well documented. And--FYI--it would work on a Pi2B2 and I've done it on a CM3L (and I'm going to do it to two more CM3Ls soon).
Thank you. Does someone have a link to an updated how-to for this? I'm finding articles describing the process, but labeled as "beta" or "experimental."

User avatar
DougieLawson
Posts: 36306
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website Twitter

Re: Setting up a Pi 3 for the long haul?

Sat May 27, 2017 4:59 pm

It's not experimental. It's documented at: https://www.raspberrypi.org/documentati ... des/msd.md
It went mainstream with the 4.4.50 kernel and is included in the 4.9.?? kernels.

No special handling needed, just run sudo apt-get update && sudo apt-get -y dist-upgrade. Or start wiith a fresh download of Raspbian 2017-04-10.

Don't use NOOBS.
Note: Having anything humorous in your signature is completely banned on this forum. Wear a tin-foil hat and you'll get a ban.

Any DMs sent on Twitter will be answered next month.

This is a doctor free zone.

W. H. Heydt
Posts: 10988
Joined: Fri Mar 09, 2012 7:36 pm
Location: Vallejo, CA (US)

Re: Setting up a Pi 3 for the long haul?

Sat May 27, 2017 5:23 pm

JackElliott wrote:
W. H. Heydt wrote: That is no longer true. Since early April this year the Raspbian downloads have included the boot-from-USB code. You still have to set the OTP bit so that a BCM2837 *will* boot over USB given the opportunity, but that process is well documented. And--FYI--it would work on a Pi2B2 and I've done it on a CM3L (and I'm going to do it to two more CM3Ls soon).
Thank you. Does someone have a link to an updated how-to for this? I'm finding articles describing the process, but labeled as "beta" or "experimental."
Ask and ye shall receive.... From the documentation on this very site: https://www.raspberrypi.org/documentati ... /README.md

levelcrow
Posts: 47
Joined: Tue Aug 12, 2014 5:04 pm

Re: Setting up a Pi 3 for the long haul?

Sat May 27, 2017 5:48 pm

You see it a lot on the internet that flash memory fails, but after doing a good deal of research I just don't think it's true on a practical level. Maybe it might have been true in the 2000s but now..?

Every flash cell has a listed write limit between 1000-10,000 writes before the insulation layer builds up too many stray electrons embedded within itself to allow for the correct electron tunneling behavior flash memory relies on. I say listed write limit because it tends to be the case that the real limit is about an order of magnitude larger whenever people try to do these flash endurance tests.

Writing 100,000 times to a single block isn't that much, and so flash devices have wear algorithms that try to spread around the writes. Wear algorithms fit into either dynamic or static; the former shifts around blocks in unused space - possibly from a pool of unused blocks specifically for the purpose, the latter shifts around all blocks and manages the data on used blocks accordingly. Someone actually did an academic study on flash drive endurance 7 years ago during which time they couldn't wear out a static wear algorithm drive in the allotted study time. Link is here: https://www.usenix.org/event/fast10/tec ... oboila.pdf.

7 years is a long time in tech. It's enough time for manufacturing processes to go over a generation or more, for previously high end machines to be sold off at discount, for reverse engineering to happen, and for the low end of tech to move up. I'm not sure if the controllers in use for low end brand name flash drives like PNY perform a static wear algorithm, but I highly suspect this is the case based on the technical marketing materials they have for their drives. I've also looked up some of the more common USB controllers found in other cheap drives like those used by Emtec. Their datasheets do say they have static wear capability.

So from this information at least the idea is that USB drives have a theoretical 1000x write capability if you are being extremely conservative. For example, an 8GB drive might have a 8TB write limit. If you were more optimistic and used the 100,000 write limit reported by some people you'd get a fairly high 800TB write limit assuming unattainably perfect wear leveling. We can probably assume SD card controllers have also progressed similarly.

Then why do we hear reports of failing SD cards? And why do USB drives fail sometimes?
I am of the opinion that these failures aren't actually due to the flash most of the time, but rather due to the controller. I've had multiple flash drives fail on me so far, and it seems almost every single one of these was due to some sort of controller problem. Though hardware trickles down the supplier totem pole by quality, software doesn't necessarily do so. A buggy controller that can't properly wear will easily prematurely wear out cells.

Not just on the flash device side, but there could also be some bugs on the kernel side. The Linux kernel and its drivers aren't perfect. The Raspbian kernel for sure isn't perfect, in fact just recently there were a couple of USB bugs fixed.

What does this mean practically? Personally, I think it means flash wearout just isn't something an average user should worry about. For example, I've been using a $4 PNY 8GB Attache USB drive for my root directory for about 2 years now. I've been logging all of the writing out of curiosity using a systemd service and a script that relies on iostat. My total usage at this point is 335.905GB. That means that it would take me 20x as much time to reach 6.7TB - well under the theoretical low 1000 write ideal static wear leveling limit of 8TB. That's 40 years! Plenty of time for a critical controller bug to show up or a critical component such as the timing crystal to break.

In this sense, it just makes most sense to try to minimize chance of controller bugs being in the controller firmware by going with a vendor with a great reputation. I'd recommend the Sandisk Extreme USB 3.0 USB drive, which has been found to have an actual SSD controller inside of it. This USB drive has really good reviews, and by using a USB 3.0 drive you can max out the USB 2.0 connection and get pretty good performance.

JackElliott
Posts: 35
Joined: Wed May 03, 2017 11:57 pm

Re: Setting up a Pi 3 for the long haul?

Sat May 27, 2017 9:57 pm

DougieLawson wrote:It's not experimental. It's documented at: https://www.raspberrypi.org/documentati ... des/msd.md
It went mainstream with the 4.4.50 kernel and is included in the 4.9.?? kernels.
Thanks, DougieLawson and W.H. Heydt. I saw that page before I posted. The second sentence reads in part, "Be warned that this feature is experimental [...]." So I figured it was, um, experimental. But at my age, so is getting out of bed in the morning!

I reckon it works, so I'll give her a try.

levelcrow, many thanks for the in-depth look at this. I'll follow the suggested documentation and use the suggested Sandisk Extreme USB drive. Anything with the word "Extreme" in it has to be good, right? ;)

User avatar
mahjongg
Forum Moderator
Forum Moderator
Posts: 12327
Joined: Sun Mar 11, 2012 12:19 am
Location: South Holland, The Netherlands

Re: Setting up a Pi 3 for the long haul?

Sat May 27, 2017 10:14 pm

the fact is that all SD-cards have some form of wear levelling algorithm, but some SD cards use the cheapest controllers they can, with dumb wear levelling algorithms that can't survive a power failure.

Often people turn off the power immediately after linux has shut down, but they forget that the SD-card doesn't know about imminent shutdowns, and can start a wear levelling process just at the worst time, after a final last batch of writes, when Linux does a cache flush for a shutdown. If you turn off power just after linux has shut down, and don't give the SD-cards the many seconds it needs to do wear levelling you *will* corrupt the filing system.

User avatar
DavidS
Posts: 4334
Joined: Thu Dec 15, 2011 6:39 am
Location: USA
Contact: Website

Re: Setting up a Pi 3 for the long haul?

Sat May 27, 2017 10:57 pm

Two sides to one coin:

SD Cards last longer than people say. I have even completely filled one with different data 30 times per day for over a year continuous without failure. So you may get away with SD Cards. Also I am still using many of the same SD cards that I used in the original RPi B when it was first released.

I have had more HDD failures per level of usage than I have SD cards. Though if you prefer tradition there is nothing stopping you from using an external USB HDD to boot and run from on an RPi 3B.
RPi = The best ARM based RISC OS computer around
More than 95% of posts made from RISC OS on RPi 1B/1B+ computers. Most of the rest from RISC OS on RPi 2B/3B/3B+ computers

JackElliott
Posts: 35
Joined: Wed May 03, 2017 11:57 pm

Re: Setting up a Pi 3 for the long haul?

Sun May 28, 2017 12:13 am

mahjongg wrote:Often people turn off the power immediately after linux has shut down, but they forget that the SD-card doesn't know about imminent shutdowns, and can start a wear levelling process just at the worst time, after a final last batch of writes, when Linux does a cache flush for a shutdown. If you turn off power just after linux has shut down, and don't give the SD-cards the many seconds it needs to do wear levelling you *will* corrupt the filing system.
That warning was worth the price of admission here! What if a fellow wanted his RPi3 project to be "UPS-aware," and in the event of a power failure, to do a graceful shutdown some minutes before the UPS's battery went dead, giving the SD card time to perform wear-leveling?

ericlp
Posts: 8
Joined: Sun Aug 28, 2016 8:44 pm

Re: Setting up a Pi 3 for the long haul?

Sun May 28, 2017 12:52 am

As they say... Flash Memory isn't considered long haul, I disagree. Just look at the failure rates of hard drives and SSD's. Just because you plugged in a USB hard drive doesn't mean jack! My pi3 has been working well with a 32Gig Samsung EVO memory card for well over a year as a webserver, weather server, and doing other crap with cronjobs. No Problems yet.

Next?

Probably the #2 reason why flash would fail is because people buy cheaper memory cards...

Solution: don't cheap out get a samsung/sandisk EVO / PRO cards.

Solution: Get BIG CARDS 32 Gigs at least, that way, cards will have plenty of room for Wear leveling ...Wear Leveling, always writes to new blocks, The controller manages this so that it prolongs the life of memory card. Cards that have plenty of room (free space) is key!

Next?

Buy 3-4 cards, Buy 3-4 Pi's. Get one pie working, backup the flash card to the other pies. If for some reason, the pi dies for (whatever reason) ... you can just drop in a brand new pie in less than 5 seconds and be on your way. Redundancy is priceless!

Next?

Buy a UPS! UPS filters out brown outs, surges, and power outages. Probably the #1 reason why a pie can die, is because the power is not reliable and the memory card gets corrupted. You could go down and buy a cheap $50 dollar APC and keep a few pie's running for a few days if that's all what was plugged into it.

Or if you are really adventurous, you could hack together a lithium ion battery ups for a raspberry pi. Maybe even hook it up to solar!

Also, a RPI 3 get's HOT! If a Rpi2 or a zero can do the job, consider that, as the CPU run a lot cooler. Get a good quality case, and if you don't have very good air circulation or it's going to be stored in a closed box, maybe consider adding a fan. But definitely get heatsinks. I would also scrape off the junk thermal paste and use some quality stuff. Most Rpi3's come with cheap aluminum. Copper ones are only a few bucks more and might be worth it...If you plan to use a pi zero, no need for fan.

Recap?

Get a UPS ... Worth it's weight in gold!
Spend the money for large brand name pro memory cards.
Buy lot's of spare parts and makes backups! (redundancy is key here)
Reduce Heat, good airflow and have a nice enclosure for dust / dirt.


Good Luck!
Last edited by ericlp on Sun May 28, 2017 1:38 am, edited 1 time in total.

User avatar
Strider64
Posts: 13
Joined: Sat May 13, 2017 12:35 pm
Location: Livonia (Detroit), Mi
Contact: Website

Re: Setting up a Pi 3 for the long haul?

Sun May 28, 2017 1:29 am

I personally don't see what the big deal with sd card failures? The simple solution would be to make a clone of the sd card to have a backup just-in-case of a sd failure. Just my .05 cents (would be .02 cents, but because of inflation). :D :D :D
"Insanity Is Doing the Same Thing Over and Over Again and Expecting Different Results" ~ Albert Einstein

JackElliott
Posts: 35
Joined: Wed May 03, 2017 11:57 pm

Re: Setting up a Pi 3 for the long haul?

Sun May 28, 2017 1:36 am

All good advice, but the temp sensor on my Pi 3's shows typical temp around 40 degrees C. That's not anywhere near dangerous for chip. Still and all, heat is the enemy of electronics.

JackElliott
Posts: 35
Joined: Wed May 03, 2017 11:57 pm

Re: Setting up a Pi 3 for the long haul?

Sun May 28, 2017 1:44 am

Strider64 wrote:I personally don't see what the big deal with sd card failures? The simple solution would be to make a clone of the sd card to have a backup just-in-case of a sd failure. Just my .05 cents (would be .02 cents, but because of inflation). :D :D :D
Respect. But in this case the device is supposed to run unattended for years, if possible, to stream our community radio station's "Listen Live" programming. Which doesn't mean that having clones of cards/flash drives & copies of all all hardware isn't a smart thing to do.

Guys, thanks a lot for the tips. This is a fun project.
Last edited by JackElliott on Sun May 28, 2017 1:57 am, edited 1 time in total.

ericlp
Posts: 8
Joined: Sun Aug 28, 2016 8:44 pm

Re: Setting up a Pi 3 for the long haul?

Sun May 28, 2017 1:45 am

JackElliott wrote:All good advice, but the temp sensor on my Pi 3's shows typical temp around 40 degrees C. That's not anywhere near dangerous for chip. Still and all, heat is the enemy of electronics.
Here in Hawaii, ambient temps with no A/C, 82-84 F, my Rpi3 runs at 60C mostly idle. My pie isn't doing much and I can probably get away with running a pi zero, thinking about getting a few for testing.

Brings up another point, how reliable is the sensor on your pi3? I'm thinking they aren't that great. Take em with a grain of salt for they could be +/- 10C.

Return to “General discussion”