6 Raspberry Pi's crashing at the exact same time

[Radturtle]

I am currently using 6 raspberry pi's all running the exact same script, but it just loops audio. I currently have 7 other Pi's running the same script elsewhere and none of them are having this problem. The 6 are all plugged into the same outlet, so I am thinking it may be a power issue, I think the power is not exactly balanced where they are located so they are just crashing, maybe? I took hooked them up to a UPS, thinking maybe it would help balance out the flow of electricity but it was still causing this issue. Is there a way I can make the Pi's not flip out over unbalanced energy? Or just automatically recover?

[jamesh]

Can you move one of the 6 to the 7 group and see what happens?

But my guess would be an environmental condition somewhere (power etc)

[Radturtle]

Well, the other 7 are scattered around in different locations. We are going to try splitting the 6 off and get them plugged in another way, and if that doesn't fix it, we were going to pull power from another location. I just haven't experienced this type of issue before with the Pi's. They were trying to blame my Pi's for being the problem, but I am blaming the power, lol. But in the meantime I am trying to figure something else out. Seeing if there is any type of way that we can make it either ignore the weird power settings, or just reboot automatically.

[aTao]

I am currently using 6 raspberry pi's all running the exact same script, but it just loops audio. I currently have 7 other Pi's running the same script elsewhere and none of them are having this problem. The 6 are all plugged into the same outlet, so I am thinking it may be a power issue, I think the power is not exactly balanced where they are located so they are just crashing, maybe? I took hooked them up to a UPS, thinking maybe it would help balance out the flow of electricity but it was still causing this issue. Is there a way I can make the Pi's not flip out over unbalanced energy? Or just automatically recover?

is there any monitoring/reporting from the UPS, does it have a mains failure alarm?
If the problem is incoming supply then you have a rubbish UPS.

What do you mean "balanced" mains?

How long between crashes, what electronic installations are nearby? Airport radar, lift(elevator) control room or such?

Where does the audio go after the RPi?

If you do want to chase the mains supply route there there are monitors for hire that you can log any problems.

[Radturtle]

I am currently using 6 raspberry pi's all running the exact same script, but it just loops audio. I currently have 7 other Pi's running the same script elsewhere and none of them are having this problem. The 6 are all plugged into the same outlet, so I am thinking it may be a power issue, I think the power is not exactly balanced where they are located so they are just crashing, maybe? I took hooked them up to a UPS, thinking maybe it would help balance out the flow of electricity but it was still causing this issue. Is there a way I can make the Pi's not flip out over unbalanced energy? Or just automatically recover?

is there any monitoring/reporting from the UPS, does it have a mains failure alarm?
If the problem is incoming supply then you have a rubbish UPS.

What do you mean "balanced" mains?

How long between crashes, what electronic installations are nearby? Airport radar, lift(elevator) control room or such?

Where does the audio go after the RPi?

If you do want to chase the mains supply route there there are monitors for hire that you can log any problems.

We found the problem with the UPS was that it doesn't notice the power difference because it was so minor. We are going to run a new cable from a different location to get our power and see if the problem still occurs. If it doesn't then yay, but if not we will search for another solution.

The UPS we are using is more of a dumb one. One that you plug into the wall and the computer and basically hit the on button, there is nothing more to it. We have other UPS's but I don't really want to mess with them too much.

The UPS doesn't alarm or anything, and the building where our Pi's are located is a temporary structure, and it was put up rather quick, which is why we have power issues. We can be sitting in there and the lights will dim and come back, lol.

It's random between crashes, and it is hard to tell, the kid that was put there to make sure the audio remained going decided every 5 minutes he should just reboot them all.

[Radturtle]

Not sure what is going on, we hooked up a new surge protector to some type of transformer that is supposed to produce "clean" electricity(making sure it is a constant flow of balanced electricity), and we're still having issues. I brought them back to my office and let it run for well over an hour without 1 hiccup. So, in all it isn't the RPI's.

[aTao]

If you have a poor mains supply and any ups/mains cleaner dont do the job then there may also be a problem with an earth connection permitting noise onto the RPi

A bomb proof solution would be..
incoming mains to ups
ups to isolating transformer
isolating transformer to 12v psu
12v psu to ubec for each RPi
RPi audio out to audio transformer
audio transformer to where the music needs to be,

just for good measure put each Pi in a metal box wit its own earth connection.

If that does not work then move away from the radar installation/nuclear physics laboratory/national grid eht switching station

[Radturtle]

If you have a poor mains supply and any ups/mains cleaner dont do the job then there may also be a problem with an earth connection permitting noise onto the RPi

A bomb proof solution would be..
incoming mains to ups
ups to isolating transformer
isolating transformer to 12v psu
12v psu to ubec for each RPi
RPi audio out to audio transformer
audio transformer to where the music needs to be,

just for good measure put each Pi in a metal box wit its own earth connection.

If that does not work then move away from the radar installation/nuclear physics laboratory/national grid eht switching station

Haha, yeah. The RPI's aren't the only problem we are having. We have our camera server in this same room, and something is causing all of the cameras to be extremely fuzzy (Brand new cameras). We have a few guys out there right now trying to figure out a solution, because we can't have this. Lol.

I can try the foolproof way, but I am not sure I can get a few metal boxes done in time. A foolproof way would be a lead box!

[W. H. Heydt]

Not sure what is going on, we hooked up a new surge protector to some type of transformer that is supposed to produce "clean" electricity(making sure it is a constant flow of balanced electricity), and we're still having issues. I brought them back to my office and let it run for well over an hour without 1 hiccup. So, in all it isn't the RPI's.

Sounds like a line conditioner.

[W. H. Heydt]

Haha, yeah. The RPI's aren't the only problem we are having. We have our camera server in this same room, and something is causing all of the cameras to be extremely fuzzy (Brand new cameras). We have a few guys out there right now trying to figure out a solution, because we can't have this. Lol.

I can try the foolproof way, but I am not sure I can get a few metal boxes done in time. A foolproof way would be a lead box!

The sounds like there is some sort of interference. I saw a situation one time where someone was having glitches all over her monitor and it was traced to an unshielded power distribution panel on the other side of the wall.

Is there anything else in the room that might be either putting miscellaneous stuff on the power lines (motors, particularly large ones) or generating electrical noise in the environment, like a microwave oven, or powerful sources outside like the aforementioned (joking about) radar installation...or perhaps either a radio or TV transmitter, or close by cell tower? What about ham operators near by with poor shielding? (I've seen that interfere with a TV set...got him in trouble too when my father sent a tape of a commercial broadcast on top of which you could clearly hear him giving his call letters to the FCC.)

[TassyJim]

It sounds like an earth loop problem or very bad mains supply.

I would go with a 12 volt battery charged by a basic transformer type charger.
Separate 12v to 5v converters for each Pi
Audio isolators on the outputs.

Earth loops are common with temporary supplies and can be a real pain to sort out.

Jim

[Aydan]

It sounds like an earth loop problem or very bad mains supply.

I would go with a 12 volt battery charged by a basic transformer type charger.
Separate 12v to 5v converters for each Pi
Audio isolators on the outputs.

Earth loops are common with temporary supplies and can be a real pain to sort out.

Jim

My first try along these lines would be to power the Pis from a 12V-to-mains-converter off a car battery without even a charger connected to the battery to separate from the mains completely. If you still have problems, then you have an EMI problem and not a mains problem.

Regards
Aydan

[Radturtle]

I suggested to them to get me a metal box, that we can ground. We currently have a rack in there and we have it grounded. I uploaded a picture so you guys can see what I'm working with, excuse the mess. This isn't my ideal location for all of this.

http://i.imgur.com/3Dq9o1Y.jpg

You can see ALL of the power on the left wall, then on top of the little table there, we have the transformer that helps balance out the energy, then on the rack I have 7 Pi's, only 6 of them are used. Along with our camera/sound system under those on the rack, followed by our security system to the right.

I personally would like a completely separate room built detached from this, but it's a little late for that to happen. We are going to revert back to our old sound system for now(which has absolutely horrible sound quality) for the time being, since the Pi's are still having issues. As for out camera system, we are ordering some ground loop isolators. That should clear up all of our noise we are getting in the lines.


Any suggestions?
(And this is a temporary structure, only being used for this month.)

I may try to hook a UPS up to the transformer, then a UPS on the other side of the transformer as-well, maybe it will give a more balanced flow through it? Electricity has never really been my thing.

[Aydan]

I have no Idea what you mean with "balanced flow", and I'm an electrical engineer.
If you have bad mains power and the powersupplies for the pi's can't cope with that, then you need a buffer like a lead acid battery and a DC/AC-converter.
A transformer won't help you with powercuts/-dips. The only thing that a transformer (i assume you're using isolation transformers) does is to sepatate the ground reference of the two circuits. They're normally used to break ground loops.

As suggested before: Use a DC/AC converter and a car battery to test if this will remove your problems. Also, if you're using amps for your audio stuff, I'd suggest to connect all the amps and the raspberrypi PSUs to the same isolation transformer and nothing else and get that circuit properly grounded again.
Also make sure that there's nothing else connected to the pi's and amps that might draw ground from somewhere else.

Regards
Aydan