Pi based scientific camera setup

[NeuroVole]

Hi all! I've spent a few weeks hacking together a system for monitoring animal behavior longitudinally. It seemed to be working well through all our initial tests, but when actually deployed in an experiment it all fell apart haha. I have a few weeks till our next use of the system, so I have some time to improve it, but I'd very much appreciate some best practices advice. Issues I've run into include power-regulation, corruption of HDD filesystems, and some problems with the picamera library. If these sound like interesting problems to you, keep reading! (also, I'm a biochemist/neurobiologist, not an EE, so I apologize in advance for incorrect terminology) Below I lay out a few issues, and any advice on any individual issue would be great, so if you get bored reading, no worries.

Basically, I'm using the [url = http://www.arducam.com/multi-camera-ada ... pberry-pi/]arducam multicamera [/url] adapter to cycle through multiple cameras and save the images to a hard drive. We went with the multicamera adapter vs individual pi's to try and reduce the number of components needed and keep costs lower for other labs. The multicamera adapter basically takes GPIO input to select which of 4 CSI port feeds through to the pi. By stacking two of these boards on top of each other, I was controlling 6 cameras.

OK FIRST ISSUE.
First of all, having multiple cameras plugged in invariably leads to a low-voltage warning. No matter what amp wall plug we used, going into the usb power supply never seemed sufficient. I measured ~4.6v on the 5V GPIO pins. Eventually, I just soldered some wires to a 5.3V power supply, and went directly to the 5V rail. I still was only reading 4.8v across those same input pins with load, with the occasional dip to 4.6, but we don't have an oscilloscope, so that's just off a voltmeter. This helped a bit, but obviously makes me nervous as I'm bypassing the protection of the power supply circuit. I'm planning to upgrade to a switching 20A 5V power supply, as lots of problems seem to coincide with voltage drops. Any advice on why this might be happening? ALSO it feels like performance is degrading over time when I use the pi's, but I havent been able to give them a couple days to cool off and reset. All I know, is that when I first used my jerry-rigged 5V direct power supply the pi was just screaming along, capturing images well and with noticeably faster performance. The next day, after about 10 hrs of image capture and a crash it behaved much more sluggish and I got more frequent low voltage warnings.

EXPLANATION INTERMISSION
We were aiming for 1hz capture rate off of 6 cameras, which meant acquiring images needed to happen on the order of 0.167s. In order to achieve that, I used the picamera continuous_capture function, which can be used as an infinite generator. the code basically goes as follows, and as far as I can tell the capture event happens at the start of each loop:

for cam in continuous_capture():
change gpio pins to activate next camera
check time since last loop
sleep till the next camera should capture, unless its already been too long

This generally works well, especially over short captures. However, there are occasional hiccups where a loop takes 2 seconds or more. I'm not sure what the bottleneck is that's causing those hiccups.

This generates a HUGE number of individual files, that I will concatenate into videos. Another strange occurrence is that occasionally, I get two pics in a row from the same camera, despite the GPIO pin swap. Its rare, but is going to make collecting the images that belong to each camera much more difficult.

OK NEXT ISSUE.
Sometimes the python code running the cameras hangs, and I cant seem to disconnect from the cameras after a keyboard interrupt. I am using a with Picamera() as cam call to try and mitigate these communication issues, but when it happens I cant even reboot the pi. it hangs on shutdown and I need to manually unplug it.

OK NEXT ISSUE.
I'm getting very inconsistent connections to our data storage HDDs (4TB, exFAT). I think this is related to the low voltage issues I'm having. Sometimes, if images are capturing, and I try and open the folder where the images are being saved, the pi kind of half-way crashes and all of the folder windows close. Then, I lost the ability to read one of the HDDs on my Mac, even though it's read just fine on the pi.

OK THANKS!
Hey, if you read all that, thanks so much! Any advice on any little issue would be greatly appreciated. I'll gladly pay anyone back with beer, or tutoring in biology.

[drgeoff]

Doesn't matter how much current the PSU can deliver if the wires connecting it to the RPi are too long and/or too thin. Wires are not not perfect conductors. They do have resistance which is directly proportional to the length and inversely proportional to the square of the diameter of the conductor. Voltage drop is directly proportional to the resistance.

[6by9]

Low voltage - I'm not surprised.
Also be aware that all cameras will be powered up simultaneously from the 3.3V regulator. Reading the IMX219 (V2.1) datasheet it pulls typically 100mA when streaming, so you've just loaded the 3.3V line with an extra 600mA.

The Pi camera stack has no knowledge of your switcher. It is trying to capture a stream of images continuously in video mode, and will pipeline the processing to achieve the frame rate you ask for. Switching the input will have potentially a frame or two of delay before a valid frame from the new sensor makes it through - that's the way it is. You can even get frames with a tear in them with part from one sensor and part from another - those can be further corrupted too.

Also the exposure, gain, and white balance control loops have no idea that you're switching their image source, so your sources better be closely matched in illumination or you'll get some odd images.

As for the lockups, if you were using MMAL directly then I'd look into it. I'm afraid that I hate Python with a passion, therefore I won't be digging into your lockups there. Raise an issue on https://github.com/waveform80/picamera/issues if you have a reproducable test case which doesn't require additional equipment.

[DarkPlatinum]

If I were you I would use the "Official" power supply. It will prevent you from getting corruptions if your pi ever does not have enough power.

[davidcoton]

If I were you I would use the "Official" power supply.

If as suggested by 6by9 above the power problem is an extra 600mA on the 3V3 line, then the limitation is the on-board 3V3 regulator, not the external supply. The cameras will need a separate 3V3 regulator. The 5V supply may or may not be adequate to feed it.

If I were you I would use the "Official" power supply. It will prevent you from getting corruptions if your pi ever does not have enough power.

How?? The official supply is good, it will avoid power problems in many cases. But it is not magic (no integrated unicorn ), so it can't prevent SDCard corruption when it is overloaded.

[DarkPlatinum]

How?? The official supply is good, it will avoid power problems in many cases. But it is not magic (no integrated unicorn ), so it can't prevent SDCard corruption when it is overloaded.

Yes my english is not very good What I meant is that he will have a less chance of getting corrupted SD cards if he uses a good supply, if the pi draws too much power it could reboot and possibly corrupt the sd card if it is updating.

[ejolson]

Any advice on any little issue would be greatly appreciated.

Welcome to the forum!

Though you've reported that it's not reliable, I'm amazed that a single Pi can power 6 cameras and it works at all. Even if you get the 5V supply sorted, it seems possible 3.3V could still be an issue. It will be quite interesting if the new 5V power supply is enough to make things reliable.

If I needed to run 6 Pi cameras I would use a cluster consisting of one Pi B+ and five Pi Zeros. The Zeros can be networked (and powered at the same time) using a powered USB hub. Furthermore they can boot in device mode without SD cards from the B+. Thus, the cost is about the same as the stacked hats. However, since there is only one camera per Pi, I believe you are less likely to run into reliability problems.

I built such a cluster, but did not attach cameras to the individual Pi computers. Instead, I used the cluster to model how a super computer works and practiced writing some parallel programs using MPI. Without the cameras the cluster is reliable and performs surprisingly well. Hopefully it would still be reliable with cameras attached. More information about this cluster is available here. Note in particular this post which discusses building a security surveillance system with a similar cluster.

[walterav1984]

(my)Best practices:

Inform (forum members) as much as you can, especially when you notice people respond on your interesting topic.

What raspberry camera's are you using new/old/noir since there are 2 versions of the arducam?
Which raspberry pi version?
Which raspbian/kernel version: lsb_release -a;uname -a #?
Are you running a desktop on each pi or just cli/shell?
Besides voltage errors are there other errors in dmesg?
How are the Exfat harddisks connected directly on USB or via nfs/samba NAS/netwerk?

If HDD's connected directly on USB, reformat to ext4 or btrfs (if you care about scientific data). If you want stability and speed Exfat is not an option it uses Fuse (File system in userspace) and buffers writes/reads in strange way that might be risky on weak CPU/RAM systems. Especially if you want to write repeatedly 100000' thousands of stills instead of big lumpy avi files which are easier and use much lower stress on a filesystem. Don't get me wrong Exfat is nice as a last resort compatible Mac/Win/Linux filesystem for temporary data transfers for single big media files. Not for archive&storage.

You use MacOSx that's fine however BTRFS/Ext4 are not supported, instead rather install virtualbox with debian/ubuntu virtualmachine with samba filesharing that will nativly read ext4/btrfs from USB harddisk(shared to your vm) and shares it to your mac (its ugly but in read only mode safe and usable). If you insist on exfat you may even use a Windows based server with exfat hdd's with a samba share to your raspberry pi but please understand that if your writing data you should do that in a way that puts the least stress on the system that's gathering it...

Like 6by9 already mentions that the camera's have some sort of warm up time of 2~ seconds to get a stable whitebalance/noise/etc every time your script asks for a still (if not in video mode), if you want a 1hz rate you should be using video with alot of discarding bad frames/bad data. Eventhough there might be a way to clock sync the 4 camera's I don't know if that means that also the data stream from the camera's comes into the pi at the same rates which can be accepted by the video capture driver. Might explain your half frames?

If you are a researcher in the Netherlands/Leiden please let me know for a drink/tour on biology .