User avatar
jbeale
Posts: 3492
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: still capture rate (still FPS)

Tue Sep 22, 2015 3:55 pm

Eben's recent interview said the replacement RPi camera was not yet determined, but they are targeting "at least as good" and "at least as good a price" as the current camera. If they are keeping the $25 price point, I'm guessing it won't be substantially higher resolution (and if you need a higher frame rate, the hardware might not keep up anyway). You can imagine them supporting two cameras, "standard" and "premium, high-res" (which I would definitely buy myself, btw) but apparently the necessary camera tuning process is a major effort and supporting more than one camera is likely outside the scope of the RPi mission, I'm guessing.

Vertigo72
Posts: 23
Joined: Wed May 29, 2013 6:41 am

Re: still capture rate (still FPS)

Tue Sep 22, 2015 4:20 pm

Higher res wouldnt necessarily impact the price of the sensor much, if at all. Smartphones these days all have 10+MP sensors. Of course, the usefulness of those pixels is questionable if they become physically too small and the optics dont improve as well, and those things certainly would impact price.

FWIW, ive tried a large (and surprisingly heavy) CS lens on the standard Pi camera module; I noticed no real improvement in image quality, only the shutter speed improved by around a factor 2x. Which was nice, but not nice enough to warrant the whopping 100gr weight for my application.

Anyway, if anyone is interested, this is my "Pi in sky" project, aerial orthophotograpy with a raspberry Pi + RPI camera:

https://www.youtube.com/watch?v=N6f3_DOcG78

The plan is/was to add several modules to a single Pi with a multiplexer and do multispectral and NDVI imaging too, but with the current sensor thats stretching what is possible. To get the above result I needed to fly high and slow on a sunny day on a field with very distinct patterns. Anything else is difficult.

User avatar
jbeale
Posts: 3492
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: still capture rate (still FPS)

Tue Sep 22, 2015 4:54 pm

Your aerial orthophotography project is truly impressive! Great use of a Pi... that should go on the RPi front page/ blog in my opinion. With applications like this, maybe the powers that be will consider an improved-spec camera :-)

Vertigo72
Posts: 23
Joined: Wed May 29, 2013 6:41 am

Re: still capture rate (still FPS)

Tue Sep 22, 2015 5:09 pm

And multiple CSI interfaces too, pretty please :)

User avatar
waveform80
Posts: 303
Joined: Mon Sep 23, 2013 1:28 pm
Location: Manchester, UK

Re: still capture rate (still FPS)

Tue Sep 22, 2015 6:47 pm

Okay, several things to go through here:
  • A Pi 2 won't help much because the camera doesn't use the CPU - the firmware runs on the GPU, and that's exactly the same on the Pi 1 as the Pi 2. If you're overclocking stuff, the Pi 2 might be more tolerant, and obviously the extra speed doesn't hurt stuff like IO, but I wouldn't expect huge differences.
  • You can still trigger captures at particular moments when using video (well, as much as you can regularly). If you've read any warnings in the picamera docs about having to do stuff quickly in custom outputs, ignore them - they're wrong (written before I understood exactly what happens). All that happens if you go slow is that the camera drops frames until you start returning buffers to it again.
  • This means capturing images when you like is trivial (actually, it'd be pretty trivial anyway: just discard the frames you don't want and write the ones you do want to disk). I'll bung a couple of quick demos below.
  • On more advanced cameras I can't say I know anything special (I don't work for the foundation), but given that the camera's firmware runs almost entirely on the GPU, and on the assumption that any future camera's firmware would do likewise (to keep the cost down), it probably couldn't offer a *much* higher resolution. I *think* I'm right in stating that one of the criteria for the GPU's capacity was specifically so the camera could do 1080p H.264 - but that is at the GPU's limit. Hopefully that's not idle speculation I'm remembering from somewhere!
Anyway, how to capture frames when you want using MJPEG video recording (bear in mind the quality won't be as good as the still port, but it will be *much* faster) - see the second recipe under here:

https://gist.github.com/waveform80/263b9c8bdcb1e9b79749

However, it's worth noting that's using 1280x720. I can't remember off the top of my head whether MJPEG works at max. res (H264 definitely doesn't) and if it does, whether it requires a larger GPU mem setting. If it does require a larger GPU mem setting, then a Pi 2 might be worth it anyway (given the extremely limited RAM on the model A+).

Hope that helps!

Dave.
Vertigo72 wrote:
waveform80 wrote:raspistill shoots pictures in the "still mode" of the camera. In this mode, a *lot* of work goes into producing the image including mode switching to the highest resolution, an aggressive denoise algorithm, etc. You're unlikely to get beyond 1fps with this. Shooting video at 15fps (and actually a lot higher) is indeed possible. Here's a little gist I whipped up a while ago which fires up video recording with MJPEG output, then extracts each JPEG from the output as it goes:

https://gist.github.com/waveform80/263b9c8bdcb1e9b79749

On my Pi2 I can *easily* produce 15fps from this (until the disk cache runs out of space and the SD card's IO limitations kick in). Now, as to "full frame", do you just mean "full field of view" (which the above will accomplish happily by 2x2 binning of the input), or do you mean "full resolution" (which the above probably won't manage due to various limitations).
Would a Pi 2 be any help in still mode? Im flying a RPI (1A+) on an autonomous RC plane for aerial mapping, and I need around 1-1.5 FPS at the full 5MP. I can however compromise a bit on jpeg quality to keep the sd card from being a bottleneck. Right now Im falling just short of that 1-1.5 FPS, and missing frames in flight. I dont think I can use the mjpeg trick because the captures are triggered externally at exact moments. So Im considering a raspberry 2, but I have no idea if it will any difference?

On a side note; I really wish someone would make a better raspberry camera module. The current one is decent enough for the price and a lot of applications, but Im sure many like me could use a ~15MP camera with better optics. The step from a 3gr $25 module to a 100+gr $200+ point and shoot camera is a bit steep.

Vertigo72
Posts: 23
Joined: Wed May 29, 2013 6:41 am

Re: still capture rate (still FPS)

Wed Sep 23, 2015 7:16 am

Wow, thanks a lot for that elaborate response and particularly for the sample code (Im not much of a programmer, so thats a big help). Ill see at what resolution the mjpeg works; if it works at or very near the full 5MP, this could be a workable solution.

Meanwhile, I have also received this multi camera adapter:
http://www.uctronics.com/multi-camera-a ... board.html

Ive yet to play with it, but I wonder how using it would affect the issue. The adapter switches between several CSI modules by setting some GPIO's. Initially the idea was to combine a regular and a NDVI (ir sensitive with special filter) module, but seeing how fast I need to be, having two completely different camera's that require (very) different exposure times and white balance is probably not realistic. Or do you think the GPU will manage that in "real time" ?

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23647
Joined: Sat Jul 30, 2011 7:41 pm

Re: still capture rate (still FPS)

Wed Sep 23, 2015 7:34 am

The standard CSI port on the Pi is two lane, which limits the maximum throughput of pixels. The compute module exposes 4 lanes, so can do a lot more. This is how the Nokia 808 42MP sensor was attached. However, things like H264 encode HW are limited to just over 1080p30, so there are other limitations. For stills, it's less relevant, as the JPEG encoder is pretty quick.

So any replacement sensor would need to be a two lane device, which limits the FPS available. So don't expect 5MP 30fps!
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7288
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: still capture rate (still FPS)

Wed Sep 23, 2015 10:22 am

Vertigo72 wrote:Wow, thanks a lot for that elaborate response and particularly for the sample code (Im not much of a programmer, so thats a big help). Ill see at what resolution the mjpeg works; if it works at or very near the full 5MP, this could be a workable solution.
Memory says we had 5MP @ 15fps MJPEG coming out of the V4L2 driver. Please be aware that the MJPEG codec is not of the same code quality as the H264 codec and there are known potential lockup conditions - they're unlikely to get fixed as it would require a complete restructure of the codec! If you can ensure the input is disabled before the output, then you should be OK.
Vertigo72 wrote:Meanwhile, I have also received this multi camera adapter:
http://www.uctronics.com/multi-camera-a ... board.html

Ive yet to play with it, but I wonder how using it would affect the issue. The adapter switches between several CSI modules by setting some GPIO's. Initially the idea was to combine a regular and a NDVI (ir sensitive with special filter) module, but seeing how fast I need to be, having two completely different camera's that require (very) different exposure times and white balance is probably not realistic. Or do you think the GPU will manage that in "real time" ?
We were amused by this when it first came out.
The only reason it works is that there is an error recovery mechanism built into the firmware as one customer wanted to get an event if they short circuited or open circuited the CSI2 lines (yes, seriously!). The GPU knows the expected frame rate, and sets a timer to IIRC 4 frame periods. If it receives no CSI data before the timer expires it does a full power cycle and reprogramming of the sensor. If this fails 3 times in a row, then the GPU gives up and throws an error to the client - game over until you recreate the camera component.
This switcher board is really taking advantage of this reset if you switch whilst the camera is streaming. The restart will be with the exposure and gain settings in place before the reset, so AGC and AWB will have a reasonable place to start from, but will need a few frames to settle again.

As jamesh says, the Compute Module has both CSI2 interfaces of the BCM2835 brought out - one is 2 lane, the other is 4 lane. Each lane is nominally 1Gbit/s, so if there were no overhead it could carry about [email protected] for 10bit/pixel Bayer. There is an overhead, so you can't hit that in practice, but that is one fundamental limit. OV5647 is limited by the speed of the ADC in the sensor to 75MPix/s (5MP15) :(
The ISP hardware can cope with somewhere between 150 and 200 MPix/s, so should just keep up with a saturated 2 lane CSI2 interface. At one point on a different project we cheated - 4 lane CSI-2 carrying 8MP30 into SDRAM, but for preview only reading every third line of the data. That maintains the Bayer order but drops the ISP throughput. When a still was requested it reprocessed the raw 8MP frame to create a JPEG. :D

The H264 encoder is between level 4.0 and 4.1 (the difference between them is the permitted output bitrate), so [email protected], or 2,048×1,[email protected]
There is a line buffer in the hardware that is (I believe) 2048 pixels wide. You can not exceed that - the codec fails to open.


You don't say how you are triggering the captures or setting things up - V4L2, python, or raspistill? Whichever you are using there are various tweaks I'd suggest to improve the burst performance. The main thing is to allow the camera sufficient buffering so it doesn't have to stop the sensor streaming at all - the defaults are a compromise on memory required vs performance. Adding "-md 2" to a raspistill command line should do it (I'd want to double check it does do that in reality - it may want -fp as well, although that screws up the display. Otherwise change this line to "if(1)").
At the moment you can't use the -bm flag for burst mode as that locks AGC and AWB.
Another option is to switch to using the video encode port for the stills captures (output[1] instead of output[2]). That's the trick that waveform80's PiCamera library can pull easily. You lose some of the extra image processing and will lose all EXIF tags, but can increase the capture rate significantly.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Vertigo72
Posts: 23
Joined: Wed May 29, 2013 6:41 am

Re: still capture rate (still FPS)

Wed Sep 23, 2015 11:00 am

6by9 wrote:If you can ensure the input is disabled before the output, then you should be OK.
Probably a dumb question, but what exactly do you mean here?
This switcher board is really taking advantage of this reset if you switch whilst the camera is streaming. The restart will be with the exposure and gain settings in place before the reset, so AGC and AWB will have a reasonable place to start from, but will need a few frames to settle again.
A few frames is fine! Anything under ~a second would be acceptable for my use case. Guess I'll have to give it a try then. The dual camera NDVI imaging probably wont work anyway, but for other reasons (exposure times are likely too long with an infrablue filter to be usable on a UAV).
As jamesh says, the Compute Module has both CSI2 interfaces of the BCM2835 brought out - one is 2 lane, the other is 4 lane.
Ive actually considered a compute board, but the dev kit is a bit bulky. I had hoped more compute boards would come to market for this very purpose, but the only one Ive seen so far is in that OTTO gif camera, and it only has one interface :(. And now, basing everything on a RPI 1 seems a bit silly, so hopefully there will be a RPI2 compute platform and someone making a compact board that exposes the CSI interfaces.
You don't say how you are triggering the captures or setting things up - V4L2, python, or raspistill?
Im (ab)using python. Can I use those -md 2 and -fp parameters there too? Its ok if the display gets screwed up, Im not using it right now. I had planned on connecting the Svideo to my FPV transmitter so I can remotely monitor it, but really, there isnt much point.
Another option is to switch to using the video encode port for the stills captures (output[1] instead of output[2]). That's the trick that waveform80's PiCamera library can pull easily. You lose some of the extra image processing and will lose all EXIF tags, but can increase the capture rate significantly.
But that doesnt work at 5MP, or does it?
As for the exif tags, can I query the camera to get exposure time, and add the exif tags manually? I want to add other exif metadata anyway (GPS coordinates, height, 3 axis angles, etc from the flight controller..), so I might as well put those back in.

Anyway, huge thanks already, also for everything else you've done for this community. Just about every thread Ive ever read on raspberry + camera's has invaluable input from you.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7288
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: still capture rate (still FPS)

Wed Sep 23, 2015 1:13 pm

Vertigo72 wrote:
6by9 wrote:If you can ensure the input is disabled before the output, then you should be OK.
Probably a dumb question, but what exactly do you mean here?
On the video_encode component you've got an input port (probably connected to the camera) and an output port (probably linked to the client to save the data). Disable the input side first, then the output side. The cleanest way is actually to set MMAL_PARAMETER_CAPTURE to false on the camera component port to video_encode. That sends a special END_OF_STREAM (EOS) message down the pipe, and when you receive that at the client you know you have all the data.
The issue is that if the video_encode blocks if it can't write into the output FIFO, but it blocks in the same thread that handles the codec flush on shutdown. It would require a rewrite of the codec to have a worker thread that blocks on FIFO full, but allows the thread feeding in frames to resume and hence handle the flush. If you've had the EOS passed, then you know there is nothing in the FIFO, so everything will clean up nicely.
Vertigo72 wrote:
This switcher board is really taking advantage of this reset if you switch whilst the camera is streaming. The restart will be with the exposure and gain settings in place before the reset, so AGC and AWB will have a reasonable place to start from, but will need a few frames to settle again.
A few frames is fine! Anything under ~a second would be acceptable for my use case. Guess I'll have to give it a try then. The dual camera NDVI imaging probably wont work anyway, but for other reasons (exposure times are likely too long with an infrablue filter to be usable on a UAV).
Your call to make. There are so many potential use cases with the Pi camera we can't cover them all with defaults. Mobile phones were a doddle in comparison!
Vertigo72 wrote:Ive actually considered a compute board, but the dev kit is a bit bulky. I had hoped more compute boards would come to market for this very purpose, but the only one Ive seen so far is in that OTTO gif camera, and it only has one interface :(. And now, basing everything on a RPI 1 seems a bit silly, so hopefully there will be a RPI2 compute platform and someone making a compact board that exposes the CSI interfaces.
CM2 is promised, but no known timescales yet - see Eben's recent interview where he said Pi2B+ is taking all the available BCM2836 supply at the moment.
Dev kits are a pain, but those aren't mass market devices so the price is high. Producing custom PCBs isn't as expensive as it used to be, but still has quite a steep learning curve.
Vertigo72 wrote:
You don't say how you are triggering the captures or setting things up - V4L2, python, or raspistill?
Im (ab)using python. Can I use those -md 2 and -fp parameters there too? Its ok if the display gets screwed up, Im not using it right now. I had planned on connecting the Svideo to my FPV transmitter so I can remotely monitor it, but really, there isnt much point.
http://picamera.readthedocs.org/en/rele ... ensor_mode Ideally set it from the constructor.
Vertigo72 wrote:
Another option is to switch to using the video encode port for the stills captures (output[1] instead of output[2]). That's the trick that waveform80's PiCamera library can pull easily. You lose some of the extra image processing and will lose all EXIF tags, but can increase the capture rate significantly.
But that doesnt work at 5MP, or does it?
As for the exif tags, can I query the camera to get exposure time, and add the exif tags manually? I want to add other exif metadata anyway (GPS coordinates, height, 3 axis angles, etc from the flight controller..), so I might as well put those back in.
Should be able to work at 5MP. There's no reason in the firmware why it shouldn't work - that's the mode we've had V4L2 delivering 5MP15 YUV frames to the client. You may need to set the resolution from the constructor to get the right config pushed into the GPU (it's the values of w & h at https://github.com/waveform80/picamera/ ... ra.py#L586 that matter).
You should be able to retrieve the exposure speed, analogue gain, and digital gain for use in the EXIF block. The firmware can send a message to the client every time they change, and I think that is what PiCamera has hooked into (-set option in raspistill/vid).
Vertigo72 wrote:Anyway, huge thanks already, also for everything else you've done for this community. Just about every thread Ive ever read on raspberry + camera's has invaluable input from you.
No problem. I'm always amazed at the numerous projects people end up using this little camera for - perhaps 7 years at Broadcom weren't quite so wasted.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Vertigo72
Posts: 23
Joined: Wed May 29, 2013 6:41 am

Re: still capture rate (still FPS)

Thu Sep 24, 2015 10:50 am

Time for a little feedback;

First of all, I need to apologize for wasting much of your time, as it appears the core of my problems where actually related to the SDcard writing speed and the mavlink deamon running in the background to talk to the flight controller. I disabled the latter for testing, replaced the sdcard with a faster one and reduced the file size a bit by decreasing jpg quality. That got me to ~600ms per photo which is adequate. There is still an occasional few second hickup, but it probably has nothing to do with the camera or its settings and everything with some background process or cache being written to disk.

Still, I tried the recommendations to see if I could go faster, as faster might still be useful when using multiple camera's. The mjpeg trick seemed to work (extremely fast). I couldnt get it to work at the full resolution, but 1920x1200 still worked and perhaps higher. But as predicted, the quality suffers. In particular I got some nasty banding issues, like when looking at a CRT with a digital camera. Thats bound to confuse my photogrammetry software, so I abandoned that approach.

Next up, I tried setting the sensor mode. I changed

Code: Select all

camera = picamera.PiCamera()
to

Code: Select all

camera = picamera.PiCamera(sensor_mode=2)
To my surprise, instead of improving things, that added almost exactly 100ms to each frame, from 600 to 700ms. I tested a few times, and it was 100% reproducible. Setting it to zero also gave me 600ms.

Out of curiosity, I tried setting the sensor mode to 3, and that created an error the moment I tried to make a photo: 'Received unexpected camera control callback event, 0x4f525245'

Oh well. 600ms will do for now.

I also briefly tested the multi camera adapter. Ive not tried anything fast yet, just some test code that makes pictures from the various ports, and it does indeed work.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7288
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: still capture rate (still FPS)

Thu Sep 24, 2015 11:42 am

Vertigo72 wrote:Next up, I tried setting the sensor mode. I changed

Code: Select all

camera = picamera.PiCamera()
to

Code: Select all

camera = picamera.PiCamera(sensor_mode=2)
To my surprise, instead of improving things, that added almost exactly 100ms to each frame, from 600 to 700ms. I tested a few times, and it was 100% reproducible. Setting it to zero also gave me 600ms.
Slightly bizarre, but at a guess it'll be because whenever you switch from preview to stills it completes the frame that it is on. With 5MP running at half the frame rate, completing the frame will take an extra 33ms. Whenever the sensor starts streaming the first frame is rubbish and will be dropped - another 66ms if at 15fps.
I wonder if we are stopping the sensor even though we don't need to change mode. There is no need to stop the sensor, so one for me to check within the firmware. It may be a buffering issue again.
Vertigo72 wrote:Out of curiosity, I tried setting the sensor mode to 3, and that created an error the moment I tried to make a photo: 'Received unexpected camera control callback event, 0x4f525245'
Totally expected. Mode 3 is the very long exposure times mode and produces frames at a maximum of 1fps. You've told the system overall that you're expecting (probably) 15fps. After 4 frame periods we still won't have seen a frame so it resets the sensor. That won't help, so it'll reset again, and again, and finally give up and notify the client.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

User avatar
jbeale
Posts: 3492
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: still capture rate (still FPS)

Thu Sep 24, 2015 3:52 pm

Vertigo72 wrote:The mjpeg trick seemed to work (extremely fast). I couldnt get it to work at the full resolution, but 1920x1200 still worked and perhaps higher. But as predicted, the quality suffers. In particular I got some nasty banding issues, like when looking at a CRT with a digital camera.
Resolution will always suffer in video mode but ideally you should not have banding. Banding can be caused by interfering nearby AC signals, conducted via power supply or capacitively or inductively coupled into the camera chip. I presume you've got some high currents (modulated with PWM?) running to the motors on your plane (plus RF transmitter for video and/or control?) However in that case I'd expect the problem to appear with still mode just as well as video mode, so maybe it is strictly a software thing.

How many bands do you see? If it's just two parts of the image with different brightness, could be AGC changing while a buffer gets part of one frame and then part of the next frame with different exposure, that is some synchronization not happening between camera and buffer.

Vertigo72
Posts: 23
Joined: Wed May 29, 2013 6:41 am

Re: still capture rate (still FPS)

Fri Sep 25, 2015 9:14 am

It wasnt external interference, as nothing was running besides the Pi and the camera. The banding was one or two thick horizontal bands and it could perfectly well be what you described. Sorry I didnt save the pictures, but the weren't much to look at as I was testing with the IR sensitive version staring at the ceiling ;).

BTW, when it comes to interference, the problem is much more the other way around. I had to use longer CSI cables and those act as antenna's, wreaking havoc mostly on my GPS reception. I had to wrap the cables in tin foil to get it down to acceptable levels. I guess shielded (and round!) CSI cables is not something anyone makes? Those flat cables are also a bit of a problem with the micro gimbal Im working on.

User avatar
jbeale
Posts: 3492
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: still capture rate (still FPS)

Fri Sep 25, 2015 4:34 pm

Actually there are a number of people who I remember posting about ribbon cable adaptors and then separating groups of conductors so it becomes in effect several smaller ribbon cables, which are more flexible for exactly that purpose (camera on a gimbal or pan/tilt mount). You want to separate them judiciously so signals and grounds stay together, though. I think this involves an adapter on each end, so the total weight and size would increase, and probably RF interference as well. Possibly you could get creative with lengthwise dividing a normal flex with a razor blade into several groups for the same effect, but that would require some pretty steady hands and I don't know anyone's made that work.

By the way, I just learned about an upcoming very small RTK module claiming cm-level GPS precision, might be useful (they specifically list autonomous UAV as an application). Not yet released and I haven't seen a price tag though. I do have some other GPS modules from this company, which work as described. http://navspark.mybigcommerce.com/conte ... cation.pdf

Vertigo72
Posts: 23
Joined: Wed May 29, 2013 6:41 am

Re: still capture rate (still FPS)

Fri Sep 25, 2015 8:09 pm

I used to cut up (P)ATA ribbon cables, but there is no way I'll manage these CSI cables. Not even going to try :) I also saw some people use HDMI cables and breakout boards, but thats not feasible for me either. Its not too bad with one camera though, and once I move to more camera's on a bigger plane, Ill mount the raspberry on a gimbal too, so there is no flex to worry about.

As for RTK; that looks neat (and probably pricey), but I dont need anything like that precision. Its not like the plane itself can maintain a track that accurate when there is wind - or even when there isnt. Maybe a very good multirotor could, but I cant. ~10 meter is perfectly acceptable for what Im trying to do, since Im typically shooting from around 100m altitude and the photo's overlap by 20-30 meter ground resolution. Then I just let the software figure out how/where they were taken exactly. Maintaining a level camera is going to be much bigger gain I hope, here is my $6 gimbal prototype to stabilize my $25 camera:

https://www.youtube.com/watch?v=tiB7pbymI24

:)

User avatar
jbeale
Posts: 3492
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: still capture rate (still FPS)

Sat Sep 26, 2015 12:06 am

the 2-axis servo mount looks very neat! I wonder how quickly the vibration from the servo's motion damps out after it stops moving (or is it always moving, also during image capture? does the geartrain backlash allow you to control it accurately enough?) Very interested to hear about the results!

Vertigo72
Posts: 23
Joined: Wed May 29, 2013 6:41 am

Re: still capture rate (still FPS)

Sat Sep 26, 2015 7:31 am

The gimbal is "always on". The servo's are fast, faster than the plane itself can roll or pitch and there isnt any real vibration in their controlled movement, although the entire assembly has enough slop that wind could potentially cause some vibration, not helped by my not so aerodynamic design ;) (in the next plane, this will all be internal). Also one of the servo's (roll) doesnt have the best resolution and the angle between two steps is small but noticeable. I just used two spare analogue servo's I had laying around, they are not the same. The one I used for pitch is actually quite good for this as it turns out, may order another one of those.

But I dont think any of that will impact the result much. Its not like the camera was stationary before because the plane never was (check the video, its rolling and pitching fairly violently against the wind). With shutter times on the order of 1/2000 - 1/5000th of a second, I expect that wont matter too much. I hope, we'll see. The goal was to ensure complete ground coverage with enough overlap for the software to be able to stitch the images and reconstruct in 3D. That didnt quite work out without the gimbal because of the constant rolling and pitching.

Now video would be another story, that wouldnt look very good, you want a smoother brushless gimbal for that. But I'll record some anyway.

Vertigo72
Posts: 23
Joined: Wed May 29, 2013 6:41 am

Re: still capture rate (still FPS)

Sat Sep 26, 2015 9:30 am

You actually made me think it through a bit more. Since the resolution of these servo's isnt infinite and the servo's potentially do move faster than the plane could roll or pitch, there is a risk that if the frame is shot while the servo is moving beween two steps, angular velocity could be higher than without servo. I did some testing on the ground, shaking the plane while recording, the video is pretty terrible and jelo is introduced. But photo's remain sharp even while shaking and moving very fast; only a small percentage of them have any noticeable, but still minor motion blur, but that may even be because Im not only rotating but also moving the camera and shooting at the ground which is close by. In some I can see some minor geometric errors though, like a reverse fish eye effect. Didnt expect that, but I doubt it will be a factor if Im not flying in a thunderstorm.

So, when is someone going to bring out a global shutter camera for the pi ? :)
Or for that matter, for anything that doesnt cost >$1000.

Vertigo72
Posts: 23
Joined: Wed May 29, 2013 6:41 am

Re: still capture rate (still FPS)

Sun Sep 27, 2015 1:19 pm

Meh. Now my camera stays level, that actually works very well but this is the result on a windy day:
Image

Pretty severe jello, look at the roof or the side walls of that whatever it is.

Have to think if this could be due to the one not so good servo or if this is something that will always happen when using a servo gimbal, despite the fast shutter times.

User avatar
jbeale
Posts: 3492
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: still capture rate (still FPS)

Sun Sep 27, 2015 2:56 pm

Beautiful detail in that photo, apart from the jello effect. Yes, looks like you improved low frequency pointing stability at the cost of high frequency stability. If you had "perfect" servos and a fast enough control loop, there should be no jello effect.

I wonder if you could improve it by using very soft foam between camera board and servo to damp out high frequencies. Would work better if camera had more mass (well, rotational inertia) but thats obviously a tradeoff...

Vertigo72
Posts: 23
Joined: Wed May 29, 2013 6:41 am

Re: still capture rate (still FPS)

Sun Sep 27, 2015 3:24 pm

ultimately, Ill need a brushless gimbal, they dont cost that much anymore, but on my current test plane I have neither the room nor lifting capacity (its already way over its intended weight and so full Im carrying the pi outside the fuse) so while next plane is being designed/built, I carry on testing just to see what I can achieve with a raspberry camera. And I agree the photo's are pretty decent aside from the jello, they seem sharper than before and thats why Im not that disappointed.

I was also thinking of dampening the movement, but that dampening would have to happen between the servo and the camera. Something like they are doing here with some rubber rubing: http://www.rcgroups.com/forums/showthread.php?t=1793759
I cant do it quite like that, because I dont have the ground clearance (no surprise, my gimbal was ripped off after the first landing) and I dont think my flight controller can drive a gimbal that way, it expects one axis per servo and not some mix, but Ill see what I can come up with.

Vertigo72
Posts: 23
Joined: Wed May 29, 2013 6:41 am

Re: still capture rate (still FPS)

Mon Sep 28, 2015 7:18 am

Im thinking off another way: disable the servo's right before the camera is triggered. I dont see an easy way to accomplish this in the pixhawk, but maybe I can use the RPI. If I 'cut' the PWM signal wire between the flight controller and servo's, the servo's will just maintain their position. Is there way to do that with GPIO that doesnt involve a relay?

BTW, I finished the photosynth despite the jello. 3D reconstruction utterly failed, probably because of it, but 2D stitching worked more or less ok: https://photosynth.net/embed.aspx?cid=6 ... 7b253313fd

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23647
Joined: Sat Jul 30, 2011 7:41 pm

Re: still capture rate (still FPS)

Mon Sep 28, 2015 8:40 am

Are you sure that jello effect is from the gimballs - I would have thought its too high a frequency - looks like vibration from the props/engines to me. There's about 15 'jellos' on that frame, which is I presume from a 30fps image, which would be at the slowest, a vibration of 450Hz. Likely higher as it would depend on the exposure time, which would be 1/30th or less. So engine rotation RPM of about 27k, Is that about right for the engines?
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

Vertigo72
Posts: 23
Joined: Wed May 29, 2013 6:41 am

Re: still capture rate (still FPS)

Mon Sep 28, 2015 9:58 am

Im not entirely sure, but even on the ground, with no motor running, there is very obvious jello in the video if the servo's are on. Here is a clip while the motor isnt even running:
https://www.youtube.com/watch?v=H5Amdxp ... e=youtu.be

(youtube pixelator makes it hard to see at times, but you'll notice clearly in the second half). Also, the prop is well balanced, the mobius shows no jello whatsoever and before I mounted the servo gimbal, I didnt have any either. it could be that wind is making the problem worse since the camera mount isnt all that rigid, but servo's play a role, im fairly certain.

But Im also a little surprised at the number of waves. Then again, Im not sure how fast a photo is taken. Shutter time doesnt mean anything, because its a rolling shutter camera. How long does it actually take from first to last line to make a full 5MP photo? Anyone know ? Im guessing its on the order of 100ms or more (10Hz) ?

Return to “Camera board”