Question about RaspiCam native resolution

[jwatte]

I recently ran an autonomous car at robogames using a Raspberry Pi 2 and the RaspiCam. Great fun!
I hacked up the raspivid example to send the preview stream to my own buffer for image recognition, while still encoding the H.264 stream to disk for after-action review. (I have plenty of takes of hitting walls and shrubberies now!)
I spawn off additional threads for image processing and path following control inside this hacked-up executable. (Multithreading. My favorite!

I'm trying to understand what the "best" resolution is for my particular needs, and would appreciate any pointers, fact tidbits, and suggestions.

I've been using 512x288, because it divides nicely into 16, and it's not too many pixels to process in real time using the host CPU. In fact, I installed GCC 4.8 and built my code with Neon, and it uses less CPU than I thought, so I probably have headroom to improve the resolution. (I started out on a Pi 1, and the 2 is a real improvement!)

Specifically, I'm interested in high per-pixel quality (so, hardware binning would be great!) and low latency (one millisecond of extra latency isn't bad; one extra frame of latency is undesirable.)

Some numbers I'm considering, based on 2592 x 1944 native resolution:
divide by 4 -> 648 x 486 -> this doesn't divide evenly into 16; what will the H.264 encoder do with this?

I'm more interested in a wide FOV (no need processing the sky and near-ground pixels) 16:9 makes it 364.5 tall, which is not a win!

Then I could go with 640x360. This is a fine resolution (well, the height is still not divisible by 16) -- but what will the hardware do? Is it "the same" as 648x486 regarding CPU efficiency and pixel quality? Will it drop a few pixels from the left/right?
Would 640x368 be better? 640x384?

I'm running at 30 fps.

[6by9]

Specifically, I'm interested in high per-pixel quality (so, hardware binning would be great!) and low latency (one millisecond of extra latency isn't bad; one extra frame of latency is undesirable.)

If you ask for a resolution < 1280x960 (or 1280x720) then on sensor binning will be used to achieve 30fps (it can actually go up to 42fps at 1280x960, or 49fps at 1280x720). 2592x1944 can only be sustained by the sensor at 15fps.
In fact the only sensor modes it will use if you have requested 30fps are
- 1920x1080 cropped FOV (up to 30fps)
- 1296x976 full FOV (up to 42fps)
- 1296x730 full FOV (up to 49fps)
The GPU is always doing a slight resize, which actually generally improves image quality due to the filtering.
You won't add an significant latency within any of those options - you've asked it to run at 30fps, and it should be able to do that fairly happily as the whole pipe is fairly nicely pipelined to run as much as possible in parallel.

Some numbers I'm considering, based on 2592 x 1944 native resolution:
divide by 4 -> 648 x 486 -> this doesn't divide evenly into 16; what will the H.264 encoder do with this?

It'll encode some (black) padding around the image to take it up to an integer number of macroblocks.

I'm more interested in a wide FOV (no need processing the sky and near-ground pixels) 16:9 makes it 364.5 tall, which is not a win!

Then I could go with 640x360. This is a fine resolution (well, the height is still not divisible by 16) -- but what will the hardware do? Is it "the same" as 648x486 regarding CPU efficiency and pixel quality? Will it drop a few pixels from the left/right?
Would 640x368 be better? 640x384?

To be honest it makes almost zero difference from the ARM side - all the heavy lifting is done in the GPU, and generally by dedicated hardware blocks. Big changes in resolution will make some difference to memory bandwidth as there are more pixels floating around, but you have to work hard (or do things very badly) to really be hitting memory bandwidth limits.

If the requested aspect ratio doesn't exactly match the sensor aspect ratio (and all those modes are 4:3 or 16:9), then yes it will determine the crop required on the input side of the ISP, and then run that through the rest of the pipe. As long as you don't end up upscaling, then generally the image quality doesn't suffer at all (downscaling tends to improve noise performance and perceived IQ, so keeping a high sensor resolution with low output resolution gives better and more predictable results than on sensor binning or skipping. There was quite a long thread on here about binning and skipping due to misunderstandings on how it is done when Bayer patterns are involved).

If you're using MMAL via a modified raspivid, then the things to remember is that format->es.video.width must be a multiple of 32, but format->es.video.crop.width can be any multiple of 2. Likewise format->es.video.height must be a multiple of 16, but format->es.video.crop.height can be any multiple of 2. Think of width/height defining the geometry of the buffer, and the crop defines which bit of the buffer to fill. Otherwise the Pi can do almost any resolution and aspect ratio you ask of it.

[jwatte]

If you're using MMAL via a modified raspivid, then the things to remember is that format->es.video.width must be a multiple of 32, but format->es.video.crop.width can be any multiple of 2. Likewise format->es.video.height must be a multiple of 16, but format->es.video.crop.height can be any multiple of 2.

Oh, that's a buffer alignment rule, not a sensor rule? Gotcha! (I was looking for the man pages for mmal and the camera component to figure this out, but sadly was disappointed

It sounds like 648 x 365 would be ideal (it's quarter of 1296x730) when my concern is mainly about the number of pixels I process on the CPU side in my image detection. But that's not even divisible by 2, so I'll play around a bit and see what I end up with.
Thanks for the additional insight!

[jwatte]

Also: When you say those formats are the only ones for 30 fps, what does that actually mean? If I ask for, say, 736x414, or 640x360, will those count as formats that will run at 30 fps, or does it have to be EXACTLY those formats?

[6by9]

Also: When you say those formats are the only ones for 30 fps, what does that actually mean? If I ask for, say, 736x414, or 640x360, will those count as formats that will run at 30 fps, or does it have to be EXACTLY those formats?

I'll refer you to previous comments on mode selection http://www.raspberrypi.org/forums/viewt ... 50#p520132
Frame rate request is in the port format. Ask for whatever you want and the GPU should oblige as long as you stay within the limits.
(A framerate of 0 is a special case that then looks at the values in parameter MMAL_PARAMETER_FPS_RANGE for what range of framerates to allow, with auto gain control therefore having greater flexibility on the exposure time).