6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 9499
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: What features/issues to address?

Sat Aug 02, 2014 9:45 am

jdb wrote:This times ten million. For a given telescope aperture, field of view and object the gains can be easily adjusted to a) not saturate the sensor and b) provide the best SNR - if not through calculation then via successive approximation.

We totally need an ASSUMING DIRECT CONTROL mode for the camera interface. Broadcom may have wizards working in the lens shading and tuning department, but I highly doubt that they ever considered the sensor<->device pairing would be used naked and attached without lens to a prime focus mount of your average garden-variety 10 inch reflector
Being able to control a couple of features of the AGC algo is one thing. Opening up full control of the ISP (eg lens shading etc) would require IP release discussions with Broadcom senior management. Seeing as they are continuing to use the IP in other chips, I doubt they'll be too keen. The support effort required for the "my settings don't work" complaints would be unsustainable too.
For the request to disable the hardware denoise I am looking at a solution that would enable arbitrary disabling of any of the ISP blocks that are configured. Doing that on some blocks would be meaningless (no demosaic?!), but it may provide a way to bypass lens shading as well.
steve_gulick wrote:need a way to bypass (or a pass-thru) of the closed ISP. This would allow the use of sensors other than the OV5647 by I2C programming directly of the sensor's registers (like it is done in most of the other sensors that there are V4L2 drivers available.
This is really needed in the Compute Module if it is going to be sold to do serious industrial machine vision. One camera sensor can't meet all needs - that's why there are so many. The OV5647 has poor low light performance. Some users might need an HDR sensor or one with a global shutter. The closed nature of the ISP has been a real limitation. If, for what ever reason, it has to remain closed, we can do without it - just give us a way around it.
That's all ARM side, so fails my features list criteria of needing to be a firmware thing to be done in this particular time window.
As above, releasing access to the ISP hardware isn't going to happen without lots of negotiation and a probably unmanageable support burden. gsh is on the case for getting the Unicam (receiver peripheral) docs released, but that is his negotiation with Broadcom lawyers.
I will make some "totally unrelated" comments:
  • the same Unicam interface has been used on phones featuring VC4 hardware blocks but driven from the ARM.
  • Kernel drivers would be open source with manufacturers obliged to release them.
  • HTC are naughty by not releasing the kernel source for the Desire 601 Dual Sim - might be worth someone chasing them. (I haven't checked the Samsung Galaxy Y and Wave Y, but they are a bit older, more limited in camera functionality, and Samsung are pretty good on releasing the kernels).
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

User avatar
gagle
Posts: 82
Joined: Fri Feb 14, 2014 6:54 pm
Contact: Website

Re: What features/issues to address?

Sat Aug 02, 2014 10:27 am

6by9 wrote:
gagle wrote:There are a lot of image filters that work with stills but don't work with videos. Is it possible to implement all of them (the ones that are already implemented for stills)?. I could do a list.
AFAIK all the filters/effects via OMX_IMAGEFILTERTYPE work. There was a quirky one with negative that the snapshot image ends up not inverted which I may investigate further if I can find the report.
Please list the ones you think are an issue. I suspect you're meaning DRC which I am highly unlikely to have time to fix (depends if it needs context to work), but one is hardly "a lot", I'm not a mind reader, and I am short on time so I'm not going to investigate vague descriptions. Sorry to be blunt.

Code: Select all

Image effects from RaspiCamControl.c

name           omx name                      raspivid  raspistill  omxcam (video/still)
                                             
negative       OMX_ImageFilterNegative       y         y           y/y
solarise       OMX_ImageFilterSolarize       y         y           y/y
sketch         OMX_ImageFilterSketch         n         y           n/y
denoise        OMX_ImageFilterNoise          ?         y           ?/y
emboss         OMX_ImageFilterEmboss         n         y           n/y
oilpaint       OMX_ImageFilterOilPaint       n         y           n/y
hatch          OMX_ImageFilterHatch          n         y           n/y
gpen           OMX_ImageFilterGpen           n         y           n/y
pastel         OMX_ImageFilterPastel         n         y           n/y
watercolour    OMX_ImageFilterWatercolor     n         y           n/y
film           OMX_ImageFilterFilm           n         y           n/y
blur           OMX_ImageFilterBlur           ?         y           ?/y
saturation     OMX_ImageFilterSaturation     n         n           n/n
colourswap     OMX_ImageFilterColourSwap     y         y           y/y
washedout      OMX_ImageFilterWashedOut      y         y           y/y
posterise      OMX_ImageFilterPosterise      y         y           y/y
colourpoint    OMX_ImageFilterColourPoint    n         n           n/n
colourbalance  OMX_ImageFilterColourBalance  n         n           n/n
cartoon        OMX_ImageFilterCartoon        y         y           y/y

denoise
	OMX_ImageFilterNoise? Changes in size in still, imperceptible changes in video
watercolor
	same as sketch
blur
	imperceptible changes in video
saturation
	can be configured using OMX_IndexConfigCommonSaturation
colourpoint
	imperceptible changes or not implemented
colourbalance
	imperceptible changes or not implemented

Image effects not in RaspiCamControl.c:

OMX_ImageFilterAntialias
OMX_ImageFilterDeRing
OMX_ImageFilterSharpen
OMX_ImageFilterDeInterlaceLineDouble
OMX_ImageFilterDeInterlaceAdvanced
OMX_ImageFilterAnaglyph
OMX_ImageFilterDeInterlaceFast
Commands used:

Code: Select all

/opt/vc/bin/raspivid -o video.h264 -t 2000 -ifx <name>
/opt/vc/bin/raspistill -o pic.jpeg -ifx <name>
OS: Arch linux.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 9499
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: What features/issues to address?

Sat Aug 02, 2014 11:50 am

gagle wrote:

Code: Select all

Image effects from RaspiCamControl.c
name           omx name                      raspivid  raspistill  omxcam (video/still)
                                             
negative       OMX_ImageFilterNegative       y         y           y/y
solarise       OMX_ImageFilterSolarize       y         y           y/y
sketch         OMX_ImageFilterSketch         n         y           n/y
denoise        OMX_ImageFilterNoise          ?         y           ?/y
emboss         OMX_ImageFilterEmboss         n         y           n/y
oilpaint       OMX_ImageFilterOilPaint       n         y           n/y
hatch          OMX_ImageFilterHatch          n         y           n/y
gpen           OMX_ImageFilterGpen           n         y           n/y
pastel         OMX_ImageFilterPastel         n         y           n/y
watercolour    OMX_ImageFilterWatercolor     n         y           n/y
film           OMX_ImageFilterFilm           n         y           n/y
blur           OMX_ImageFilterBlur           ?         y           ?/y
saturation     OMX_ImageFilterSaturation     n         n           n/n
colourswap     OMX_ImageFilterColourSwap     y         y           y/y
washedout      OMX_ImageFilterWashedOut      y         y           y/y
posterise      OMX_ImageFilterPosterise      y         y           y/y
colourpoint    OMX_ImageFilterColourPoint    n         n           n/n
colourbalance  OMX_ImageFilterColourBalance  n         n           n/n
cartoon        OMX_ImageFilterCartoon        y         y           y/y
OK, I can have a look. As I said it'll be that the raspivid use case uses a different image format in memory. It depends on how easy the effect code is to alter the image read from memory. More awkward is likely to be that the video encode format is an interleaved chroma plane, when everything else is two discrete chroma planes.
gagle wrote:denoise
OMX_ImageFilterNoise? Changes in size in still, imperceptible changes in video
watercolor
same as sketch
blur
imperceptible changes in video
saturation
can be configured using OMX_IndexConfigCommonSaturation
colourpoint
imperceptible changes or not implemented
colourbalance
imperceptible changes or not implemented
denoise is unlikely to do much on top of the denoising already done in the camera stack.
Ignore saturation - use OMX_IndexConfigCommonSaturation instead as it is then done in the hardware.
colourpoint and colourbalance take parameters. I can't remember them off the top of my head, and can't remember the defaults either. It's just possible that the effect module isn't included in the build as it was a relatively recent addition.
gagle wrote:Image effects not in RaspiCamControl.c:

OMX_ImageFilterAntialias
OMX_ImageFilterDeRing
OMX_ImageFilterSharpen
OMX_ImageFilterDeInterlaceLineDouble
OMX_ImageFilterDeInterlaceAdvanced
OMX_ImageFilterAnaglyph
OMX_ImageFilterDeInterlaceFast[/code]
AntiAlias and DeRing aren't implemented. Sharpen there is a dedicated control for that again does the processing in hardware.
The flavours of deinterlace is only applicable for deinterlacing interlaced video, but that camera isn't interlaced so why would raspicam reference them?
Anaglph?! Never seen that one before. Stereoscopic effect, and we haven't got stereo support in yet.

NB We're supporting MMAL. If the enum isn't there, then it isn't supported. IL is an open spec so Khronos et al may well have defined enums we don't support - get over it.
There have actually been comments about deprecating IL in favour of MMAL at some point once all the codecs stuff is done via MMAL.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

seumas
Posts: 4
Joined: Wed Dec 04, 2013 11:08 pm

Re: What features/issues to address?

Sat Aug 02, 2014 9:28 pm

I'd love to be able to drop text and/or image overlays into the camera video pipeline before the encoder, so here's a vote for that :D

BoehserWolf
Posts: 44
Joined: Thu Jun 12, 2014 2:53 pm

Re: What features/issues to address?

Sun Aug 03, 2014 3:44 pm

I'm a bit late on the party but my vote goes to text overlay. Can't think of the right place - maybe right before the decoder's?

ethanol100
Posts: 650
Joined: Wed Oct 02, 2013 12:28 pm

Re: What features/issues to address?

Mon Aug 04, 2014 9:05 am

With the firmware https://github.com/Hexxeh/rpi-firmware/ ... 25a3e8e069 a bug has been introduced(reported by gordon77 here: http://www.raspberrypi.org/forums/viewt ... 19#p591419). If we request a resized resolution it takes much longer then the non resized one.
i.e. running "time raspiyuv -o /dev/null" needs 0m5.959s
with the older firmware "time raspiyuv -o /dev/null -w 640 -h 480" 0m5.889s
and with the new firmware "time raspiyuv -o /dev/null -w 640 -h 480" 0m8.024s.

The resizer creates a delay of ~2sec. (Same for raspistill, so guessing something is wrong with the resizer or the wrong mode is choosen?)

Would be nice if you could fix this.

gordon77
Posts: 5167
Joined: Sun Aug 05, 2012 3:12 pm

Re: What features/issues to address?

Mon Aug 04, 2014 10:05 am

Is there anyway to make raspistill capture pictures quicker, without having to wait for the settling time every time ?

I use the camera mainly in the manual settings so it doesn't need auto exposure, auto AWB etc but it still takes time to settle, can this be elimnated ? Maybe by starting the camera at the start and then taking shots without needing the settling time ?

Picamera achieves it by capturing stills from video but I haven't mastered how to set the camera parameters through it, or run it as a subprocess.

gordon77

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 9499
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: What features/issues to address?

Mon Aug 04, 2014 10:22 am

ethanol100 wrote:With the firmware https://github.com/Hexxeh/rpi-firmware/ ... 25a3e8e069 a bug has been introduced(reported by gordon77 here: http://www.raspberrypi.org/forums/viewt ... 19#p591419). If we request a resized resolution it takes much longer then the non resized one.
i.e. running "time raspiyuv -o /dev/null" needs 0m5.959s
with the older firmware "time raspiyuv -o /dev/null -w 640 -h 480" 0m5.889s
and with the new firmware "time raspiyuv -o /dev/null -w 640 -h 480" 0m8.024s.

The resizer creates a delay of ~2sec. (Same for raspistill, so guessing something is wrong with the resizer or the wrong mode is choosen?)

Would be nice if you could fix this.
Fixed. It was dropping into the low framerate / long exposures mode. Sorry for that one.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

gordon77
Posts: 5167
Joined: Sun Aug 05, 2012 3:12 pm

Re: What features/issues to address?

Mon Aug 04, 2014 10:26 am

Thanks for the quick service :D

gordon77

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 9499
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: What features/issues to address?

Mon Aug 04, 2014 10:27 am

gordon77 wrote:Is there anyway to make raspistill capture pictures quicker, without having to wait for the settling time every time ?

I use the camera mainly in the manual settings so it doesn't need auto exposure, auto AWB etc but it still takes time to settle, can this be elimnated ? Maybe by starting the camera at the start and then taking shots without needing the settling time ?

Picamera achieves it by capturing stills from video but I haven't mastered how to set the camera parameters through it, or run it as a subprocess.
raspistill is demo code, and ARM side code, so I won't be looking at it.
The -t option should give you the flexibility you need anyway, but you really need to set all the parameters (-ss, -ISO, -awb off, -awbg <r_gain> <b_gain>) to avoid getting odd captures.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

gordon77
Posts: 5167
Joined: Sun Aug 05, 2012 3:12 pm

Re: What features/issues to address?

Mon Aug 04, 2014 12:14 pm

Thanks for the answer.

I've tried all options and can't get less than 700mS between images, which seems slow when the camera can do 30/60fps.

gordon77

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 9499
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: What features/issues to address?

Mon Aug 04, 2014 1:18 pm

gordon77 wrote:Thanks for the answer.

I've tried all options and can't get less than 700mS between images, which seems slow when the camera can do 30/60fps.
You've changed your query from startup to between shots!

Shot to shot time is high because the sensor is currently switching back to preview between each and every one. There is a parameter MMAL_PARAMETER_CAMERA_BURST_CAPTURE that tells the camera not to do that, and pull request https://github.com/raspberrypi/userland/pull/191 is integrating exactly that. The downside is that currently AGC and AWB don't run when that flag is set (see item 1 on my list), so people complain that their images go dark (actually the scene goes dark and the camera doesn't adapt).
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

gordon77
Posts: 5167
Joined: Sun Aug 05, 2012 3:12 pm

Re: What features/issues to address?

Mon Aug 04, 2014 1:39 pm

Thanks, and sorry for the poor explanation in the first place.

Sounds like its ongoing work :D

gordon77

Momoko_Fan
Posts: 1
Joined: Wed Apr 09, 2014 3:59 pm

Re: What features/issues to address?

Mon Aug 04, 2014 3:40 pm

Just gonna mention Periodic Intra Refresh again, in case anything changed or became possible since April.
One of the modes in MMAL_VIDEO_INTRA_REFRESH_T, maybe MMAL_VIDEO_INTRA_REFRESH_CYCLIC or MMAL_VIDEO_INTRA_REFRESH_PSEUDO_RAND. Could help with achieving true CBR and interference mitigation if the video is transmitted over wireless or networks where packet loss is possible.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 9499
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: What features/issues to address?

Mon Aug 04, 2014 6:09 pm

Some progress today.
- Fixed the bug that was selecting the wrong (long exposures) mode on stills captures
- Sorted the default digital gain so that exposure mode can be set to off before starting the camera. It'll just adopt dig gain x1.0
- Pulling in code from other branches to do the basic text overlay. It's no the solution I'd like, but it's the only one that has a chance of working with stills, and it's the easier option.
- Abortive investigations of the lens shading issue. Trying to pull in the latest code is a nightmare (about 18months of branch divergence!), so I'm going to have to attempt to pick the changes apart and manually merge :(
I won't ask Dom to do a release yet - I'll see what progress gets made tomorrow and probably ask him at end of play then.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

User avatar
waveform80
Posts: 359
Joined: Mon Sep 23, 2013 1:28 pm
Location: Manchester, UK
Contact: Website Twitter

Re: What features/issues to address?

Tue Aug 05, 2014 12:35 am

6by9 wrote: ...
Almost already done.
I'd exposed the parameter a few weeks back (I wondered if people would notice!), but it requires the flash driver I've mentioned above to actually do anything to GPIO lines.
I noticed ;) But then I tend to keep an eye on commits in the interface/mmal branch of the raspberrypi/userland tree!

Anyway, looks like I'm going to have my work cut out adding bits to picamera over the next few weeks ... and just when I thought I was beginning to catch up to you guys!

On the subject of effects not working in video recording where they do for stills, I haven't tested this in ages (probably near the first few versions of picamera) but I did document that only a couple of effects worked in video recording. From the other comments in this thread it sounds like either I missed a few, or they've started working since that test.

If you've got the time to look into them, I'd be most grateful if you could have a look at the couple of issues I tagged "upstream" in picamera's bug repo (but please don't prioritize them over any of the existing requests - neither of them are for new functionality, so they're really not important):
  • Split recording fails when bitrate is set to 0 (sps/pps headers aren't emitted before I-frames with bitrate 0 - also, bitrate 0 and quantization 0 both seem to be "special" settings)
  • MJPEG recording seems to have some issues (which ultimately lock up the camera) at resolutions >640x480
  • There also seems to be an issue with burst mode in that some frames intermittently come back black (or severely underexposed); different to the fade-to-black issue which, as previously discussed, seems to be sorted. I've yet to come up with a consistent test case for this one other than to note that removing all delays between captures seems to maximize the effect. I'll try and write up a proper ticket for it this week.
Speaking of new functionality, text overlays is something picamera users have been asking me about for a good while now. I was hoping it might be possible to implement something via OpenGL, but given my complete lack of knowledge in the area it was going to involve a long delay while I learned about it! If there's a possibility of including some (even rudimentary) via MMAL, that'd probably be a much easier solution.

Anyway, many thanks for all the work!

Dave.
Author of / contributor to a few pi related things (picamera, Sense HAT emulator, gpio-zero, piwheels, etc.), and currently a software engineer at Canonical responsible for Ubuntu Server and Core on the Raspberry Pi.

ethanol100
Posts: 650
Joined: Wed Oct 02, 2013 12:28 pm

Re: What features/issues to address?

Tue Aug 05, 2014 6:38 am

waveform80 wrote: ...
There also seems to be an issue with burst mode in that some frames intermittently come back black (or severely underexposed); different to the fade-to-black issue which, as previously discussed, seems to be sorted. I've yet to come up with a consistent test case for this one other than to note that removing all delays between captures seems to maximize the effect. I'll try and write up a proper ticket for it this week.
...
Yes I noticee that too, but we can increase the capture rate to ~4fps for full frame stills wich is better then the ~1fps we get without the burst mode. So setting at least a delay of 250 seems to work for me.

shuckle
Posts: 565
Joined: Sun Aug 26, 2012 11:49 am
Location: Finland

Re: What features/issues to address?

Tue Aug 05, 2014 6:54 am

waveform80 wrote:
I vote for this as I suffer it badly (http://www.raspberrypi.org/forums/viewt ... 43&t=77893). And I am not alone as you can see from http://www.raspberrypi.org/forums/viewt ... 43&t=83570

I have test setup if needed.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 9499
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: What features/issues to address?

Tue Aug 05, 2014 8:42 am

waveform80 wrote:Anyway, looks like I'm going to have my work cut out adding bits to picamera over the next few weeks ... and just when I thought I was beginning to catch up to you guys!
You only have to worry for a couple of weeks, then you'll have loads of time with few new features! Is that a good or bad thing?!
waveform80 wrote:Split recording fails when bitrate is set to 0 (sps/pps headers aren't emitted before I-frames with bitrate 0 - also, bitrate 0 and quantization 0 both seem to be "special" settings)
MJPEG recording seems to have some issues
OK, I'll add them to the list of things.
A quick check of the component source for video_encode:
OMX_IndexParamPortDefinition: 201
Query / set the output format. \code{eCompressionFormat} specifies
the format to which to encode. \code{nBitrate} specifies the target
bit rate (unless overridden by \code{OMX_IndexParamVideoBitrate}). The
output will either use VBR, or no rate control at all if the bitrate
is zero.

OMX_IndexParamVideoBitrate: 201
Query / set the output bitrate type and target bitrate. Overrides
any setting made via the port definition. Can only be set when in LOADED.

OMX_IndexConfigVideoBitrate: 201
Allows changing the target bitrate while the component is EXECUTING. Only
valid if rate control is enabled (see \code{OMX_IndexParamVideoBitrate}).
So bitrate=0 means no rate control, but I'm not sure what that means in reality within the codec. It is the codec code itself that does the duplication of the header bytes, and that is an area I have never really picked through.
We also have
OMX_IndexParamBrcmVideoEncodeMinQuant: 201
Query / set the codec's minimum quantization level.

OMX_IndexParamBrcmVideoEncodeMaxQuant: 201
Query / set the codec's maximum quantization level.

OMX_IndexParamBrcmVideoInitialQuant: 201
Query / set the codec's initial quantization level.

OMX_IndexParamBrcmVideoRCSliceDQuant: 201
Query / set the codec's Slice DQuant level
which I guess determine the compression level when bitrate=0, but I don't know for sure. Which quantisation parameter are you meaning with your comment?

MJPEG is a pain. It ideally wants to be writing to a circular buffer, but currently can't cope with wrapping around the end, mainly because the code to rewrite headers into either MJPEG A or B formats can't handle it. If I can juggle it, I may just see if I can increase the size of the buffer - not great, but may be a suitable quick fix.

Burst mode: I'll pick up ethanol100's latest changes and see what is going on. I'm guessing that the sensor is still being stopped between shots, in which case it may be picking up the corrupt frame on restart.

Overlays: Hopefully something basic and a bit hacky today that will allow overlaying one text string (and some debug functions - you may as well have them seeing as the code is written! Graphing of analog gain, exposure time, etc over time anyone?)
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 9499
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: What features/issues to address?

Tue Aug 05, 2014 3:22 pm

Some changes pushed today.
- MJPEG codec and framework modified to allocate a 4MB output FIFO. Hacky solution, but it's quicker than actually fixing the issue of buffering in the codec.

- New (to RPi) annotate stage added. New parameter MMAL_PARAMETER_ANNOTATE taking a MMAL_PARAMETER_CAMERA_ANNOTATE_T (userland repo update should follow the firmware release)

Code: Select all

#define MMAL_CAMERA_ANNOTATE_MAX_TEXT_LEN 32
typedef struct MMAL_PARAMETER_CAMERA_ANNOTATE_T
{
   MMAL_PARAMETER_HEADER_T hdr;

   MMAL_BOOL_T enable;
   char text[MMAL_CAMERA_ANNOTATE_MAX_TEXT_LEN];
   MMAL_BOOL_T show_shutter;
   MMAL_BOOL_T show_analog_gain;
   MMAL_BOOL_T show_lens;
   MMAL_BOOL_T show_caf;
   MMAL_BOOL_T show_motion;
} MMAL_PARAMETER_CAMERA_ANNOTATE_T;
All the "show" flags may be of some interest, but not much use. If you enable it and add some text, then it will add it centred at the top of the image as white text. This is the quick and dirty solution. I'm still hoping to get a fully fledged overlay component sorted, so please don't go asking for lots of bells and whistles on this.

- Inline headers mode for segmentation and VBR: queried with the codecs guys. Had some pointers and it seems to work, so that's been pushed too.

- Effects. All the effects appear to be built in. I'll make a separate post on this thread with effects settings because it'll be a little long. I suspect that the default parameters for colourpoint and colourbalance aren't very strong effects. I've lost track of what has and has not been released for documentation. Note MMAL_PARAMETER_IMAGEFX_PARAMETERS_T which sets the effect and associated parameters.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 9499
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: What features/issues to address?

Tue Aug 05, 2014 3:25 pm

Documenting the parameters that the image effects can take. Where an effect is not mentioned, then it doesn't support any parameters.

Saturation and denoise will NOT be supported. Please use the appropriate MMAL and IL parameters for directly controlling them in the ISP.

solarise - 1, 4, or 5 parameters. "<yuv> <x0> <y0> <y1> <y2>". <yuv> processes as YUV data if set. "Input values from 0 up to x0 (actually x0-1) are remapped linearly onto the range 0 to y0. Input values from x0 up to 255 are remapped linearly from y1 to y2.". <yuv> can be omitted (default 0) for 4 parameters. Or only setting one parameter sets the YUV mode, but uses default values for x0, y0, y1, y2 (128,128,128,0).

colourpoint - Retains chroma in only one quadrant of the U/V space. 1 parameter - which quadrant of the YUV space to retain. 0 - green, 1 - red/yellow, 2 - blue, 3 - purple.

colourbalance - Affect RGB colour balance, changing each channel by a given percentage. Up to 6 parameters:
/* parameters are at most 6 numbers: 1 unsigned integer, 3 unsigned integers
* and 2 signed integers.
*
* First is the lens shading strength, in 8.8 format. 108 is default setting
* 0 means lens shading has no effect. This parameter is not optional.
*
* Next three are multipliers of red, green & blue in 8.8 format.
* "256 256 256" would be unchanged.
*
* Last two are constant additions to u and v, which are added after
* applying the three channel factors. These are optional and can be
* skipped out from the parameter string, they will then assumed to be
* zero.
*
* If only three numbers are given then the green factor is assumed to
* be zero.
*/
colourswap - Swaps RGB to GBR or BRG. 1 parameter is the order (0 = to GBR, 1 = to BRG).

posterise - Reduces the quantisation steps on the image. 1 parameter controls number of steps (range 2-32). Default is 4.

blur - 1 parameter setting the size of the kernel. Range 1-2.

film - 3 parameters. <strength> <u> <v>. All three are 0-255.

watercolour - 0 or 2 parameters. If 2, then sets U&V values.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

User avatar
waveform80
Posts: 359
Joined: Mon Sep 23, 2013 1:28 pm
Location: Manchester, UK
Contact: Website Twitter

Re: What features/issues to address?

Tue Aug 05, 2014 11:13 pm

6by9 wrote:A quick update.

Changes pushed internally to add the MMAL parameters to control the software denoise. I can't remember the two new enums off the top of my head, but they're something like MMAL_PARAMETER_VIDEO_DENOISE_DISABLE, and MMAL_PARAMETER_STILLS_DENOISE_DISABLE. Both will just take a MMAL_PARAMETER_BOOLEAN_TYPE with true DISABLING the algorithms (I was obviously being lazy when the original code was written and didn't want to initialise an enable to true or something.
As suspected, disabling hardware denoise is a bit trickier. I have asked questions of the relevant team, but a lot of them aren't in for various reasons.
Quick query regarding this: I'm just experimenting with adding this to picamera as a r/w property. When I query MMAL_PARAMETER_VIDEO_DENOISE_DISABLE or MMAL_PARAMETER_STILLS_DENOISE_DISABLE on the camera's control port (is this the right port to be querying?), or the corresponding output port (stills for STILLS_DENOISE, video for VIDEO_DENOISE), both come back true on a newly initialized camera instance. In other words, both come back already disabled ... I would've thought the default would be false (both enabled)? Incidentally, this is with firmware 701. It seems happy for me to query or set the parameters on either the control port or the stills/video output ports (there seems no difference between them; e.g. setting the value on the control port modifies it on the stills & video ports, and vice versa).


Dave.
Author of / contributor to a few pi related things (picamera, Sense HAT emulator, gpio-zero, piwheels, etc.), and currently a software engineer at Canonical responsible for Ubuntu Server and Core on the Raspberry Pi.

User avatar
jbeale
Posts: 3707
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: What features/issues to address?

Tue Aug 05, 2014 11:21 pm

6by9 wrote:- New (to RPi) annotate stage added. New parameter MMAL_PARAMETER_ANNOTATE taking a MMAL_PARAMETER_CAMERA_ANNOTATE_T (userland repo update should follow the firmware release)
Text annotation is a very interesting feature for me. Does this work with video output, and permit an updated line of text on each frame? In other words (with some additional userspace coding) would the annotate stage enable burned-in timecode that uniquely identifies every video frame? That would be a very nice capability.

User avatar
waveform80
Posts: 359
Joined: Mon Sep 23, 2013 1:28 pm
Location: Manchester, UK
Contact: Website Twitter

Re: What features/issues to address?

Tue Aug 05, 2014 11:39 pm

waveform80 wrote:
6by9 wrote:A quick update.

Changes pushed internally to add the MMAL parameters to control the software denoise. I can't remember the two new enums off the top of my head, but they're something like MMAL_PARAMETER_VIDEO_DENOISE_DISABLE, and MMAL_PARAMETER_STILLS_DENOISE_DISABLE. Both will just take a MMAL_PARAMETER_BOOLEAN_TYPE with true DISABLING the algorithms (I was obviously being lazy when the original code was written and didn't want to initialise an enable to true or something.
As suspected, disabling hardware denoise is a bit trickier. I have asked questions of the relevant team, but a lot of them aren't in for various reasons.
Quick query regarding this: I'm just experimenting with adding this to picamera as a r/w property. When I query MMAL_PARAMETER_VIDEO_DENOISE_DISABLE or MMAL_PARAMETER_STILLS_DENOISE_DISABLE on the camera's control port (is this the right port to be querying?), or the corresponding output port (stills for STILLS_DENOISE, video for VIDEO_DENOISE), both come back true on a newly initialized camera instance. In other words, both come back already disabled ... I would've thought the default would be false (both enabled)? Incidentally, this is with firmware 701. It seems happy for me to query or set the parameters on either the control port or the stills/video output ports (there seems no difference between them; e.g. setting the value on the control port modifies it on the stills & video ports, and vice versa).


Dave.
Quick update on this: I'm reasonably certain the initial reading of true (indicating that denoise is disabled) is a bug. The video denoise setting can be observed while running the preview; toggling it on and off results in a noticeable change in the picture quality (at least under artificial light in dim conditions; in daylight I suspect it'll be less noticeable). From this I can tell that denoise is definitely enabled when the camera is first initialized, but for some reason the attribute returns the opposite until it's been set explicitly (once it's been set explicitly it all seems to operate as expected).


Dave.
Author of / contributor to a few pi related things (picamera, Sense HAT emulator, gpio-zero, piwheels, etc.), and currently a software engineer at Canonical responsible for Ubuntu Server and Core on the Raspberry Pi.

User avatar
waveform80
Posts: 359
Joined: Mon Sep 23, 2013 1:28 pm
Location: Manchester, UK
Contact: Website Twitter

Re: What features/issues to address?

Tue Aug 05, 2014 11:55 pm

waveform80 wrote:Quick update on this: I'm reasonably certain the initial reading of true (indicating that denoise is disabled) is a bug. The video denoise setting can be observed while running the preview; toggling it on and off results in a noticeable change in the picture quality (at least under artificial light in dim conditions; in daylight I suspect it'll be less noticeable). From this I can tell that denoise is definitely enabled when the camera is first initialized, but for some reason the attribute returns the opposite until it's been set explicitly (once it's been set explicitly it all seems to operate as expected).


Dave.
Eurgh - correction to the above (and a lesson for others: don't try and think about double negatives at a quarter to 1 in the morning...)! Denoise is definitely enabled on initialization, and the operation of the setting is sort-of-correct (it *doesn't* return the wrong value until set explicitly, it returns the correct setting at all times). However, the sense of the setting is backward: instead of MMAL_PARAMETER_VIDEO_DENOISE_DISABLED it should be MMAL_PARAMETER_VIDEO_DENOISE_ENABLED; I flipped the meaning of the setting in the new picamera attribute and then got horribly confused when setting video_denoise to True made the image more noisy! In other words, you might've *tried* to be lazy when labelled it DISABLED, but you actually did the right thing (but with the wrong name! ;)

Sorry for the confusion!


Dave.
Author of / contributor to a few pi related things (picamera, Sense HAT emulator, gpio-zero, piwheels, etc.), and currently a software engineer at Canonical responsible for Ubuntu Server and Core on the Raspberry Pi.

Return to “Camera board”