palavrov
Posts: 24
Joined: Wed Jul 11, 2018 12:11 pm

How to tunnel camera to video splitter to image encode?

Mon Aug 27, 2018 10:56 am

Hi,

Our OpenMAX based application need to take few still jpeg pictures at 8MP as fast as possible. The shortest time that I got for two jpegs in sequence is about 700ms between each jpeg which is too much and I want to use the picamera approach taking video frames instead of still images. How can I pipeline the IL components to connect camera->video_splitter->image_encoder? In the documentation of the video splitter it is written that:
When using proprietary communication on the input port, a single still image can be captured at one of the image output ports. The image is supplied as stripes via buffers on the output port. The image will be captured every time the port transitions from IDLE to EXECUTING.

But when I try to tunnel between video splitter output port 251 to image encoder input port 340 got error that these two ports are not compatible.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5549
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: How to tunnel camera to video splitter to image encode?

Tue Aug 28, 2018 10:54 am

Set OMX_IndexParamBrcmDisableProprietaryTunnels on one of the ports.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

palavrov
Posts: 24
Joined: Wed Jul 11, 2018 12:11 pm

Re: How to tunnel camera to video splitter to image encode?

Tue Sep 04, 2018 1:08 pm

Seems that need something else or doing something wrong but didn't get over that error.

I tried to set OMX_IndexParamBrcmDisableProprietaryTunnels with all combinations OMX_TRUE and OMX_FALSE on ports 71, 250, 251 on the relevant components without success.

Something else that catch my attention was that when configure the tunnel between 71 and 250 got OMX_EventPortSettingsChanged on all output ports of the splitter (251,252,253,254). Do I need to follow special order in order to configure these tunnels?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5549
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: How to tunnel camera to video splitter to image encode?

Tue Sep 04, 2018 2:31 pm

OMX_IndexParamBrcmDisableProprietaryTunnels on either the video_splitter output port in question, or the image_encode input port.

On configuring the input port to the splitter you get an OMX_EventPortSettingsChanged on all output ports as it has updated the output resolutions to match (IIRC it does not change the colour format should it be set). video_splitter does not support resizing, therefore setting the input port first is really a prerequisite.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

palavrov
Posts: 24
Joined: Wed Jul 11, 2018 12:11 pm

Re: How to tunnel camera to video splitter to image encode?

Tue Sep 04, 2018 5:35 pm

Set OMX_IndexParamBrcmDisableProprietaryTunnels to TRUE and then to FALSE on both image_encoder input port and video_splitter output port but still got the same OMX_ErrorPortsNotCompatible error.

Is it necessary to configure any other parameters on the splitter to make it passing the still image to the image_encoder?

For the moment the only configuration that I do is to set this OMX_IndexParamBrcmDisableProprietaryTunnels for the output port ...

palavrov
Posts: 24
Joined: Wed Jul 11, 2018 12:11 pm

Re: How to tunnel camera to video splitter to image encode?

Mon Sep 10, 2018 8:08 am

Got back to this issue and still don't have any progress :(

After rereading the documentation think that this could help give light:
When using proprietary communication on the input port, a single still image can be captured at one of the image output ports.

So for the input port 250 we have:
OMX_IndexConfigBrcmUseProprietaryCallback Forces use of proprietary tunnelling on the input port.

and
OMX_IndexParamCameraPoolToEncoderFunction Used to set up proprietary communication on an input port.

which seems like something that worth to be tried as well. The problem is that corresponding data structures are not defined nor documented i.e.
OMX_CONFIG_BRCMUSEPROPRIETARYTUNNELTYPE

and
OMX_PARAM_CAMERAPOOLTOENCODERFUNCTIONTYPE

BTW trying to configure OMX_IndexParamBrcmDisableProprietaryTunnels on image_encode input port fails with bad index error

Is there any working example how to tunnel video_splitter to image_encoder with openmax?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5549
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: How to tunnel camera to video splitter to image encode?

Mon Sep 10, 2018 8:47 am

Do not try and use OMX_IndexConfigBrcmUseProprietaryCallback or OMX_IndexParamCameraPoolToEncoderFunction directly - they are used by the MMAL framework to reuse the lowest level of the IL components in a way that supports an optimised zero copy behaviour. OMX_IndexParamCameraPoolToEncoderFunction in particular is passing function pointers in the VPU's address space around which will mean absolutely nothing to the ARM.

Sorry, I've been confusing myself with how MMAL works.

As per the IL spec section 3.2.3.6, when you call OMX_SetupTunnel the components will try to negotiate a format that both ends can support. That can include trying to use proprietary mechanisms. image_encode and video_splitter support slight variations on proprietary tunnelling, and therefore can not enable it anyway. They should fall back to trying to agree the PortDefinition based on the port format of the output port on the source component.

Can you provide the port definition that you are setting on the video_splitter output port please? You can also try manually setting it on the image_encode input port and I suspect it will fail, which would explain why the overall tunnel setup fails.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

palavrov
Posts: 24
Joined: Wed Jul 11, 2018 12:11 pm

Re: How to tunnel camera to video splitter to image encode?

Mon Sep 10, 2018 9:48 am

I'm not explicitly configuring port definitions on the video_splitter i.e. expected that they will be taken from these on the camera output port - which could be a problem so will try to configure both output ports with the same definitions.

palavrov
Posts: 24
Joined: Wed Jul 11, 2018 12:11 pm

Re: How to tunnel camera to video splitter to image encode?

Mon Sep 10, 2018 6:07 pm

This is the content of port definitions for the video_splitter output port:

Code: Select all

nSize: 96
nPortIndex: 251
eDir: output
nBufferCountActual: 1
nBufferCountMin: 1
nBufferSize: 79104
bEnabled: false
bPopulated: false
eDomain: OMX_PortDomainVideo
  format:
    cMIMEType: (null)
    nFrameWidth: 3280
    nFrameHeight: 2464
    nStride: 3296
    nSliceHeight: 16
    nBitrate: 0
    xFramerate: 0
    bFlagErrorConcealment: false
    eCompressionFormat: OMX_VIDEO_CodingUnused
    eColorFormat: OMX_COLOR_FormatYUV420PackedPlanar
bBuffersContiguous: false
nBufferAlignment: 16

The version with still images was rotating the picture by 90 degrees at the camera so I removed the rotation for the video frames but the tunnel call still report that ports are not compatible.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5549
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: How to tunnel camera to video splitter to image encode?

Mon Sep 10, 2018 8:49 pm

It may or may not be the issue (no source to hand to check), but setting an nSliceHeight of 16 and 1 buffer is not the most efficient way to transfer an 8MPix image - that's 154 round trips of the single buffer to transfer the image, with no possibility of pipelining as there are no buffers to permit it.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

palavrov
Posts: 24
Joined: Wed Jul 11, 2018 12:11 pm

Re: How to tunnel camera to video splitter to image encode?

Tue Sep 11, 2018 7:15 am

I didn't touch the nSliceHeight or something buffer related at all - all the values are the default ones. Which values do you recommend to set? My goal is to get two shoots as fast as possible and every optimization is more than welcome :)

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5549
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: How to tunnel camera to video splitter to image encode?

Tue Sep 11, 2018 9:32 am

If you're not memory constrained then set nSliceHeight to nFrameHeight rounded up to a multiple of 16 (2464 already is a multiple of 16).
Using slices is a memory optimsation step at the expense of latency (which can be partially mitigated with more buffers). JPEG encoding is based on 16x16 macroblocks, therefore lends itself to encoding in stripes.
If trying to get every last ms out then it is just possible that half a dozen 16-line high stripe buffers is faster than full frame - full frame has to wait for the whole frame to be produced before starting encoding, but stripes have more setup overhead so the hardware is having to start and stop more frequently. I couldn't say which is more costly, and we're probably only talking 1-2ms.

Sorry, getting IL code up and running is a pain (which is why we typically recommend MMAL instead), so I'm not going to try creating a test case. If you can post a simple test case somewhere then I will have a look, but otherwise it's pretty tough to provide guidance.

I assume you have successfully tried tunneling camera port 71 to image_encode 340.
I'll also say that running the video port at the full 8MPix is not really a normal use case. Image buffers default to assuming a max of 1080P for preview or video frames.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

palavrov
Posts: 24
Joined: Wed Jul 11, 2018 12:11 pm

Re: How to tunnel camera to video splitter to image encode?

Tue Sep 11, 2018 6:58 pm

Setting nSliceHeight to nFrameHeight didn't improve the shooting time in my tests - probably the impact is too small to be correctly measured.

Here is my test code - hope to be easy to read and self documented.

https://github.com/palavrov/camera-app

palavrov
Posts: 24
Joined: Wed Jul 11, 2018 12:11 pm

Re: How to tunnel camera to video splitter to image encode?

Wed Sep 12, 2018 8:56 pm

There were some problems with the Makefile - fixed - now the project builds on latest raspbian and reports OMX_ErrorPortsNotCompatible that we need help to be fixed.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5549
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: How to tunnel camera to video splitter to image encode?

Thu Sep 13, 2018 11:04 am

Now I have a test case I can investigate.

image_encode only supported image domain configuration on the input port. video_splitter uses the video domain on the output port. Most components accept either image or video domain on their input ports, but for some reason not image_encode. I've added that support now and will throw up a test firmware image once I've pushed the changes for review.
(MMAL, as used by PiCamera, avoids this as it wraps them both into a single video domain).

Your app then stalls as you enable the output of video_splitter and waits for that to complete before enabling the image_encode input port. That will fail as the port enable will wait for buffers to be allocated.
Please look at the ilclient library and the hello_pi hello_video example for the expected behaviour under IL. Sorry, I haven't got the time to debug third party apps in detail.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5549
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: How to tunnel camera to video splitter to image encode?

Thu Sep 13, 2018 1:57 pm

Test firmware at https://drive.google.com/file/d/1L7yLeT ... sp=sharing
It's only the start_x variant, so please ensure you only have start_x=1, and not start_debug=1, in /boot/config.txt
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

palavrov
Posts: 24
Joined: Wed Jul 11, 2018 12:11 pm

Re: How to tunnel camera to video splitter to image encode?

Fri Sep 14, 2018 5:21 pm

Thanks @6by9 - the test firmware pass the tunnel setup and indeed our application stall in the wait of the port enable.

I was wondering if there is a way to know which ports need a preallocated buffers and for which the omx infrastructure will take care - that way will handle all the ports in the same way?

palavrov
Posts: 24
Joined: Wed Jul 11, 2018 12:11 pm

Re: How to tunnel camera to video splitter to image encode?

Fri Sep 14, 2018 6:49 pm

I'm puzzled ... allocating buffers for port 251 didn't succeed but fail with OMX_ErrorIncorrectStateOperation

As suggested I'm analyzing the source of libilclient but can't get what you are pointing at. Could you please explain your idea and give me some hints?

palavrov
Posts: 24
Joined: Wed Jul 11, 2018 12:11 pm

Re: How to tunnel camera to video splitter to image encode?

Fri Sep 14, 2018 7:07 pm

Hmmm ... just a random cowboy shot in the dark go trough the wait i.e. I first enabled both 251 & 340 and just after that wait for both port enabled events. Is it requirement to open both port of one tunnel like that?

palavrov
Posts: 24
Joined: Wed Jul 11, 2018 12:11 pm

Re: How to tunnel camera to video splitter to image encode?

Fri Sep 14, 2018 7:21 pm

Got it ... this is the way implemented in ilclient_enable_tunnel() as well.

Now got to the next stall - waiting for the output buffer to be populated with the encoded jpeg. That was expected because that part of the code is from the still shots version and reading again the documentation of image_encoder wonder what means this
The image will be captured every time the port transitions from IDLE to EXECUTING.
i.e. is the image captured when:
  • the component state transition from IDLE to EXECUTING?
  • the port transition from DISABLED to ENABLED?
  • something else?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5549
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: How to tunnel camera to video splitter to image encode?

Fri Sep 14, 2018 8:44 pm

TBH I don't know who wrote that bit of text, but the image_encode component should create a new encoded frame for each complete frame passed in, no state transition or port state change required.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

palavrov
Posts: 24
Joined: Wed Jul 11, 2018 12:11 pm

Re: How to tunnel camera to video splitter to image encode?

Sat Sep 15, 2018 12:08 pm

Finally got it working but with few quirks:

  • camera preview port 70 must have the same frame width and height as these for port 71 - otherwise there is noting out from the image_encoder
  • nStride parameter must be rounded to 32 not to 16 as suggested - otherwise setting OMX_IndexParamPortDefinition fails with OMX_ErrorBadParameter
  • Every resolution bigger than FullHD (1920/1080) doesn't work i.e. again there is nothing out from the image_encoder. Actually this is not quiet true because 1984/1080 and 1920/1136 works too i.e. this seems like limitation of the frame size around 2142720 or 2181120 pixels? Also it works with 2208/992 i.e. 2190336 pixels ...
Giving more memory to the VPU didn't help.

My goal is to take 2 x 8MP shots as fast as possible and need to go trough this FullHD resolution limit - would you please give me some hints about it?
Last edited by palavrov on Sat Sep 15, 2018 3:36 pm, edited 1 time in total.

palavrov
Posts: 24
Joined: Wed Jul 11, 2018 12:11 pm

Re: How to tunnel camera to video splitter to image encode?

Sat Sep 15, 2018 1:29 pm

Based on your previous remarks about buffers size optimized for 1080p video/preview could this cause the problem?
If so do I need to increase the buffers on all ports on this pipeline or it could be enough to increase just for single component?

palavrov
Posts: 24
Joined: Wed Jul 11, 2018 12:11 pm

Re: How to tunnel camera to video splitter to image encode?

Sat Sep 15, 2018 3:35 pm

Seems that got over this as well by configuring camera ports 70 & 71 with OMX_IndexParamPortMaxFrameSize to the desired one.

Now the next issue is to distinguish the frames in the output stream ... I'm receiving countless events on the .FillBufferDone callback but never OMX_EventBufferFlag event which was the case when to pipe was configured for still images on the image_encoder component output port.

palavrov
Posts: 24
Joined: Wed Jul 11, 2018 12:11 pm

Re: How to tunnel camera to video splitter to image encode?

Sat Sep 15, 2018 5:43 pm

Configuring OMX_IndexConfigSingleStep on the video_splitter output port solve the problem very well and as bonus now the missing OMX_EventBufferFlag events come too.

Next issue - how to rotate the frame on 90 degrees - setting OMX_IndexConfigCommonRotate on the camera still image output port 72 works but doesn't work on video output port 71 i.e. again there are no buffers on the output of image_encode.

Return to “OpenMAX”

Who is online

Users browsing this forum: No registered users and 1 guest