OpenMAX IL demos


11 posts
by tjormola » Mon Sep 30, 2013 11:33 am
Hi!

I developed some OpenMAX IL demos during the process of learning how to control the RaspiCam camera module from application code. Someone might find these useful.

Homepage: http://solitudo.net/software/raspberryp ... max-demos/
GitHub: https://github.com/tjormola/rpi-openmax-demos/
Posts: 6
Joined: Mon Sep 30, 2013 10:47 am
by Thaddy » Sun Oct 06, 2013 4:42 pm
I do.

Tnx!
Posts: 165
Joined: Wed Mar 06, 2013 11:28 am
by dom » Sun Oct 06, 2013 5:20 pm
I have written a fair bit of openmax client code and find it very hard. You have to get an awful lot right before you get anything useful out.
Just lots of OMX_ErrorInvalidState, and OMX_ErrorBadParameter messages if you are lucky. Nothing happening at all if you are not...

So, congratulations on getting things working, and good to see you making the code available.
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 4011
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge
by tjormola » Mon Oct 07, 2013 11:16 pm
Thanks.

Updated code, most importantly fixed bugs related to YUV I420 data handling so valid I420 is emitted. Now e.g. this example works where H.264 encoded live picture from the RaspiCam is streamed over network using Gstreamer (tested with version 1.2.0 on both receiving end and sending Raspberry Pi):

First on a Linux desktop machine:

Code: Select all
gst-launch-1.0 -e  tcpserversrc host=DESKTOP_HOST port=5000 ! h264parse ! avdec_h264 ! videoconvert ! ximagesink


Then on the Raspberry Pi:

Code: Select all
./rpi-camera-dump-yuv 2>/dev/null | ./rpi-encode-yuv  2>/dev/null | gst-launch-1.0 -e fdsrc ! tcpclientsink host=DESKTOP_HOST port=5000
Posts: 6
Joined: Mon Sep 30, 2013 10:47 am
by SnowLeopard » Sat Oct 19, 2013 12:26 am
Nice work. Your code is helping me understand a lot.

Since the output reports the output port (201) supports a bunch of other video formats, I tried each of the others by changing format.eCompressionFormat (line 714) to no avail. Even reducing the width and height didn't help. Same error for each codec other than h264:
Enabling ports...
Received event 0x00000000 command complete, hComponent:0x001198e0, nData1:0x00000003, nData2:0x00000046
Received event 0x00000000 command complete, hComponent:0x001198e0, nData1:0x00000003, nData2:0x00000047
Received event 0x00000001 error, hComponent:0x0011b9e8, nData1:0x80001000, nData2:0x00000001
OMX error: error event received: 0x80001000 insufficient resource

Presumably, it's the OMX_SendCommand on port 201 that's failing, but I would have expected it to return with an error and then your omx_die with the explicit message to appear instead.
Posts: 106
Joined: Sun Aug 18, 2013 6:10 am
by ejarosek » Sun Oct 20, 2013 2:24 am
Out of curiosity, was there a limitation to the MMAL libraries that caused you to use OpenMAX IL instead or was it just to do as a learning experience?
Posts: 28
Joined: Mon Aug 27, 2012 6:39 pm
by SnowLeopard » Wed Oct 23, 2013 5:24 pm
ejarosek wrote:Out of curiosity, was there a limitation to the MMAL libraries that caused you to use OpenMAX IL instead or was it just to do as a learning experience?


OpenMax has documentation, MMAL does not?
Posts: 106
Joined: Sun Aug 18, 2013 6:10 am
by tjormola » Fri Oct 25, 2013 5:45 pm
ejarosek wrote:Out of curiosity, was there a limitation to the MMAL libraries that caused you to use OpenMAX IL instead or was it just to do as a learning experience?

I want to use Raspberry Pi with the camera module as a car dash camera so I can record my own crazy Russian crash videos like this http://www.youtube.com/watch?v=xS_Sc708MS4 ;)
I want to create Matroska files with the video track and a subtitle track showing speed and location using a USB GPS module. In order to do this so that everything is in sync, I need to drive the camera from the application code, e.g. piping from raspivid just wouldn't do the trick. So I started my research how this could be done.

The RaspiCam docs at http://www.raspberrypi.org/wp-content/u ... tation.pdf says that the tools are using MMAL which is a layer on top of this thing called OpenMAX. So I researched what MMAL and OpenMAX are. OpenMAX is a standard, MMAL is not, there was the full specification available for OpenMAX, no docs for MMAL. And despite OpenMAX seems to be quite difficult to use and produces verbose code, still it was obvious to me that I want to use OpenMAX in my application to drive the camera, not MMAL.

Initially I thought I use libav for video handling in my application. But then I found out about this great plugin for Gstreamer called gst-omx: http://cgit.freedesktop.org/gstreamer/gst-omx/ By using Gstreamer, I'd get lots of stuff for free. And also my application is being written in Vala, and there's Vala bindings for Gstreamer so it would be very easy.

Unfortunately, gst-omx doesn't support the OpenMAX camera module, so first I needed to fix that. You can check out the progress here: https://bugzilla.gnome.org/show_bug.cgi?id=709970 Summa summarum, hopefully soon we have way to get live H.264 video from the RaspiCam to Gstreamer pipeline natively.

So I created these standalone demos to teach myself OpenMAX and then use that knowledge to implement RaspiCam support for Gstreamer and then use Gstreamer API to create this car dash camera application.
Posts: 6
Joined: Mon Sep 30, 2013 10:47 am
by tjormola » Fri Oct 25, 2013 6:00 pm
SnowLeopard wrote:Nice work. Your code is helping me understand a lot.

Since the output reports the output port (201) supports a bunch of other video formats, I tried each of the others by changing format.eCompressionFormat (line 714) to no avail. Even reducing the width and height didn't help. Same error for each codec other than h264:
Enabling ports...
Received event 0x00000000 command complete, hComponent:0x001198e0, nData1:0x00000003, nData2:0x00000046
Received event 0x00000000 command complete, hComponent:0x001198e0, nData1:0x00000003, nData2:0x00000047
Received event 0x00000001 error, hComponent:0x0011b9e8, nData1:0x80001000, nData2:0x00000001
OMX error: error event received: 0x80001000 insufficient resource

Presumably, it's the OMX_SendCommand on port 201 that's failing, but I would have expected it to return with an error and then your omx_die with the explicit message to appear instead.

Yeah, the hardware reports that a bunch of codecs are supported. But unfortunately, it looks like actually H.264 is the only codec that is really implemented in the firmware. I had to blacklist these codecs in my gst-omx work that I described in the previous posting. See https://github.com/tjormola/gst-omx/blo ... src.c#L738

I wonder if someone from the Raspberry Pi Foundation working on the firmware could shed some light on this? Why the extra codecs are listed to be supported but actually aren't? Are you planning to add support for e.g. MJPEG? If not, shouldn't you only return H.264 as supported codec when getting the OMX_IndexParamVideoPortFormat parameter?
Posts: 6
Joined: Mon Sep 30, 2013 10:47 am
by peepo » Mon Dec 30, 2013 10:52 am
tjormola,

much appreciate your released code, and have played with it somewhat...

however your intention was to create " a car dash camera so I can record"

did you ever manage to create and release the necessary code?
ie I am clueless and struggling with this aspect of openMAX.

planning to use openGL textures and
hoping to avoid gstreamer dependency...
User avatar
Posts: 268
Joined: Sun Oct 21, 2012 9:36 am
by tjormola » Mon Dec 30, 2013 6:07 pm
That car dash camera application has been on hold for a few months now due to lack of time. I'm planning to base it on Gstreamer and also support non-rpi-camera cameras so that wouldn't probably be very useful in your OpenGL use case anyway..
Posts: 6
Joined: Mon Sep 30, 2013 10:47 am