MattOwnby
Posts: 58
Joined: Thu Aug 16, 2012 7:22 pm

OpenMAX+GLES2 : Decode and render JPEG (with source code)

Tue Oct 08, 2013 8:09 pm

Greetings,

It's been a while (a year+) since my last tutorial ( http://www.raspberrypi.org/phpBB3/viewtopic.php?t=15463 ).

I've been wanting to release a follow-up to that for quite a while and I've finally had enough people inquiring about it to prompt me to do it.

This sample code demonstrates what I consider to be one of the most useful features of the Raspberry Pi: the ability to decode media using OpenMAX directly to a texture so that it can be rendered in arbitrary ways using GLES2.

Benefits of this approach:
- very good performance (almost completely hardware accelerated)
- flexible (you can render the resulting image in very arbitrary ways with GLES2)

UDPATE: I found a race condition bug in the v1 source code which never seemed to show up on the older Pi B, but shows up all the time on my new Pi2 B that I just got. I've found the problem and uploaded a fixed v2 source code.

Download here: http://www.rulecity.com/browse/rpi/jpeg_gles2_v2.tar.gz
Last edited by MattOwnby on Fri Apr 24, 2015 7:31 pm, edited 2 times in total.

MattOwnby
Posts: 58
Joined: Thu Aug 16, 2012 7:22 pm

Re: OpenMAX+GLES2 : How to decode and render a JPEG

Tue Oct 08, 2013 8:32 pm

Something I forgot to mention is I believe the JPEG size is limited to 2048x2048 (max size of texture for pi hardware?). If you try to go higher, it will throw an exception.

wondras
Posts: 3
Joined: Tue Sep 25, 2012 4:06 pm

Re: OpenMAX+GLES2 : How to decode and render a JPEG

Tue Oct 08, 2013 9:32 pm

Awesome work, Matt! :)

Am curious what others will cook up with this...

Chris-pi
Posts: 10
Joined: Sat Oct 19, 2013 2:41 am

Re: OpenMAX+GLES2 : Decode and render JPEG (with source code

Sat Oct 19, 2013 3:08 am

Many thanks Matt, great work and very helpful.

i have been struggling trying to decode mjpeg stream from my tenvis mini319W wifi camera. i have made code based on jpeg.c/h that parses single jpeg image data from mjpeg stream and displays them on screen. works ok with other (iphone ip_cam app) wifi camera, but does not work with tenvis camera.

investigating, i have made file from one image cut from stream from tenvis camera. it displays ok if i double click it in file explorer, so i believe it is valid/good jpg image. yet it will not display when i use either "jpeg_gles2" or my app (based on jpeg.c/h) to display it.

i'm wondering if omx hardware on raspberry pi somehow does not support this jpg format? and therefore i should give up and buy new camera. or ...????? i do not think it is "abbreviated" or "progressive scan" format.

Any thoughts would be appreciated.

i have attached jpg file i created from tenvis mjep stream.

Thanks
Chris
Attachments
tenvis3.jpg
tenvis3.jpg (14.86 KiB) Viewed 17479 times

MattOwnby
Posts: 58
Joined: Thu Aug 16, 2012 7:22 pm

Re: OpenMAX+GLES2 : Decode and render JPEG (with source code

Sun Oct 20, 2013 5:48 am

Chris-pi wrote:Many thanks Matt, great work and very helpful.

i have been struggling trying to decode mjpeg stream from my tenvis mini319W wifi camera. i have made code based on jpeg.c/h that parses single jpeg image data from mjpeg stream and displays them on screen. works ok with other (iphone ip_cam app) wifi camera, but does not work with tenvis camera.

investigating, i have made file from one image cut from stream from tenvis camera. it displays ok if i double click it in file explorer, so i believe it is valid/good jpg image. yet it will not display when i use either "jpeg_gles2" or my app (based on jpeg.c/h) to display it.

i'm wondering if omx hardware on raspberry pi somehow does not support this jpg format? and therefore i should give up and buy new camera. or ...????? i do not think it is "abbreviated" or "progressive scan" format.

Any thoughts would be appreciated.

i have attached jpg file i created from tenvis mjep stream.
Thanks for the feedback!

Have you tried stepping through the code with gdb to see on which line it is failing?

MattOwnby
Posts: 58
Joined: Thu Aug 16, 2012 7:22 pm

Re: OpenMAX+GLES2 : Decode and render JPEG (with source code

Sun Oct 20, 2013 4:02 pm

I just tried it on the .jpg you attached and it seems to be working perfectly. Perhaps when you attached the jpg, the message board software modified it somehow?

Code: Select all

pi@raspberrypi /mnt/jpeg_gles2 $ ./jpeg_gles2 tenvis3.jpg 
^CProperly shutting down...
Total elapsed milliseconds: 173435
Total frames displayed: 27720
Total frames / second is 159.829331

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5748
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: OpenMAX+GLES2 : How to decode and render a JPEG

Sun Oct 20, 2013 4:22 pm

MattOwnby wrote:Something I forgot to mention is I believe the JPEG size is limited to 2048x2048 (max size of texture for pi hardware?). If you try to go higher, it will throw an exception.
2048x2048 is the max EGL texture size. You can decode larger jpegs, but you will need a image_decode->resize->egl_render pipeline.
If you set nSliceHeight to 16, then the decode/resize will happen in stripes and very large jpegs can be decoded without taking any more gpu memory than the resized-down texture size.

Chris-pi
Posts: 10
Joined: Sat Oct 19, 2013 2:41 am

Re: OpenMAX+GLES2 : Decode and render JPEG (with source code

Sun Oct 20, 2013 5:16 pm

Wow, it worked for you. that is good news.

i downloaded tenvis3.jpg so that i would have same file as you. i renamed it tenvis4.jpg. again, it did not work for me.

screen goes black, after a bit of time i type Ctrl-c, and this is what is in cmd window
-----------------------------------------------------------
pi@raspberrypi ~/Chris_data/jpeg_gles2 $ ./jpeg_gles2 tenvis4.jpg
JPEGOpenMax::DecompressJPEGStart exception: Waiting timed out
JPEGOpenMax::WaitJPEGDecompressorReady exception: Not decoding
JPEGOpenMax::DecompressJPEGStart exception: Waiting timed out
JPEGOpenMax::WaitJPEGDecompressorReady exception: Not decoding
JPEGOpenMax::DecompressJPEGStart exception: Waiting timed out
JPEGOpenMax::WaitJPEGDecompressorReady exception: Not decoding
^CProperly shutting down...
JPEGOpenMax::DecompressJPEGStart exception: Waiting timed out
JPEGOpenMax::WaitJPEGDecompressorReady exception: Not decoding
Total elapsed milliseconds: 8048
Total frames displayed: 4
Total frames / second is 0.497018
terminate called after throwing an instance of 'std::runtime_error'
what(): OMX_FreeBuffer failed
Aborted
pi@raspberrypi ~/Chris_data/jpeg_gles2 $
---------------------------------------------------------

image size is 240x320 (judging by ffc0 frame in jpg file, and that is size i set camera to display). does that mean i don't need to worry about 2048x2048 max texture size?

i will try stepping through the code with gdb as you suggest. when i had problem with jpeg.c based code it was having problem with
ret =
ilclient_wait_for_event
(decoder->imageDecoder->component,
OMX_EventPortSettingsChanged,
decoder->imageDecoder->outPort, 0, 0, 1, 0, 5);
returning -1 after
OMX_EmptyThisBuffer(decoder->imageDecoder->handle,
pBufHeader);
i changed 5 to 50 but result was the same.

i am new to linux and pi, so don't feel you will insult me with obvious suggestions :-)

Thanks again for your time with this. i will let you know results of gdb stepping.

MattOwnby
Posts: 58
Joined: Thu Aug 16, 2012 7:22 pm

Re: OpenMAX+GLES2 : Decode and render JPEG (with source code

Sun Oct 20, 2013 5:50 pm

Chris-pi wrote: i am new to linux and pi, so don't feel you will insult me with obvious suggestions :-)
Ok, here are a few obvious things I can think of:

- have you tested with the pre-built binary that I included in my .tar.gz file? (to rule out build issues on your end)
- have you tested with the .jpg file that I included in my .tar.gz file to make?

Chris-pi
Posts: 10
Joined: Sat Oct 19, 2013 2:41 am

Re: OpenMAX+GLES2 : Decode and render JPEG (with source code

Sun Oct 20, 2013 6:55 pm

-i tried your pre compiled jpeg_gles2, same result (no working).
-i tried with your included lair1.jpg, it worked OK.

Chris-pi
Posts: 10
Joined: Sat Oct 19, 2013 2:41 am

Re: OpenMAX+GLES2 : Decode and render JPEG (with source code

Mon Oct 21, 2013 1:00 am

It is working now !!!
--------------------------------------
sudo apt-get update
sudo apt-get upgrade
--------------------------------------
did the trick. i was only a few months out of date but....

Thank you so much for all your help. Knowing that my jpeg worked for you was big help. i was ready to start buying wifi camera's until i found one that worked. Not sure how many i would have bought ;-)

chris

anihilator
Posts: 1
Joined: Sat Nov 23, 2013 2:48 pm

Re: OpenMAX+GLES2 : Decode and render JPEG (with source code

Sat Nov 23, 2013 3:25 pm

Hello,

I have same problem like Chris-pi but update & upgrade solution doesn't works for me. Any idea why only some types of JPGs are compatible with your code?

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5748
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: OpenMAX+GLES2 : Decode and render JPEG (with source code

Sat Nov 23, 2013 4:51 pm

anihilator wrote:I have same problem like Chris-pi but update & upgrade solution doesn't works for me. Any idea why only some types of JPGs are compatible with your code?
Porgressive jpegs won't work.

taj1989
Posts: 2
Joined: Fri Apr 04, 2014 4:28 pm

Re: OpenMAX+GLES2 : Decode and render JPEG (with source code

Fri Apr 04, 2014 4:43 pm

Hi guys,

I'm in the same situation, just bought some IP cameras, and trying to decode the snapshots with openmax. They are all wanscam cameras, but different models.
Decoding is working with the first camera but not working with the other one. The test.jpg does not work, but test2.jpg works.
Here are the files:
http://wikisend.com/download/947528/jpeg.rar
I ran an jpg analysis on this site: http://www.imageforensic.org/
There was only one difference I noticed: the file format for the working picture is: JPEG image data,
and for the one that not works: JPEG image data, JFIF standard 1.01, baseline, precision 0, 4360x480
These are from that website.
Can somebody run a decode on my pictures please? I tried apt-get update/upgrade, but didn't help.
Thanks for your help!

crustea
Posts: 1
Joined: Thu Jan 15, 2015 2:26 pm

Re: OpenMAX+GLES2 : Decode and render JPEG (with source code

Thu Jan 15, 2015 2:31 pm

This is pretty exciting !
Matt, would you mind sharing stats / performances ?

Any chance this could read jpg images sequences, 720p at 30fps ?
I'm really intrigued, this would be a game changer for an open project I have in mind..
http://forum.openframeworks.cc/t/gpu-jp ... ject/18221

Best,
Joanie

User avatar
PeterO
Posts: 6095
Joined: Sun Jul 22, 2012 4:14 pm

Re: OpenMAX+GLES2 : Decode and render JPEG (with source code

Thu Jan 15, 2015 4:02 pm

Old thread, and "Matt" has not visited the forums for 7 months so you probably won't get a reply !
PeterO
Discoverer of the PI2 XENON DEATH FLASH!
Interests: C,Python,PIC,Electronics,Ham Radio (G0DZB),1960s British Computers.
"The primary requirement (as we've always seen in your examples) is that the code is readable. " Dougie Lawson

MattOwnby
Posts: 58
Joined: Thu Aug 16, 2012 7:22 pm

Re: OpenMAX+GLES2 : Decode and render JPEG (with source code

Fri Apr 24, 2015 7:45 pm

crustea wrote:This is pretty exciting !
Matt, would you mind sharing stats / performances ?

Any chance this could read jpg images sequences, 720p at 30fps ?
I'm really intrigued, this would be a game changer for an open project I have in mind..
http://forum.openframeworks.cc/t/gpu-jp ... ject/18221

Best,
Joanie
I am getting (as of today) 112 FPS decoding a 640x480 JPEG.

MattOwnby
Posts: 58
Joined: Thu Aug 16, 2012 7:22 pm

Re: OpenMAX+GLES2 : Decode and render JPEG (with source code

Fri Apr 24, 2015 7:45 pm

Just so everyone knows, there was a long-standing bug in this code that never showed up for me before. I just found, fixed it, and uploaded said fix.

efs113
Posts: 3
Joined: Sun Jun 14, 2015 2:55 am

Re: OpenMAX+GLES2 : Decode and render JPEG (with source code

Sun Jun 14, 2015 3:05 am

I'm trying to use jpeg_gles2 to display a stream of JPEG images.

As a test, I've modified the original code to accept two JPEGs on the command. It reads both in prior to the main while loop. A modulus within the while loop switches decoding between the two images. Rather than include the full main.cpp, here is the diff of the changes:

Code: Select all

<       if (argc != 2)
---
>       if (argc != 3)
105,107c105,111
<       byteSA fileJPEG = read_file(argv[1]);   // load in jpeg file
<       const uint8_t *pBufJPEG = fileJPEG.data();
<       size_t stSizeBytes = fileJPEG.size();
---
>       byteSA fileJPEG1 = read_file(argv[1]);  // load in jpeg file
>       const uint8_t *pBufJPEG1 = fileJPEG1.data();
>       size_t stSizeBytes1 = fileJPEG1.size();
>
>       byteSA fileJPEG2 = read_file(argv[2]);  // load in jpeg file
>       const uint8_t *pBufJPEG2 = fileJPEG2.data();
>       size_t stSizeBytes2 = fileJPEG2.size();
110c114
<       pJPEG->SetInputBufSizeHint(stSizeBytes);
---
>       pJPEG->SetInputBufSizeHint(50000);
119,120c123,129
<               pJPEG->DecompressJPEGStart(pBufJPEG, stSizeBytes);
<               pJPEG->WaitJPEGDecompressorReady();
---
>                 if (uFramesDisplayed % 2) {
>                   pJPEG->DecompressJPEGStart(pBufJPEG1, stSizeBytes1);
>                   pJPEG->WaitJPEGDecompressorReady();
>                 } else {
>                   pJPEG->DecompressJPEGStart(pBufJPEG2, stSizeBytes2);
>                   pJPEG->WaitJPEGDecompressorReady();
>                 }
I have two images I am trying to decode and render. If I pass the same image on the command line twice, the program executes successfully. For example, I can call "jpeg_gles2 img1.jpg img1.jpg" and the image renders. I can also call "jpeg_gles2 img2.jpg img2.jpg" and the image renders.

However, if I call "jpeg_gles2 img1.jpg img2.jpg", the first image renders and then I get the following error:
JPEGOpenMax::WaitJPEGDecompressorReady exception: Waiting timed out
jpeg_gles2: JPEGOpenMax.cpp:181: void JPEGOpenMax::EmptyThisBuffer(OMX_BUFFERHEADERTYPE*): Assertion `m_pHeaderOutput == 0' failed.
Aborted

Any suggestions would be greatly appreciated.

Thanks,
Eric

MattOwnby
Posts: 58
Joined: Thu Aug 16, 2012 7:22 pm

Re: OpenMAX+GLES2 : Decode and render JPEG (with source code

Mon Jun 15, 2015 10:26 pm

Make sure that SetInputBufSizeHint is the maximum size of any jpeg you want to decode. I see you have it hard-coded at 50,000 which could bite you later if you forget to change it.

Uncomment "//#define VERBOSE" inside OMXComponent.h and it will spam a ton of debug info to the console. It looks like it is blocking on some other openmax event that it is waiting for you to handle which you are not handling.

efs113
Posts: 3
Joined: Sun Jun 14, 2015 2:55 am

Re: OpenMAX+GLES2 : Decode and render JPEG (with source code

Tue Jun 16, 2015 2:54 am

Thanks for the quick reply Matt.

50000 bytes is large enough for both JPEGs I am trying to decode. As I noted in the original post, both decode successfully using my modified code if I pass the same image twice on the command line. I only run into problems when I try to alternate between the images.

I enabled debug. Output is below. The issue appears to be a unexpected EventPortsSettingsChanged(). There is a comment in the code, JPEGOpenMax.cpp:180, that says "we don't expect to get a port settings changed event after we've already set up the renderer". This is where the issue is. Not sure why I am getting the port change setting.

I'm trying, unsuccessfully, to modify the jpeg_gles2 program to decode and render a stream of JPEGs. I'm new to OpenMAX and the learning curve is quite steep.

Code: Select all

Handle c42d68 got event: OMX_EventCmdComplete (0) ndata1: 2 ndata2: 320 pEventData 0
Handle c42d68 got event: OMX_EventCmdComplete (0) ndata1: 2 ndata2: 321 pEventData 0
Handle c42ea8 got event: OMX_EventCmdComplete (0) ndata1: 2 ndata2: 220 pEventData 0
Handle c42ea8 got event: OMX_EventCmdComplete (0) ndata1: 2 ndata2: 221 pEventData 0
Handle c42d68 got event: OMX_EventCmdComplete (0) ndata1: 0 ndata2: 2 pEventData 0
Handle c42d68 got event: OMX_EventCmdComplete (0) ndata1: 3 ndata2: 320 pEventData 0
Handle c42d68 got event: OMX_EventCmdComplete (0) ndata1: 0 ndata2: 3 pEventData 0
Got EmptyBufferDone
Handle c42d68 got event: OMX_EventPortSettingsChanged (3) ndata1: 321 ndata2: 0 pEventData 0
Handle c42ea8 got event: OMX_EventCmdComplete (0) ndata1: 3 ndata2: 220 pEventData 0
Handle c42ea8 got event: OMX_EventPortSettingsChanged (3) ndata1: 221 ndata2: 0 pEventData 0
Handle c42d68 got event: OMX_EventCmdComplete (0) ndata1: 3 ndata2: 321 pEventData 0
Handle c42ea8 got event: OMX_EventCmdComplete (0) ndata1: 0 ndata2: 2 pEventData 0
Handle c42ea8 got event: OMX_EventPortSettingsChanged (3) ndata1: 221 ndata2: 0 pEventData 0
Handle c42ea8 got event: OMX_EventCmdComplete (0) ndata1: 0 ndata2: 3 pEventData 0
Handle c42ea8 got event: OMX_EventCmdComplete (0) ndata1: 3 ndata2: 221 pEventData 0
Handle c42d68 got event: OMX_EventBufferFlag (4) ndata1: 321 ndata2: 1 pEventData 0
Handle c42ea8 got event: OMX_EventBufferFlag (4) ndata1: 221 ndata2: 17 pEventData 0
Got FillBufferDone
Got EmptyBufferDone
Handle c42ea8 got event: OMX_EventCmdComplete (0) ndata1: 2 ndata2: 221 pEventData 0
Handle c42d68 got event: OMX_EventPortSettingsChanged (3) ndata1: 321 ndata2: 0 pEventData 0
Handle c42ea8 got event: OMX_EventCmdComplete (0) ndata1: 3 ndata2: 221 pEventData 0
JPEGOpenMax::WaitJPEGDecompressorReady exception: Waiting timed out
Got FillBufferDone
Handle c42ea8 got event: OMX_EventCmdComplete (0) ndata1: 2 ndata2: 221 pEventData 0
jpeg_gles2: JPEGOpenMax.cpp:181: void JPEGOpenMax::EmptyThisBuffer(OMX_BUFFERHEADERTYPE*): Assertion `m_pHeaderOutput == 0' failed.
Aborted

efs113
Posts: 3
Joined: Sun Jun 14, 2015 2:55 am

Re: OpenMAX+GLES2 : Decode and render JPEG (with source code

Tue Jun 16, 2015 11:52 am

If the two input JPEG images I am attempting to decode are encoded at different resolutions, will this result in an EventsPortsSettingsChanged?

MattOwnby
Posts: 58
Joined: Thu Aug 16, 2012 7:22 pm

Re: OpenMAX+GLES2 : Decode and render JPEG (with source code

Tue Jun 16, 2015 5:28 pm

efs113 wrote:If the two input JPEG images I am attempting to decode are encoded at different resolutions, will this result in an EventsPortsSettingsChanged?
Most likely it would. I think you've found the problem. You are going to need to check for and handle the port settings changed events. Hopefully that won't be too hard since you have an example of how to do it :) I would strongly recommend stepping through the code with 'gdb' to troubleshoot this further.

MaximilianBrunner
Posts: 7
Joined: Thu Feb 04, 2016 9:42 pm

Re: OpenMAX+GLES2 : How to decode and render a JPEG

Thu Feb 04, 2016 9:49 pm

dom wrote:
MattOwnby wrote:Something I forgot to mention is I believe the JPEG size is limited to 2048x2048 (max size of texture for pi hardware?). If you try to go higher, it will throw an exception.
2048x2048 is the max EGL texture size. You can decode larger jpegs, but you will need a image_decode->resize->egl_render pipeline.
If you set nSliceHeight to 16, then the decode/resize will happen in stripes and very large jpegs can be decoded without taking any more gpu memory than the resized-down texture size.
Could you please explain how/where to set this nSliceHeight variable?
If i understand you correctly this can be used to resize the jpeg-image before decoding it.

Thanks, Max

stefano.setti
Posts: 24
Joined: Wed Sep 18, 2019 1:13 pm

Re: OpenMAX+GLES2 : Decode and render JPEG (with source code)

Mon Oct 21, 2019 2:53 pm

Is it still working on last raspbian version (Buster) on Raspberry Pi 3 B?
When compile I get the following error

Code: Select all

/usr/bin/ld: video/VideoObjects/VideoObjectGLES2_EGL.o: undefined reference to symbol 'eglCreateImageKHR'
/usr/bin/ld: //opt/vc/lib/libbrcmEGL.so: error adding symbols: DSO missing from command line

Return to “C/C++”